DETECTING MACHINE-GENERATED TEXT BY ITS STATISTICAL PROPERTIES

Abstract

The article considers the modification of the machine-generated text detection method Fast-DetectGPT. Unlike the original method, which uses the logarithmic probabilities of all tokens in the dictionary to classify text, the proposed method uses only the probabilities of the 20 most probable tokens. This reduces the amount of data that needs to be processed for classification: 20 tokens instead of 50,000 for each token of the input text. The required 20 tokens can be obtained by querying cloud services that provide access to large language models, rather than running the model locally on the device. The proposed method uses statistical parameters of the text and probable tokens, so the classification result can be explained. This is important in the case of establishing the author of the text.  To detect machine-generated text, you can use not only the d_score metric, but also perplexity, prompt_logprobs_mean, and vocab_samples_mean. Also, to calculate d_score and vocab_samples_mean, you can use the logarithmic probabilities of only the 20 most likely tokens, rather than the entire dictionary. In the case of the facebook/opt125m and gpt2 models, such classification by the vocab_samples_mean value allows you to get even better accuracy than using the entire dictionary. When using the d_score value, the results for these two models are opposite - more accurate classification occurs based on the entire dictionary. The accuracy of macro f1 classification by prompt_mu_score and perplexity is not less than 0.97 in all experiments, but these values do not depend on the probabilities of possible tokens. The accuracy of classification by prompt_logprobs_mean is high even when using only the 20 most likely tokens instead of the entire dictionary. This approach allows the use of cloud services for access to large language models and commercial models, and does not require the deployment of a local model.