METHOD FOR DETECTING TEXT DRIFT IN LANGUAGE MODELS

Abstract

In the modern world of rapid artificial intelligence (AI) adaption across various aspects of life, a need for the adaptation and continual updating of intelligent systems to the changing conditions and demands of the real world emerges, including language models. MLOps systems play a crucial role by ensuring efficient implementation, monitoring, and continuous updating of AI systems to maintain their accuracy and relevance to the current usage context. A key aspect of the effectiveness of language models and technologies of natural language processing, in general, is their ability to timely detect changes in the environment in which the created models operate, i.e., changes like incoming data and how this environment differs from that with which the model was trained. This is the task of detecting data drift - changes in the data distributions that the models operate with, which can significantly affect their performance. 

The most common methods and algorithms for detecting data drift in textual corpora to assess the behavior of language models are reviewed. Semantic changes are identified using word embeddings. A series of experiments are conducted to detect data drift in word embedding values using a set of text corpora, where new context and semantics of the target words is gradually injected. Then drift values are calculated using the selected methods, comparison of results using additional metrics and visual evaluation are proposed.

The results of applying these methods assess the appropriateness of their use for detecting drift, sensitivity, and response to the absence of changes. The most basic metrics for determining the distance between average embeddings and words and some clustering methods demonstrate the most accurate results. In language model application drift detection in text data can be efficiently performed using word embeddings and a variety of metrics for tracking change, since word embeddings specifically represent semantics and context of words in context-aware language models.