Anomaly Detection in Text Using Cross-Entropy and Language Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection methods in textual items, such as count models and snapshot language models, often fail to accurately identify anomalies due to discrepancies in anomaly definitions, leading to missed alerts or false positives, particularly in call center conversations where context and surprise value are critical.
Innovation Solution
A method that trains a language model on textual items, calculates cross-entropy values for new items, and slides the model based on thresholds or perplexity, incorporating feedback to adjust anomaly detection, ensuring both count and snapshot models are considered for accurate anomaly identification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a count model is used to detect anomalies based on frequency thresholds, then anomalies with high frequency deviations are detected, but false positives occur when frequent but expected texts are flagged
Solution Approach 1:
The patent combines count-based anomaly detection with snapshot language model-based cross-entropy detection into a unified system. The count model identifies potential anomalies based on frequency deviations, while the snapshot language model evaluates the linguistic surprise of these texts. Only texts that satisfy both conditions (frequency anomaly AND high cross-entropy) are flagged, eliminating false positives from frequent but expected texts.
Solution Approach 2:
The snapshot language model acts as an intermediary filter between the count model and the final anomaly detection. It processes the texts identified by the count model and applies cross-entropy evaluation to determine whether they represent genuine anomalies or merely frequent but expected patterns, thereby mediating the detection process to reduce false positives.
2Measurement precision
If a snapshot language model is used to detect anomalies based on cross-entropy, then surprising texts are detected, but anomalies with normal frequency but high surprise value are missed by count-based models
Solution Approach 1:
The patent merges the strengths of both count-based and snapshot language model-based detection. The system first uses the count model to identify texts with frequency anomalies, then applies the snapshot language model to evaluate cross-entropy. This combination ensures that both frequent anomalies and rare but surprising texts are captured, as the snapshot model's high cross-entropy detection complements the count model's frequency-based detection.
Solution Approach 2:
The system dynamically adjusts the detection process by using both count-based and cross-entropy-based evaluation. Rather than relying on a static threshold, it adapts by considering both the frequency deviation and the linguistic surprise value of each text, allowing it to detect anomalies across different frequency ranges and linguistic contexts.
3Adaptability or versatility
If the language model is continuously updated with new textual items, then the model adapts to changing linguistic patterns, but computational complexity and processing time increase
Solution Approach 1:
Instead of continuously updating the language model with every new textual item, the system employs periodic updates at predetermined intervals or when certain conditions are met. This allows the model to adapt to changing linguistic patterns over time while avoiding the computational overhead of continuous retraining, thus balancing adaptability with computational efficiency.
Solution Approach 2:
The system performs preliminary evaluation using the snapshot language model on incoming texts before committing to full model updates. By pre-filtering texts and evaluating their cross-entropy against the current model, it determines whether updates are necessary, thereby avoiding unnecessary computational complexity while maintaining adaptability to genuine linguistic changes.
4Measurement precision
If both count model and snapshot language model are used together, then detection accuracy improves, but system complexity increases
Solution Approach 1:
The system segments the anomaly detection process into two distinct stages: first, the count model identifies potential anomalies based on frequency deviations; second, the snapshot language model evaluates these candidates using cross-entropy. This segmentation allows each model to specialize in its strength while working together in a coordinated pipeline, improving overall detection accuracy without creating an intractably complex monolithic system.
Solution Approach 2:
Rather than having both models evaluate every text independently (which would be excessively complex), the system applies the snapshot language model only to texts already flagged by the count model. This partial application of the more complex model reduces overall system complexity while maintaining the benefits of dual-model detection for the most suspicious cases.
Data Source
AI summary
In an implementation, a method for detecting anomalies in textual items is provided. The method includes: receiving a first plurality of textual items by a computing device; training a language model using the received first plurality of textual items by the computing device; after training the language model, receiving a second plurality of textual items by the computing device; calculating a cross-entropy for each textual item in the second plurality of textual items by the computing device using the language model; and detecting an anomaly in at least one of the textual items of the second plurality of textual items by the computing device using the calculated cross-entropies.