Anomaly Scoring via Noise-Contrastive Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed computing systems, identifying anomalous events is challenging due to the complexity of user activity data with numerous attributes, making it impractical to determine whether incoming requests are anomalous or non-anomalous, and existing methods may flag legitimate activity as suspicious if not analyzed within the context of individual user patterns.
Innovation Solution
The use of scoring models trained with noise contrastive estimation techniques to generate anomaly scores based on conditional probabilities for each attribute in incoming requests, allowing for the determination of whether requests are anomalous or non-anomalous by comparing them to historical user activity, and taking action accordingly.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional anomaly detection methods are used to analyze user activity data with numerous attributes, then comprehensive analysis of all attributes is attempted, but the complexity of determining whether incoming requests are anomalous becomes impractical
Solution Approach 1:
The patent segments the complex anomaly detection problem into two distinct phases: an offline training phase where a scoring model is built using historical data, and an online detection phase where the trained model rapidly evaluates incoming requests. This segmentation transforms an otherwise impractical real-time analysis into a feasible operation by pre-processing the complexity during offline training.
Solution Approach 2:
The patent performs preliminary action by training the scoring model offline using historical user activity data before actual anomaly detection is needed. The model learns patterns and relationships from past data in advance, so that during online operation, only simple score calculation is required rather than complex real-time analysis of all attributes.
2Ease of operation
If anomaly detection is performed without considering individual user patterns, then processing is simpler, but legitimate activity may be flagged as suspicious
Solution Approach 1:
The patent applies local quality by customizing the anomaly detection for each individual user rather than applying a uniform threshold to all users. The scoring model learns user-specific patterns from historical data, allowing the detection system to adapt to each user's unique behavior characteristics. This ensures that legitimate activity specific to each user is not incorrectly flagged as suspicious.
3Measurement precision
If a scoring model is trained on historical user activity to improve anomaly detection accuracy, then identification of potentially anomalous requests is enhanced, but computational resources and training time are consumed
Solution Approach 1:
The patent performs the computationally intensive model training in advance during an offline phase, separating it from time-critical online operations. By completing the training work beforehand using historical data, the system avoids consuming computational resources and time during actual anomaly detection operations, achieving high accuracy without real-time performance penalties.
Data Source
AI summary
Techniques for monitoring a computing environment for anomalous activity are presented. An example method includes receiving a request to invoke an action within the computing environment. An anomaly score is generated for the received request by applying a probabilistic model to properties of the request. The anomaly score generally indicates a likelihood that the properties of the request correspond to historical activity within the computing environment for a user associated with the request. The probabilistic model generally comprises a model having been trained using historical activity within the computing environment for a plurality of users, the historical activity including information identifying an action performed in the computing environment and contextual information about a historical request. Based on the generated anomaly score, one or more actions are taken to process the request such that execution of requests having anomaly scores indicative of unexpected activity may be blocked pending confirmation.