Word Embeddings for Anomaly Classification in Event Logs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current anomaly detection methods in enterprise networks are inefficient due to the need to apply rules to all events, leading to slow threat detection and potential spread of threats, especially with the vast amount of data processed daily.
Innovation Solution
A method and system using word embeddings to represent features in event logs as vectors, allowing for rapid identification of anomalous events by comparing each feature to its contextual features, thereby flagging only potentially anomalous logs for further rule-based verification.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If rule-based anomaly detection is applied to all events in enterprise networks, then detection accuracy is maintained, but processing speed deteriorates and computational efficiency decreases
Solution Approach 1:
The patent segments the anomaly detection process into two distinct stages: (1) a fast filtering stage using word embeddings to identify potentially anomalous events by comparing contextual likelihood, and (2) a verification stage applying rule-based detection only to the filtered subset. This segmentation resolves the contradiction by maintaining detection accuracy through the verification stage while achieving high processing speed through the efficient embedding-based filtering stage.
Solution Approach 2:
The patent performs preliminary action by pre-training word embedding models on historical event logs to learn normal contextual patterns before actual anomaly detection. During detection, these pre-trained embeddings quickly score new events to identify anomalies, avoiding the need to apply comprehensive rules to all events. This preliminary modeling enables fast detection while maintaining accuracy through the learned contextual understanding.
2Reliability
If rule-based detection is applied to all events, then comprehensive threat detection is achieved, but computational resources are excessively consumed
Solution Approach 1:
The patent applies partial action by using the word embedding filtering stage to identify a subset of potentially anomalous events, then applying the more computationally intensive rule-based detection only to this partial set rather than all events. This approach maintains reliable threat detection for the identified anomalies while significantly reducing overall computational resource consumption by avoiding unnecessary rule evaluation on normal events.
3Measurement precision
If traditional anomaly detection methods are used, then all events are processed thoroughly, but the time required for threat detection increases
Solution Approach 1:
The patent extracts the contextual pattern recognition function from the traditional rule-based detection process and implements it separately using pre-trained word embeddings. This extraction enables rapid identification of potentially anomalous events based on learned contextual patterns, significantly reducing detection time while maintaining analysis accuracy through the subsequent verification of extracted candidates using comprehensive rules.
Data Source
AI summary
Aspects of the present disclosure describe systems and methods for rapidly detecting threats or other security breaches in enterprise networks. In particular, all enterprise network communications may be monitored to detect anomalous events. In one example, each event log in a collection of event logs may be evaluated, wherein an event log having one or more features is monitored and identified as being anomalous based on identifying one or more anomalous features therein. Anomalous features are identified as being anomalous based on the existence of one or more features in the event log that deviate from characteristic contextual features. Rules or models may thereafter applied to each event log containing the anomalous feature.