N-gram Correlation Analysis for System Log Anomaly Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing anomaly detection methods in data processing security are limited as they primarily focus on activity attributes, failing to capture correlations between activity and non-activity attributes, which are crucial for modeling behavioral patterns and detecting anomalies effectively.
Innovation Solution
The method involves partitioning a system log into segments based on common characteristics, generating distinct n-grams from successive events within these segments, and identifying correlations between non-activity attributes to create a richer behavioral model that captures both activity and non-activity correlations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If anomaly detection focuses only on activity attributes, then the detection method is simple, but the detection precision is insufficient
Solution Approach 1:
The patent segments the behavioral model into two distinct components: activity attributes (describing actions) and non-activity attributes (describing context). This segmentation allows the system to capture correlations between these two types of attributes without creating an overly complex monolithic model, thereby improving anomaly detection precision while managing complexity through structured organization.
Solution Approach 2:
The patent adds a new dimension to anomaly detection by incorporating non-activity attributes alongside traditional activity attributes. This dimensional expansion transforms the detection space from single-attribute analysis to multi-attribute correlation analysis, enabling more precise anomaly detection by considering relationships between activity and context attributes.
2Reliability
If the system captures correlations between activity and non-activity attributes, then behavioral modeling becomes more accurate, but the processing complexity increases
Solution Approach 1:
The patent segments correlation processing into distinct phases: first identifying correlations within activity attributes, then identifying correlations within non-activity attributes, and finally integrating these correlations into a unified behavioral model. This segmented approach improves behavioral modeling accuracy while managing processing complexity through systematic organization of correlation identification tasks.
Solution Approach 2:
The patent performs preliminary correlation identification on non-activity attributes before integrating them with activity attribute correlations. This preliminary action allows the system to pre-process and organize context correlations, making the subsequent integration phase more efficient and manageable, thereby improving overall model accuracy without proportionally increasing processing complexity.
3Adaptability or versatility
If traditional n-gram models are used for anomaly detection, then the implementation is straightforward, but the ability to detect complex anomalies is limited
Solution Approach 1:
The patent extends the traditional n-gram model to serve multiple functions: it processes both activity attributes and non-activity attributes, identifies correlations within each attribute type, and integrates these correlations into a unified behavioral model. This multi-functional extension enhances anomaly detection capability while managing model structure complexity through systematic integration of correlation data.
Data Source
AI summary
Identifying correlations between events recorded in a computer system log, the recorded events are generated by a plurality of processes executing on the computer. A system log is partitioned into a plurality of segments, each segment associated with a characteristic found in an event, each segment including one or more events having a same characteristic value. A plurality of attributes of the events in a segment are selected. The attributes selected do not describe an action of the event. One or more distinct n-grams are generated, each distinct n-gram including the selected attributes from successive events within the segment. A distinct n-gram is distinct from all other generated n-grams. A correlation is identified for each first selected attribute of each successive event of an n-gram with all other second selected attributes from each successive event of the n-gram, and the correlations are recorded for each first selected attribute.