Hidden Markov Model Anomaly Detection for Network Activity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current automated systems for detecting anomalous computer-implemented service activity face challenges in setting thresholds, leading to high false positives or missed attacks, and are not descriptive enough to capture the sequence of actions involved in cyber attacks, making it difficult to detect and prevent unauthorized access and fraud.
Innovation Solution
The deployment of an ensemble of Hidden Markov Models (HMMs) that analyze sequences of computer actions to detect anomalous activity by assigning probabilities to underlying states, allowing for the detection of patterns that would otherwise go unnoticed, including the use of Gaussian HMM, Gaussian Mixed Model HMM, and Multinomial HMM to classify actions and generate timing vectors for correlation with account activity patterns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If threshold-based detection systems are used to identify anomalous activity, then automated detection capability is improved, but false positives increase and measurement precision deteriorates
Solution Approach 1:
The patent transitions from fixed threshold-based detection to probabilistic modeling using Hidden Markov Models. Instead of using static thresholds to determine anomaly presence, the system models the temporal patterns and probabilities of normal behavior sequences. This allows the system to adapt to varying operational conditions and reduce false positives by understanding contextual patterns rather than relying on rigid cutoff values.
Solution Approach 2:
The patent replaces the mechanical threshold-based decision system with a probabilistic statistical model. Rather than using simple count-based thresholds (e.g., 'more than 5 login attempts'), the system employs HMMs that model the temporal dependencies and probability distributions of normal behavior sequences, enabling more nuanced and accurate anomaly detection.
2Device complexity
If simple threshold-based detection is used, then system complexity is reduced, but the ability to capture attack sequences deteriorates
Solution Approach 1:
The patent segments the detection task into multiple independent HMM components, each modeling a specific aspect of normal behavior (e.g., login patterns, data access sequences, API call rhythms). By dividing the complex detection problem into manageable probabilistic models, the system can capture sophisticated attack sequences without requiring a single monolithic complex system.
Solution Approach 2:
The patent introduces the temporal dimension to anomaly detection by modeling sequences of actions over time. Instead of detecting anomalies based solely on static thresholds of individual actions, the system analyzes the temporal patterns and sequences in which actions occur. This temporal dimension enables detection of sophisticated multi-step attacks that would be invisible to simple threshold-based systems.
3Reliability
If traditional anomaly detection systems are used, then false positives are generated, but reducing thresholds to catch more attacks increases false positives further
Solution Approach 1:
The HMM-based system continuously learns from observed behavior patterns and updates its probabilistic models of normal activity. By monitoring actual sequences of actions and comparing them against the learned models, the system receives feedback that allows it to refine its anomaly detection capabilities over time, reducing false positives while maintaining high sensitivity for actual attacks.
Data Source
AI summary
Devices and techniques are generally described for anomalous network activity detection. In various examples, first application log data comprising a plurality of computer log actions may be received. A hidden Markov model associated may generate a first hidden state vector, where a first value of the first hidden state vector is associated with anomalous activity. A first timing vector associated with a first account may be determined, wherein the timing vector indicates first time periods of typical activity and second periods of typical inactivity for the first account. A first number of elements of the first hidden state vector that have the first value and which correspond to the second time periods of the first timing vector may be determined. Network access may be disabled for the first account.