Counterfactual Anomaly Detection in Computing Environments
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current anomaly detection systems in computing environments lack efficiency in identifying and isolating sources of malicious or unwanted behavior, often relying on rigid thresholds and pre-generated behavior profiles, which can lead to false positives and missed anomalies.
Innovation Solution
The implementation of unsupervised machine learning to detect anomalies by analyzing input streams of data instances over time, using counterfactual processing to isolate principal values and categorical attributes contributing to anomalies, and automatically flagging or remediating source points, such as computing systems or network components.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If rigid thresholds and pre-generated behavior profiles are used for anomaly detection, then the system structure is simple and easy to implement, but the detection accuracy decreases leading to false positives and missed anomalies
Solution Approach 1:
The system transforms static anomaly detection parameters into dynamic, adaptive parameters that evolve with incoming data. Machine learning models continuously update behavior profiles and threshold values based on observed patterns, enabling the system to adapt to changing normal behaviors while maintaining simple implementation through automated parameter adjustment.
Solution Approach 2:
The anomaly detection system performs self-learning and self-adjustment by automatically generating behavior profiles from observed data without requiring manual configuration. The system serves itself by continuously training models on incoming data streams, automatically updating detection thresholds, and adapting to new normal behaviors, thereby improving accuracy while maintaining ease of implementation.
2Measurement precision
If unsupervised machine learning and counterfactual processing are implemented, then detection accuracy and anomaly isolation capability improve, but the system complexity increases
Solution Approach 1:
The complex unsupervised machine learning system is divided into modular components: data ingestion modules, behavior profile generation modules, anomaly scoring modules, and counterfactual analysis modules. Each module performs a specific function and can be independently configured and maintained, reducing overall system complexity while enabling advanced detection capabilities.
Solution Approach 2:
Behavior profiles serve as intermediaries between raw data and anomaly detection algorithms. These pre-generated contextual representations simplify the input to complex machine learning models, making the system more manageable. The profiles act as a buffer that structures incoming data in ways that are more suitable for subsequent analysis by counterfactual processing and anomaly detection components.
3Ease of operation
If pre-defined rules are used for anomaly detection, then the system is easy to operate and interpret, but the system adaptability to new threats decreases
Solution Approach 1:
The system transitions from static pre-defined rules to dynamic adaptive models that continuously evolve with incoming data. Behavior profiles and detection thresholds automatically adjust to reflect changing normal behaviors and emerging threat patterns, enabling the system to adapt to new threats while maintaining ease of operation through automated adaptation rather than manual rule updates.
Solution Approach 2:
The system performs preliminary analysis by generating behavior profiles and establishing baseline patterns before actual anomaly detection occurs. This preliminary structuring of data and patterns enables the system to quickly adapt to new threats by comparing them against established baselines, maintaining ease of operation through pre-computed references while improving adaptability to emerging threats.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Anomaly and causation detection in computing environments are disclosed. An example method includes receiving an input stream of data instances for a time series, each of the data instances being time stamped and including at least one principle value and a set of categorical attributes; generating anomaly scores for each of the data instances over continuous time intervals; detecting a change in the anomaly scores over the continuous time intervals for the data instances; and identifying which of the set of categorical attributes of the data instances caused the change in the anomaly scores using a counterfactual analysis. The counterfactual analysis may comprise removing a portion of the data instances; regenerating the anomaly scores for each of the remaining data instances over the continuous time intervals; and if the anomaly scores are improved, identifying the portion as a cause of anomalous activity. Recommendations to remediate the cause may be generated.