Unsupervised Anomaly Detection Using SVD for Cybersecurity Threats
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current anomaly detection technologies in cybersecurity often require predefined patterns or supervised learning, which can be inefficient in identifying potential threats without prior knowledge and may not effectively extract causal information about detected anomalies.
Innovation Solution
A novel unsupervised anomaly detection method using singular value decomposition (SVD) to identify anomalous datapoints and extract causal features, allowing for the detection of potential cybersecurity threats without pre-defined patterns and providing insights into the causes of anomalies through angular relationships and magnitude information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If supervised learning or rules-based anomaly detection is used, then detection accuracy can be improved with known threat patterns, but the system requires pre-defined patterns and cannot effectively identify novel threats without prior knowledge
Solution Approach 1:
The system performs self-service by automatically learning the normal behavior patterns from the dataset itself without requiring external pre-defined rules or supervised training labels. The unsupervised learning algorithm autonomously identifies what constitutes normal activity and detects anomalies that deviate from this learned baseline, enabling both accurate detection and adaptability to novel threats simultaneously.
Solution Approach 2:
Instead of defining what anomalies look like through pre-defined patterns or labels (traditional approach), the system inverts the approach by first learning what normal behavior looks like and then identifying anything that deviates from this normal pattern as potentially anomalous. This inversion allows the system to detect both known and novel threats without requiring prior knowledge of specific attack patterns.
2Productivity
If anomaly detection is performed without extracting causal information, then detection speed can be improved, but the system cannot provide insights into the causes of detected anomalies
Solution Approach 1:
The system segments the anomaly detection process into two distinct phases: (1) a fast detection phase that quickly identifies anomalous datapoints using unsupervised learning, and (2) a causal analysis phase that extracts meaningful insights about the causes of detected anomalies. This segmentation allows the system to maintain high detection speed while separately providing comprehensive causal information when needed.
Solution Approach 2:
The system extracts causal information as a separate output from the anomaly detection process. By taking out the causal analysis as a distinct function that operates on already-detected anomalies, the system maintains fast detection speed while providing additional explanatory information about the causes of anomalies, thus preventing loss of causal information without compromising detection efficiency.
3Reliability
If complex anomaly detection algorithms are used, then detection capability can be improved, but the device complexity and computational requirements increase
Solution Approach 1:
The system uses computationally efficient unsupervised learning algorithms that can be executed quickly and discarded or re-run as needed, rather than relying on complex, resource-intensive models that require extensive training infrastructure. This approach maintains strong anomaly detection capability while significantly reducing device complexity and computational requirements.
Solution Approach 2:
The system changes the fundamental parameters of the detection approach by using unsupervised learning instead of supervised learning, which allows for simpler algorithmic complexity. By changing from pattern-matching based on pre-defined rules to statistical anomaly detection based on learned normal behavior patterns, the system achieves reliable detection capability with reduced computational overhead and device complexity.
Data Source
AI summary
Computer-implemented method of detecting potential cybersecurity threats from collected data pertaining to a monitored network, the collected data comprising network data and/or endpoint data. The method comprises structuring the collected data as at least one data matrix, each row of the data matrix being a datapoint and each column corresponding to a feature. The method also comprises identifying one or more datapoints as anomalous, thereby detecting a potential cybersecurity threat. The method also comprises extracting causal information about the anomalous datapoint based on an angular relationship between a second-pass coordinate vector of the anomalous datapoint and a second-pass coordinate vector of one or more features. The second-pass coordinate vectors are determined by applying a second-pass singular value decomposition (SVD) to a residuals matrix. The residuals matrix is computed between the data matrix and an approximation of the data matrix by applying a first-pass truncated SVD to the data matrix.