Anomaly Detection System for Network Hunt Data Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current Managed Detection and Response (MDR) services rely heavily on manual data analysis, which is slow, tedious, and error-prone, making it difficult to detect cyberattacks and breaches in a timely and effective manner.
Innovation Solution
A machine learning anomaly detection system that implements a detection pipeline with multiple stages, including pre-processing, dimensionality reduction, anomaly scoring using unsupervised machine learning algorithms, and an interpretability layer, to detect anomalies in hunt data from computer networks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual data analysis is used in MDR services, then human expertise can be applied to detect cyberattacks, but the process becomes extremely slow, tedious, and error-prone
Solution Approach 1:
The patent replaces manual mechanical analysis with automated machine learning models and algorithms that process hunt data, anomaly scores, and security events automatically, eliminating the slow and tedious manual review process while maintaining or improving detection accuracy through consistent application of detection rules
Solution Approach 2:
The system enables self-service anomaly detection by automatically generating anomaly scores, identifying outliers in hunt data, and prioritizing security events without requiring continuous human intervention, allowing the system to serve itself in detecting and flagging potential cyberattacks
2Measurement precision
If human analysts manually examine all hunt data, then subtle patterns and novel attack types can be identified, but the process becomes extremely time-consuming and resource-intensive
Solution Approach 1:
The patent segments the analysis process into distinct automated stages: data collection, feature extraction, anomaly scoring using machine learning models, and result interpretation. This segmentation allows each component to specialize in specific detection tasks, improving pattern recognition efficiency while reducing overall analysis time
Solution Approach 2:
The system introduces an intermediary layer of machine learning models and anomaly scoring algorithms that act as mediators between raw hunt data and human analyst review. This intermediary automatically filters and prioritizes data, presenting only the most suspicious anomalies to analysts for final verification
3Measurement precision
If more features are extracted from hunt data for analysis, then anomaly detection accuracy improves, but the complexity of the detection system increases
Solution Approach 1:
The patent applies preliminary feature extraction and selection before the main anomaly detection process. Hunt data is pre-processed to extract relevant features and compute anomaly scores using machine learning models trained on historical data, so that the main detection system receives pre-prepared, high-quality input features that improve accuracy without requiring the system to handle raw complex data
Data Source
AI summary
An anomaly detection system is disclosed capable of reporting anomalous processes or hosts in a computer network using machine learning models trained using unsupervised training techniques. In embodiments, the system assigns observed processes to a set of process categories based on the file system path of the program executed by the process. The system extracts a feature vector for each process or host from the observation records and applies the machine learning models to the feature vectors to determine an outlier metric each process or host. The processes or hosts with the highest outlier metrics are reported as detected anomalies to be further examined by security analysts. In embodiments, the machine learnings models may be periodically retrained based on new observation records using unsupervised machine learning techniques. Accordingly, the system allows the models to learn from newly observed data without requiring the new data to be manually labeled by humans.