Unsupervised ML Anomaly Detection for Network Security
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Managed Detection and Response (MDR) services face challenges in efficiently detecting cyberattacks due to the manual analysis of large volumes of data, which is time-consuming, error-prone, and unable to identify subtle patterns or novel attack types, leading to potential undetected breaches.
Innovation Solution
A machine learning anomaly detection system that implements a pipeline with pre-processing, dimensionality reduction, and anomaly scoring using unsupervised machine learning models, capable of identifying outlier processes or hosts without manual labeling, thereby prioritizing alerts and reducing analyst workload.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual data analysis is used in MDR services, then security experts can detect and respond to cyberattacks, but the process becomes extremely slow, tedious, and error-prone
Solution Approach 1:
The patent replaces manual mechanical analysis by security experts with an automated machine learning system that processes metadata and hunt data. The system uses trained ML models to perform anomaly detection, substituting human analysts' mechanical work with automated computational processes that are faster and more consistent.
Solution Approach 2:
The system enables self-service through automated anomaly detection where the ML models independently analyze data without requiring continuous human intervention. The unsupervised learning models automatically identify anomalies and generate findings, allowing the system to serve itself in detecting security threats without constant expert oversight.
2Adaptability or versatility
If manual hunt process is used, then security analysts can examine data for compromise signals, but human analysts are not always sensitive to subtle patterns and cannot easily identify novel attack types
Solution Approach 1:
The patent replaces human pattern recognition capabilities with machine learning models that can detect subtle patterns and novel attack types. The ML systems process vast amounts of metadata and hunt data, identifying patterns that human analysts might miss due to cognitive limitations or bias.
Solution Approach 2:
The system changes the parameters of detection by using multiple ML models with different algorithms and approaches (e.g., unsupervised learning, anomaly detection). This multi-model approach allows the system to detect various types of attacks by changing detection parameters and perspectives, improving adaptability to different attack patterns.
3Reliability
If large volumes of data are analyzed manually, then comprehensive security monitoring is achieved, but the process becomes tedious and error-prone
Solution Approach 1:
The patent replaces tedious manual data processing with automated ML-based analysis systems. The system automatically ingests, processes, and analyzes large volumes of metadata and hunt data, eliminating the manual effort required while maintaining or improving detection reliability through consistent automated processing.
Solution Approach 2:
The system introduces ML models as intermediaries between raw security data and human analysts. These intermediaries automatically process and filter large volumes of data, presenting only relevant findings to analysts. This intermediary layer simplifies operational complexity by handling the tedious processing while maintaining reliable detection through automated analysis.
Data Source
AI summary
An anomaly detection system is disclosed capable of reporting anomalous processes or hosts in a computer network using machine learning models trained using unsupervised training techniques. In embodiments, the system assigns observed processes to a set of process categories based on the file system path of the program executed by the process. The system extracts a feature vector for each process or host from the observation records and applies the machine learning models to the feature vectors to determine an outlier metric each process or host. The processes or hosts with the highest outlier metrics are reported as detected anomalies to be further examined by security analysts. In embodiments, the machine learnings models may be periodically retrained based on new observation records using unsupervised machine learning techniques. Accordingly, the system allows the models to learn from newly observed data without requiring the new data to be manually labeled by humans.