Anomaly Detection Pipeline for Scalable ML Deployment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing user and entity behavior analytics (UEBA) solutions for anomaly detection are not scalable and resource-intensive, requiring large computational resources due to the ingestion and processing of extensive data volumes, making them impractical for large-scale deployments.
Innovation Solution
A multi-layer anomaly detection pipeline architecture that processes data in directed acyclic graph (DAG) form, comprising layers for data ingestion, filtering, normalization, feature extraction, and machine learning, allowing for efficient and scalable anomaly detection by reducing data cardinality and computational overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine learning models are used to analyze 30 days of data for anomaly detection, then detection accuracy is improved, but computational resource consumption increases significantly
Solution Approach 1:
The patent segments the 30-day data analysis into multiple smaller time windows (e.g., hourly, daily aggregates). Instead of processing all 30 days of data at once through the machine learning model, the system divides the data into manageable chunks that can be processed incrementally, reducing the computational load per processing cycle while maintaining detection accuracy over time.
Solution Approach 2:
The patent performs preliminary data aggregation and feature extraction in earlier layers of the pipeline before the data reaches the machine learning model. By pre-processing and summarizing data statistics (mean, standard deviation, percentiles) in advance, the system reduces the amount of raw data that needs to be processed by computationally intensive ML models, thereby lowering overall resource consumption.
2Reliability
If UEBA solutions process large volumes of data to detect anomalies, then detection reliability is improved, but system scalability deteriorates
Solution Approach 1:
The patent divides the anomaly detection system into multiple independent processing layers (data ingestion, aggregation, feature extraction, machine learning, alert generation). Each layer processes data independently and feeds results to the next layer, enabling horizontal scaling. This modular architecture allows the system to handle larger data volumes by adding more processing capacity at specific layers without requiring complete system redesign.
Solution Approach 2:
The patent transforms the data processing approach from a single monolithic process to a multi-dimensional pipeline with different processing granularities. By operating at multiple time resolutions (hourly, daily, weekly aggregates) and processing dimensions simultaneously, the system can scale efficiently while maintaining reliable anomaly detection across diverse data volumes and rates.
3Measurement precision
If the system ingests and processes extensive historical data, then anomaly detection accuracy is improved, but processing time increases
Solution Approach 1:
The patent performs preliminary data aggregation and statistical computation in earlier pipeline stages, calculating summary statistics (means, standard deviations, percentiles) for historical data before it reaches the anomaly detection phase. This pre-computation allows the system to make accurate anomaly detections without reprocessing the entire historical dataset each time, significantly reducing processing time while maintaining accuracy.
Solution Approach 2:
The patent implements dynamic data processing where the system adjusts the amount of historical data processed based on current needs and available resources. The pipeline can dynamically select which time windows and data granularities to process, allowing fast responses for urgent anomalies while still leveraging historical context for accurate baseline comparison, thus balancing processing time and accuracy.
Data Source
AI summary
Some implementations of the disclosure provided a method including operations of obtaining a data set, performing feature extraction operations resulting to extract features according to the first time window, performing aggregation operations for each feature of the extracted features with historical features resulting in a set of aggregated features, performing feature engineering on the aggregated features on a per entity basis resulting in generation of set of feature vectors, performing an anomaly detection process on the set of feature vectors including providing the set of feature vectors as input to a machine learning model resulting in generation of a label for each feature vector of the set of features, and performing a remedial action determination process including performing a threshold comparison with each label and, responsive to satisfaction of the threshold comparison by a first label, causing performance of one or more remedial actions.