Anomaly Detection via Event Aggregation and ML Ensembles
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current User and Entity Behavior Analysis (UEBA) systems face challenges in scalability for data collection, storage, analysis, and visualization, particularly in large networks where processing large amounts of event data can lead to delays in detecting anomalous entities.
Innovation Solution
A scalable framework is provided for timely online analysis of event data, utilizing event data aggregation and ensemble machine-learning models across different analytics modules to detect anomalous entities, with features extracted and processed in real-time to enhance flexibility and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional UEBA systems process large amounts of event data in real-time, then detection accuracy is maintained, but processing delays increase and scalability deteriorates
Solution Approach 1:
The system segments the monolithic data processing pipeline into multiple specialized microservices: event ingestion service, feature extraction service, anomaly detection service, and visualization service. Each service processes specific aspects of the data independently, enabling parallel processing that maintains detection accuracy while reducing overall processing time and improving scalability.
Solution Approach 2:
The system transitions from processing individual events sequentially to aggregating events into time-windowed buckets across multiple dimensions (time, entity type, event category). This dimensional transformation enables batch processing optimizations while preserving the temporal patterns necessary for accurate anomaly detection through ensemble machine learning models.
2Reliability
If UEBA systems aggregate all event data centrally for analysis, then comprehensive detection is achieved, but system scalability and performance deteriorate under network growth
Solution Approach 1:
The system divides the centralized data processing architecture into distributed microservices deployed across multiple nodes. Each microservice handles a specific processing function independently, allowing the system to scale horizontally by adding more service instances without requiring centralized processing of all data, thus maintaining detection comprehensiveness while improving scalability.
Solution Approach 2:
The system introduces event brokers and message queues as intermediaries between data sources and analysis services. These intermediaries buffer and route events asynchronously, decoupling the ingestion rate from processing rate and enabling the system to handle variable data volumes efficiently while maintaining comprehensive detection coverage.
3Measurement precision
If complex feature extraction and analysis is performed on all event data, then detection accuracy improves, but processing time and computational resources increase
Solution Approach 1:
The system applies feature extraction and anomaly detection selectively rather than uniformly to all events. It uses filtering mechanisms to identify high-risk event patterns that warrant detailed analysis, applying complex processing only to relevant subsets of data. This partial action approach maintains high detection accuracy for critical anomalies while improving overall processing throughput by avoiding unnecessary computation on normal events.
Data Source
AI summary
In some examples, a system extracts features from event data representing events in a computing environment, trains ensembles of machine-learning models for respective analytics modules of a plurality of different types of analytics modules, and detects, by the different types of analytics modules using the respective trained ensembles of machine-learning models, an anomalous entity in response to further event data.