Ensemble ML Anomaly Detection for Scalable Network Visibility
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing security systems struggle with scalability and network visibility, as security appliances and installed software products fail to effectively detect malicious activities across network segments and require time-consuming upgrades or swaps during traffic increases, and lack the ability to utilize contextual information from other segments.
Innovation Solution
A distributed data processing system employing machine learning techniques for real-time and batch processing of network events, enabling scalable and comprehensive anomaly detection and threat identification by analyzing user and entity behaviors across the entire network, without relying on pre-existing signatures or rules.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If security appliances are deployed to monitor network traffic, then security detection capability is improved, but scalability deteriorates due to time-consuming appliance swaps or upgrades during traffic increases
Solution Approach 1:
The patent replaces physical security appliances with a software-based distributed system that uses machine learning models running on standard computing infrastructure. This substitution eliminates the need for hardware appliance swaps during scaling events, allowing the system to dynamically adapt to traffic increases through software deployment rather than mechanical hardware replacement.
Solution Approach 2:
The system divides the security monitoring function into distributed machine learning models that operate independently across multiple network segments. Each model processes local traffic patterns while contributing to a collective security posture, enabling scalable deployment without requiring centralized appliance management or coordination during scaling operations.
2Measurement precision
If security appliances are deployed at specific network locations, then localized security monitoring is improved, but network visibility deteriorates due to limited awareness of activities on other network segments
Solution Approach 1:
The distributed machine learning system performs multiple functions simultaneously: each local model provides precise localized monitoring of its network segment while also contributing to global network visibility by sharing patterns and anomalies. This multi-functionality eliminates the information loss inherent in isolated appliance deployments, as the system maintains both local precision and global awareness through its distributed architecture.
Solution Approach 2:
The system merges localized detection capabilities with centralized pattern analysis by combining data from multiple distributed models. The ensemble machine learning approach integrates findings from individual segment models to create a comprehensive view of network-wide threats, thereby combining the benefits of localized precision monitoring with global information awareness.
3Ease of manufacture
If traditional security systems are used, then implementation simplicity is improved, but detection capability deteriorates due to inability to detect unknown threats and insider attacks
Solution Approach 1:
The patent replaces rule-based security systems with machine learning models that automatically adapt to detect unknown threats. The system maintains implementation simplicity by using standardized computing infrastructure and automated model training processes, while significantly improving detection capability through the ability to identify novel attack patterns and insider threats that predefined rules cannot detect.
Solution Approach 2:
The system dynamically changes detection parameters by continuously training machine learning models on evolving network traffic patterns. This allows the system to adapt its detection criteria based on learned behavior rather than static rules, improving reliability in detecting unknown threats while maintaining ease of implementation through automated parameter optimization rather than manual rule configuration.
Data Source
AI summary
A security platform employs a variety techniques and mechanisms to detect security related anomalies and threats in a computer network environment. The security platform is “big data” driven and employs machine learning to perform security analytics. The security platform performs user/entity behavioral analytics (UEBA) to detect the security related anomalies and threats, regardless of whether such anomalies/threats were previously known. The security platform can include both real-time and batch paths/modes for detecting anomalies and threats. By visually presenting analytical results scored with risk ratings and supporting evidence, the security platform enables network security administrators to respond to a detected anomaly or threat, and to take action promptly.