Enterprise Threat Event Augmentation With Bloom Filter Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Enterprise networks face challenges in efficiently monitoring and investigating sophisticated security threats due to their complexity, requiring improved techniques for recording, deduplicating, and analyzing security events.
Innovation Solution
A data lake system is created for enterprise security, utilizing an asynchronous stream of security events, with deduplication and metadata augmentation through bloom filters, and schema organization to facilitate monitoring and analysis, along with endpoint instrumentation for enhanced threat detection and visualization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a data lake system stores all security events from enterprise networks, then the quantity of security data is increased, but the complexity of monitoring and investigating threats increases
Solution Approach 1:
The patent segments security event data by organizing it into multiple data lakes according to different schemas and security functions. Each data lake contains specific types of security events organized by particular schemas, allowing the system to divide the large volume of security data into manageable, organized segments that can be monitored and investigated more efficiently
Solution Approach 2:
The patent introduces bloom filters as an intermediary mechanism between raw security events and the data lake storage system. These probabilistic data structures enable efficient deduplication by quickly identifying potential duplicates before full storage, reducing the actual quantity of unique security events that need to be stored and processed while maintaining data integrity
2Reliability
If security events are deduplicated using traditional methods, then duplicate removal is achieved, but the processing time and computational resources increase
Solution Approach 1:
The patent applies bloom filters as a preliminary filtering step before performing full deduplication checks. The bloom filter provides a probabilistic but fast preliminary assessment of whether an event is a duplicate, allowing the system to quickly eliminate obvious duplicates without performing computationally expensive full comparisons, thus reducing overall processing time while maintaining high deduplication accuracy
Solution Approach 2:
The patent replaces traditional mechanical deduplication methods (such as exact matching algorithms and full data comparisons) with probabilistic bloom filter-based detection. This substitution uses probabilistic mathematics rather than deterministic mechanical comparison, dramatically reducing computational complexity and processing time while maintaining acceptable accuracy for security event deduplication
3Ease of operation
If security events are organized with multiple schemas and metadata, then the ease of monitoring and analysis is improved, but the device complexity increases
Solution Approach 1:
The patent applies different schemas to different data lakes based on their specific security functions and monitoring requirements. Rather than imposing a single complex universal schema, each data lake is organized with the specific schema and metadata structure that best suits its particular security monitoring purpose, making each individual data lake easier to monitor and analyze while the overall system remains manageable through this localized approach
Data Source
AI summary
An endpoint in an enterprise network is instrumented with sensors to detect security-related events occurring on the endpoint. Event data from these sensors is augmented with contextual information about, e.g., a source of each event in order to facilitate improved correlation, analysis, and visualization at a threat management facility for the enterprise network.