Bloom-Filtered Security Data Lake for Threat Event Deduplication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Enterprise networks face challenges in efficiently monitoring and identifying sophisticated security threats due to their complexity, requiring improved techniques for recording, deduplicating, and investigating security events.
Innovation Solution
A filtered data lake for enterprise security is created using an asynchronous stream of security events, employing deduplication with bloom filters and metadata augmentation to facilitate monitoring and investigation, along with live threat discovery through query usage analysis and data augmentation for improved correlation and visualization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional security event monitoring is used in complex enterprise networks, then comprehensive security coverage is achieved, but system complexity and difficulty in identifying threats increase
Solution Approach 1:
The patent segments security event data into structured schemas with standardized fields (source, destination, event type, timestamp, etc.), dividing the complex monitoring task into manageable, organized components that can be efficiently processed and analyzed
Solution Approach 2:
The patent introduces an intermediary processing layer that receives events from multiple data recorders, deduplicates them using bloom filters, enriches them with metadata, and presents a consolidated view to security analysts, simplifying the complexity of monitoring across distributed endpoints
2Loss of information
If all security events are retained in the data lake, then complete investigation capability is maintained, but storage requirements and processing overhead increase
Solution Approach 1:
The patent performs preliminary deduplication using bloom filters before events are fully stored in the data lake, proactively eliminating duplicate events from multiple recorders and reducing the total data volume that needs to be retained for complete investigation
Solution Approach 2:
The patent transforms raw security events into standardized schemas with consistent parameter structures, enabling efficient storage, indexing, and querying while maintaining complete investigative capability through structured metadata organization
3Productivity
If duplicate security events are not filtered, then all events are available for analysis, but monitoring efficiency and threat identification speed decrease
Solution Approach 1:
The patent applies bloom filter-based deduplication as a preliminary filtering step before events reach the analysis phase, efficiently identifying and removing duplicates while preserving all unique security events for complete threat investigation
Solution Approach 2:
The patent replaces traditional mechanical deduplication methods with bloom filter-based probabilistic filtering, providing faster and more scalable duplicate detection that maintains event completeness while significantly improving processing efficiency
4Adaptability or versatility
If no metadata augmentation is applied, then data storage remains simple, but correlation and visualization capabilities are limited
Solution Approach 1:
The patent creates universal metadata schemas that can represent diverse security events from different sources and types using a common structure, enabling correlated analysis across heterogeneous data while maintaining manageable complexity through standardized field definitions
Data Source
AI summary
A data lake for enterprise security is created from an asynchronous stream of security events by deduplicating objects and creating metadata related to downstream security functions. Deduplication of objects may be efficiently performed with a bloom filter as objects are ingested into the data lake. The objects may also be augmented with metadata arranged in schemas to facilitate monitoring and use within the data lake.