Threat Analytics System Using Bloom Filters for Event Aggregation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Cyber-threat detection systems face issues with false positives and negatives due to incorrect information in threat intelligence feeds, leading to unnecessary investigatory burdens on online system operators.
Innovation Solution
A threat analytics system that aggregates event information using space-efficient data structures like Bloom filters, compares it with threat indicators, and reports red flag events while validating threat indicators to reduce false positives and negatives.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If threat intelligence feeds are used to detect cyber-threats, then threat detection capability is improved, but false positives and false negatives increase due to incorrect information
Solution Approach 1:
The system implements feedback mechanisms by continuously monitoring the accuracy of threat intelligence feeds and adjusting their usage accordingly. When a feed demonstrates high false positive rates, the system automatically reduces its reliance on that feed or adjusts weighting parameters, thereby maintaining reliable threat detection while minimizing accuracy degradation from incorrect information.
Solution Approach 2:
The system combines multiple threat intelligence feeds with different characteristics and reliability profiles into a composite threat detection approach. By aggregating information from diverse sources and applying weighted analysis, the system achieves more accurate threat detection than any single feed could provide alone, effectively resolving the contradiction between detection capability and accuracy.
2Measurement precision
If comprehensive event information is collected and analyzed, then threat detection accuracy is improved, but processing time and computational resources increase
Solution Approach 1:
The system segments event information into distinct categories and processing streams, allowing parallel analysis of different event types. By dividing the comprehensive event data into manageable segments that can be processed simultaneously through specialized analysis pipelines, the system maintains high detection accuracy while reducing overall processing time through concurrent operations.
Solution Approach 2:
The system performs preliminary filtering, aggregation, and preprocessing of event information before detailed threat analysis. By preparing and organizing event data in advance using efficient data structures, the system reduces the computational burden during actual threat detection, thereby maintaining accuracy while significantly reducing processing time for comprehensive analysis.
3Ease of manufacture
If traditional data structures are used to store event information, then ease of implementation is maintained, but space efficiency and processing speed deteriorate
Solution Approach 1:
The system changes the parameter representation of event data by implementing compact encoded formats and optimized data structures that store information more efficiently. By transforming event parameters into space-efficient representations while maintaining full analytical capability, the system achieves faster processing speeds and reduced memory usage without sacrificing ease of implementation, as the encoding schemes are designed to be transparent to existing analysis logic.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A security monitoring system operated by a downstream client continually collects event information indicating events that have occurred within the computing environment of the downstream client. The monitoring system, using software provided by a threat analytics system, aggregates the event information into a secure and space efficient data structure. The monitoring system transmits the data structures storing event information to the threat analytics system for further processing. The threat analytics system also receives threat indicators from intelligence feed data sources. The threat analytics system compares the event information received from each security monitoring system against the threat indicators collected from the intelligence feed data sources to identify red flag events. The threat analytics system processes the event information to synthesize all information related to the red flag event and reports the red flag event to the downstream client.