Security Alert Compression via Redundancy Elimination
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computer security systems face bottlenecks in processing and transmitting large volumes of unstructured raw data from multiple monitoring devices, leading to delayed detection of potential threats due to redundant data logs being processed alongside non-redundant ones, which increases the load on the system and requires scaling event managers, causing compatibility issues.
Innovation Solution
A method and system that identifies redundant raw alerts within movable time windows using digital signatures and meta-definition databases, marking and removing redundant alerts while aggregating, compressing, and encrypting non-redundant ones, thereby reducing system load and enabling efficient processing of unique alerts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the number of event managers is scaled according to the number of monitoring devices, then the system can handle larger volumes of data, but compatibility issues arise and system complexity increases
Solution Approach 1:
The patent combines multiple event manager functions into a single centralized system that uses distributed processing and metadata-based routing. Instead of scaling out multiple event managers, the system merges their capabilities into one platform that can handle data from any number of monitoring devices through a unified architecture, thereby maintaining high productivity while reducing system complexity.
Solution Approach 2:
The event manager is designed with universal compatibility to work with various types of monitoring devices and data formats. The system uses metadata schemas and adaptable processing pipelines that can accommodate different device types without requiring device-specific event managers, thus achieving high data processing capacity with a single multi-functional system rather than multiple specialized systems.
2Reliability
If all raw alerts are processed and transmitted, then no critical threats are missed, but system load increases and processing speed decreases
Solution Approach 1:
The system performs preliminary filtering and classification of raw alerts before full processing using metadata extraction and digital signature matching. By pre-identifying redundant alerts through comparison against known patterns and metadata schemas, the system prepares data in advance, ensuring critical threats are captured while reducing the volume of data requiring intensive processing, thus maintaining reliability while improving processing speed.
Solution Approach 2:
The system applies partial processing to all alerts (metadata extraction and signature matching) and full processing only to non-redundant alerts. This selective approach processes more alerts than strictly necessary (excessive action on metadata level) to ensure no threat is missed, while avoiding excessive full processing that would slow down the system, thereby balancing reliability and productivity.
3Productivity
If redundant alerts are identified and removed, then system load is reduced and processing efficiency improves, but the complexity of data processing increases
Solution Approach 1:
The system creates simplified copies of alert data in the form of metadata extracts and digital signatures that can be quickly compared for redundancy detection. Instead of processing entire raw alerts, the system works with these lightweight copies for identification purposes, then processes only the necessary portions of original alerts. This copying approach improves processing efficiency while managing complexity by separating identification from detailed analysis.
Solution Approach 2:
The system transforms raw alert data into different parameter representations through metadata extraction, creating structured summaries that capture essential characteristics without preserving all original data. By changing the parameters from complete raw data to condensed metadata representations, the system enables efficient redundancy detection and reduces processing complexity while maintaining the ability to identify unique threats.
4Loss of energy
If data is compressed and optimized before transmission, then transmission efficiency improves and bandwidth usage reduces, but processing time before transmission increases
Solution Approach 1:
The system performs compression and optimization as preliminary actions immediately after alert identification and before transmission to the command centre. By preparing data in advance with metadata extraction, redundancy removal, and compression, the system reduces the burden on transmission channels and ensures data is ready for efficient transfer, thereby reducing bandwidth usage while managing processing time through proactive preparation.
Solution Approach 2:
The system uses periodic batching of alert processing and transmission, accumulating alerts over defined time periods before compressing and transmitting them collectively. This periodic approach allows the system to spread processing load over time rather than compressing every alert individually in real-time, reducing overall processing time while still achieving bandwidth efficiency through batched compression and transmission.
Data Source
Figure 1
Figure 2a~2b
Figure 3
AI summary
This document discloses a method and system for just-in-time compression and optimization of raw unstructured in-line and in-transit data by identifying low entropy data blocks or duplicated information security information in raw computer security alerts within a series of time windows. In particular, the method and system automatically manages; processes; and optimizes in-line and in-transit data blocks or raw information security alerts received from a plurality of information surveillance sources and/or peripheral monitoring devices simultaneously. The data blocks or raw information security alerts that are found to be unique in the various time windows are transposed into meta-definition tables to be further processed while redundant data blocks or raw alerts contained within each particular time window are identified, marked and processed accordingly.