SIEM Database Deduplication via Event Record Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The management of Security Information and Event Management (SIEM) databases becomes increasingly costly and complex as enterprise environments grow, due to the challenge of efficiently identifying and removing duplicate event records, which impacts performance and resource utilization.
Innovation Solution
A method that allows for the identification and removal of duplicate event records in linear time with constant storage space, enabling simultaneous or parallel deduplication across multiple network devices, utilizing a deduplication module that maps event time and record number values to ordered pairs of device identifiers and event source names.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional deduplication methods are used to identify and remove duplicate event records, then duplicate records can be removed, but the process requires excessive storage space and time, impacting system performance
Solution Approach 1:
The patent divides the deduplication process into multiple independent threads, each handling a specific network device. This segmentation allows parallel processing of deduplication tasks across different devices without interfering with each other, reducing overall processing time and complexity while maintaining data quality.
Solution Approach 2:
The patent performs deduplication operations in advance before the event records are fully processed or stored. By identifying and removing duplicate records beforehand, the system prevents redundant data from accumulating, reducing storage requirements and improving subsequent processing efficiency.
2Reliability
If comprehensive deduplication is performed across all network devices, then data quality improves, but storage space requirements increase significantly
Solution Approach 1:
The patent segments the deduplication task by network device, creating independent threads for each device. This allows the system to process and store only the necessary deduplication metadata for each device separately, reducing overall storage requirements while maintaining comprehensive data quality across all devices.
Solution Approach 2:
The patent extracts and removes only the duplicate event records from the database while retaining unique records. By selectively removing only the redundant data rather than processing all data, the system reduces storage space requirements while improving data quality through the elimination of duplicates.
3Ease of operation
If duplicate removal is performed sequentially for each network device, then processing is simple, but total processing time increases significantly
Solution Approach 1:
The patent segments the deduplication process into multiple independent threads, one for each network device. Each thread can process its assigned device simultaneously without interfering with others, transforming sequential processing into parallel processing and significantly reducing total processing time while maintaining operational simplicity.
Solution Approach 2:
The patent implements a dynamic threading model where the number and allocation of threads can adapt to the number of network devices and processing requirements. This dynamic approach allows the system to optimize processing speed by creating multiple concurrent threads while maintaining ease of operation through automated thread management.
Data Source
AI summary
A method comprises receiving information characterizing events from respective ones of a plurality of network devices each comprising one or more event sources, storing event records in a security information and event management database with each event record corresponding to a given event and comprising a device identifier, an event source name, an event time and an event record number, obtaining a set of event records from the security information and event management database for a specified network device in a specified time range, identifying whether respective ones of the event records in the set are duplicate event records based at least in part on mappings of event time and event record number values to ordered pairs of device identifier and event source name values, and removing event records in the set identified as duplicate event records from the security information and event management database.