Type-Dependent Event Deduplication for High-Throughput Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current event deduplication methods face challenges in accurately identifying and removing duplicate events in high-throughput systems, often leading to false positives and negatives, and are inefficient when dealing with multiple event types, especially in systems that rely on upstream data sources for uniqueness.
Innovation Solution
A method and system for type-dependent event deduplication that involves receiving a stream of events, determining their types, parsing them based on these types, and performing type-specific deduplication operations to identify and remove duplicates, utilizing a processor and non-transitory storage medium with executable code to configure these operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If generic event deduplication methods are used, then processing simplicity is maintained, but deduplication accuracy deteriorates due to false positives and negatives
Solution Approach 1:
The patent segments the deduplication process into distinct phases: event type determination, type-specific parsing, and type-dependent deduplication operations. Each event type (security events, application events, system events) receives customized processing rules, allowing high accuracy without overwhelming complexity through structured organization
Solution Approach 2:
The patent applies local quality by implementing different deduplication strategies tailored to specific event types. Each event type has its own parsing rules and deduplication criteria, ensuring that the processing method matches the characteristics of each event category, thereby improving accuracy without uniformly increasing complexity
2Productivity
If type-agnostic deduplication is used, then system simplicity is maintained, but processing efficiency deteriorates in high-throughput systems
Solution Approach 1:
The patent performs preliminary action by determining event types and applying type-specific parsing rules before the actual deduplication operation. This pre-processing step organizes events into categories with known deduplication strategies, enabling faster processing during high-throughput scenarios without requiring complex real-time analysis
Solution Approach 2:
The patent changes parameters by dynamically selecting different deduplication algorithms and comparison criteria based on event type. This allows the system to optimize processing efficiency for each event category while maintaining a unified framework, improving overall productivity without proportionally increasing system complexity
3Loss of energy
If duplicate events are not removed, then data completeness is maintained, but processing overhead increases
Solution Approach 1:
The patent implements feedback mechanisms by comparing events against type-specific deduplication criteria and using the results to determine whether to process or discard events. This feedback-driven approach ensures that only genuine duplicates are removed while maintaining data completeness for unique events, reducing processing overhead without losing important information
Data Source
AI summary
A system, method, and computer-readable medium are disclosed for performing a type-dependent event deduplication operation. The type-dependent event deduplication operation comprising: receiving a stream of events, the stream of events comprising a plurality of events, each event of the plurality of events having an associated event type; determining an event type of the plurality of events; parsing the plurality of events based upon the associated event type, the parsing providing a plurality of parsed events; and, performing a type-dependent event deduplication operation on the plurality of parsed events, the type-dependent event deduplication operation deduplicating events based upon the event type.