Event Deduplication Using Hash Sets and Bloom Filters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing monitoring systems face inaccuracies in counting events due to multiple occurrences of the same event meeting metric definitions, leading to double counting.
Innovation Solution
A system utilizing stateful workloads to assign unique identifiers to events satisfying metric definitions, employing a hash set for initial deduplication and a scalable Bloom filter for handling larger volumes of unique identifiers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If events are counted based on metric definitions without deduplication, then event detection is simple and fast, but measurement precision deteriorates due to double counting
Solution Approach 1:
The system segments the deduplication process into multiple stages: first using a hash set for exact matching of event signatures, then using a Bloom filter for probabilistic deduplication. This multi-stage segmentation allows the system to achieve high measurement precision while managing complexity through hierarchical processing
Solution Approach 2:
The patent introduces an intermediary deduplication layer between event ingestion and metric calculation. This intermediary component assigns unique signatures to events and filters duplicates before they reach the metric computation logic, thereby improving measurement precision without requiring changes to the core metric definition system
2Measurement precision
If a hash set is used for deduplication, then measurement precision improves, but loss of substance worsens due to high memory consumption
Solution Approach 1:
The deduplication storage is segmented into two parts: a small hash set for exact matches and a larger Bloom filter for probabilistic filtering. This segmentation allows the system to use minimal memory for the critical exact matching function while using the more memory-efficient Bloom filter structure for the bulk of deduplication operations
Solution Approach 2:
The system uses a Bloom filter, which is a space-efficient probabilistic data structure that trades a small amount of false positive rate for dramatically reduced memory consumption compared to a full hash set. This allows the system to handle large volumes of events with limited memory resources
3Productivity
If cloud scale event volumes are processed, then productivity increases, but measurement precision deteriorates due to deduplication challenges
Solution Approach 1:
The system performs preliminary deduplication by assigning unique signatures to events and checking against the hash set and Bloom filter before events are fully processed and stored. This preliminary action removes duplicates early in the pipeline, ensuring that subsequent processing operates on deduplicated data without requiring re-processing
Solution Approach 2:
An intermediary deduplication service layer is introduced between the high-throughput event ingestion system and the metric computation system. This intermediary handles the computationally intensive signature generation and deduplication checks, allowing the core systems to maintain high productivity while measurement precision is preserved through accurate duplicate detection
Data Source
AI summary
The technology disclosed herein provides a mechanism to avoid double counting when generating metrics from monitoring events received from an endpoint (e.g., end user's web browser, or mobile application). A unique identifier is assigned to each monitoring event matching a metric definition. When the number of unique identifiers assigned is below a predetermined threshold, a deduplication system determines whether the unique identifier for a given monitoring event is duplicative of any other unique identifiers, corresponding to other monitoring events, using a hash set. When the number of unique identifiers exceeds the predetermined threshold, the unique identifiers are automatically added to a probabilistic data structure, such as a scalable Bloom filter. In this scenario, the deduplication system would determine whether the unique identifier for the given monitoring event is duplicative of any other identifiers in the probabilistic data structure.