HBase Key Generation for Unique Event Counting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Determining unique events from a stream of events in social networking systems is challenging due to repeated occurrences of the same event, which incurs high performance costs.
Innovation Solution
A database management system with an event ingestion unit, unique events query unit, and set definition unit that generates an HBase key for each unique event, ensuring only single occurrences are recorded in a set table, allowing for efficient querying of unique event counts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the system counts all events in a stream to determine unique events, then the accuracy of unique event counts is improved, but the performance cost increases significantly
Solution Approach 1:
The patent segments the event processing task by dividing the stream of events into manageable units and using a distributed computing architecture. The event stream is partitioned across multiple nodes, with each node processing a subset of events independently. This segmentation allows the system to accurately count unique events while distributing the computational burden, thereby maintaining measurement precision without overwhelming single-point performance bottlenecks.
Solution Approach 2:
The patent introduces an intermediary data structure (such as a Bloom filter or hash-based deduplication layer) that sits between the raw event stream and the counting mechanism. This intermediary pre-processes events to identify and eliminate duplicates before they reach the final counting stage, significantly reducing the computational work required while preserving the accuracy of unique event counts.
2Reliability
If the system processes repeated events to ensure accurate unique counts, then the reliability of event data is improved, but the processing time increases
Solution Approach 1:
The patent applies preliminary action by pre-processing the event stream to identify and mark duplicate events before the main counting operation. Techniques such as hashing event identifiers, maintaining sliding window buffers, or using probabilistic data structures are employed in advance to flag repeated events. This preliminary deduplication ensures reliable unique event identification while minimizing the time required during the actual counting phase.
3Quantity of substance
If the system stores all event occurrences to determine uniqueness, then the completeness of event data is improved, but the storage requirements increase
Solution Approach 1:
The patent employs disposable, ephemeral data structures such as in-memory hash sets, temporary buffers, or session-based storage that are created, used, and discarded within specific time windows or processing batches. These short-living storage objects allow the system to track unique events completely within their scope without requiring persistent long-term storage, thereby maintaining data completeness while minimizing overall storage requirements.
Solution Approach 2:
The patent changes storage parameters by using compact data representations such as bitmaps, hashed identifiers, or compressed formats instead of storing full event objects. By transforming the representation of event data into more space-efficient forms, the system maintains the ability to determine uniqueness accurately while significantly reducing the volume of storage required.
Data Source
AI summary
A method and a system for determining unique counts of events from a stream of events are described. One or more sets of events are defined. A set is defined based on one or more parameters including a set type and zero or more set qualifiers. When an event occurs information related to the event is obtained and used to create a Hbase row key. The Hbase row key includes the set type, associated set qualifiers and a set member qualifier. The Hbase row key is added to the Hbase table when the Hbase row key is not already included in the Hbase table.