Hash Sketch Cardinality Estimation for Memory-Limited Counting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for determining the number of unique visitors to high-traffic web pages are impractical due to memory and processing requirements, as maintaining a list of individual identifiers becomes excessively large and processor-intensive.
Innovation Solution
A method and system using hash logic to generate hash values, specify discrete statistical values in a sketch-frequency table, and estimate cardinality based on this table, reducing the need for extensive memory and processing by iteratively updating cardinality estimates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a list of individual identifiers is maintained to determine the number of unique visitors, then the accuracy of cardinality measurement is improved, but the memory requirements and processing complexity increase excessively for high-traffic web pages
Solution Approach 1:
The patent uses hash values as simplified copies of individual identifiers. Instead of storing the actual identifiers (cookies, IP addresses), the system stores only their hash representations, which are much more compact and can be aggregated efficiently in the sketch-frequency table while preserving the ability to estimate unique element counts
Solution Approach 2:
The patent transforms the cardinality measurement problem from exact counting to probabilistic estimation by changing the parameter from discrete individual identifiers to continuous hash value distributions. The sketch-frequency table captures the distribution of hash values and their frequencies, allowing cardinality estimation through statistical analysis rather than direct enumeration
2Measurement precision
If a list of individual identifiers is maintained to determine the number of unique visitors, then the accuracy of cardinality measurement is improved, but the processing complexity and time requirements increase excessively for high-traffic web pages
Solution Approach 1:
The patent segments the processing into two distinct phases: (1) hash computation phase where each identifier is transformed into a hash value and the sketch-frequency table is updated, and (2) estimation phase where the cardinality is computed from the pre-processed table. This segmentation allows the heavy lifting to be done incrementally during data ingestion rather than requiring complex real-time processing
Solution Approach 2:
The patent performs preliminary hashing and frequency counting during data ingestion, building the sketch-frequency table incrementally as elements arrive. This preliminary action transforms the raw data into a pre-processed statistical representation that can be queried efficiently later, avoiding the need for complex retrospective analysis
Data Source
AI summary
A computer-implemented method for efficiently estimating the number of unique elements in a collection of elements comprises generating, via hash logic, hash values for each element of the collection of elements. The method further comprises specifying, in a sketch-frequency table, a set of discrete statistical values associated with the hash values and, for each discrete statistical value of the set of discrete statistical values, information indicative of a frequency at which binary representations of the hash values are associated with the discrete statistical value. The cardinality of the collection of elements is estimated based on the sketch-frequency table.