Geometric Hashing for Memory-Efficient Cardinality Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for determining the number of unique visitors to high-traffic web pages are impractical due to memory and processing requirements, as maintaining a list of individual identifiers becomes too large and processor-intensive.
Innovation Solution
A method using hash logic to generate hash values conforming to a geometric distribution, setting bit positions in an array based on these values, and estimating the number of unique elements from the set bits, reducing memory and processing needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a list of individual identifiers is maintained to determine the number of unique visitors, then the accuracy of unique visitor count is improved, but the memory requirements and processing complexity increase significantly for high-traffic web pages
Solution Approach 1:
The patent uses hash values (copies) of individual identifiers instead of the identifiers themselves. Each identifier is transformed into a hash value that conforms to a geometric distribution, allowing the system to work with compact numerical representations rather than storing full identifier lists, thereby reducing memory requirements while preserving the ability to estimate unique visitor counts
Solution Approach 2:
The patent changes the parameter representation from storing complete identifier data to storing only the position of the lowest-order set bit in hash values. This parameter transformation converts complex identifier storage into simple integer storage (bit positions), dramatically reducing the quantity of data needed while maintaining estimation accuracy through the geometric distribution property
2Measurement precision
If a list of individual identifiers is maintained to determine the number of unique visitors, then the accuracy of unique visitor count is improved, but the processing complexity increases significantly for high-traffic web pages
Solution Approach 1:
The patent extracts only the essential information needed for counting (the position of the lowest-order set bit in hash values) while discarding redundant data. This extraction approach simplifies processing by working with minimal necessary information (bit positions) rather than complete identifier lists, reducing computational complexity while maintaining measurement precision
Solution Approach 2:
The patent uses temporary hash values and bit position representations as disposable intermediaries in the counting process. These short-lived computational objects (hash values conforming to geometric distribution) are generated, processed to extract bit positions, and then discarded, replacing the need for persistent complex data structures and reducing overall processing complexity
Data Source
AI summary
A computer-implemented method for efficiently estimating the number of unique elements in a collection of elements comprises generating, via hash logic, hash values associated with the elements. The hash values specify bit positions within an array of bits. Hash values output from the hash logic conform to a geometric distribution such that bit positions of the array of bits corresponding to lower orders bits are more likely to be generated than bit positions corresponding to higher-order bits. Bits of the array of bits corresponding to the bit positions are set. The number of bits of the array of bits that are set is counted. Estimation logic estimates the number of unique elements of the collection of elements as a function of the number of bits of the array of bits that are set.