Probabilistic Sketch Registers for Memory-Efficient Cardinality Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing probabilistic data structures for estimating cardinalities in distributed data streams, such as HyperLogLog, are memory-intensive and inefficient for modern applications with high traffic demands, necessitating a more memory-efficient solution that maintains idempotency, commutativity, and mergeability.
Innovation Solution
A probabilistic sketch data structure that uses a stepwise approximated geometric distribution for sketch updates, combining a geometrically distributed random value with a uniformly distributed random value to create an efficient, idempotent, and commutative estimation method requiring only basic arithmetic and bit shift operations, and a Max Likelihood evaluation approach.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If HyperLogLog or known estimation methods are used to support large estimation ranges, then the estimation range is sufficient, but the memory footprint becomes too large and inefficient
Solution Approach 1:
The sketch register is divided into two sections: a first section storing the highest observed update value and a second section storing flag bits indicating observation of next lower update values. This segmentation allows compact representation of multiple observed values without requiring proportional memory increase, resolving the contradiction between estimation range and memory footprint.
Solution Approach 2:
The patent changes the parameter representation from storing complete update values to storing flag bits that indicate the presence of next lower values. This parameter transformation enables the system to track multiple observed values (n+1 values) using minimal additional memory beyond the primary update value, achieving efficient memory utilization while maintaining large estimation range capability.
2Loss of information
If conventional approaches store only one update value per sketch register, then the memory usage is simple, but information about multiple observed update values is lost
Solution Approach 1:
Instead of storing all possible update values (which would require excessive memory), the patent stores the highest observed value and uses flag bits to indicate the presence of a limited number of next lower values. This partial action approach captures sufficient information for accurate cardinality estimation while avoiding the memory overhead of storing complete value histories.
Solution Approach 2:
Flag bits serve as intermediaries that convey information about observed update values without requiring direct storage of those values. The flag bits act as a compact representation mechanism, allowing the system to retain information about multiple observed values while using minimal memory resources.
3Quantity of substance
If the patent stores n+1 observed update values in a single register using flag bits, then memory efficiency improves, but the device complexity increases
Solution Approach 1:
The patent adds a new dimension to the sketch register by dividing it into multiple sections (first section for highest value, second section for flag bits). This dimensional transformation allows the register to encode multiple pieces of information (highest value plus indications of lower values) within the same memory space, achieving improved memory efficiency while managing complexity through structured organization.
Data Source
AI summary
Methods and technologies are disclosed for the sketch efficient estimation of large-scale multi-sets in distributed, stream-oriented environments. Sketch updates are idempotent and commutative, to support duplicate set elements and varying element sequences. They are also mergeable to support distributed sketch recording. The recording process uses stepwise approximated geometric distributions, efficiently generated from NLZ values of received sketch updates, by using only multiplications with powers of two and integer additions. Sketch registers are subdivided into a portion storing an observed max update value for the register, and a portion storing set of flag bits indicating observed next smaller update values for the register. A Max Likelihood base sketch data evaluation, based on the assumption of statistically independent sketch registers is proposed. The limited number of different probabilities created by the stepwise approximated geometric distributions leads to a Max Likelihood function with coefficients that can be calculated solely with integer arithmetic.