LogLog Cardinality Estimation via Harmonic Mean
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional methods for determining cardinality in big data are inefficient, requiring excessive computing resources and time, and are inaccurate across the full spectrum of cardinality ranges, necessitating multiple operations to compensate for inaccuracies.
Innovation Solution
The implementation of a system that uses a compound data object processed with a single function to estimate the number of unique objects in a multiset, eliminating the need for subsequent processes by automatically adjusting based on the estimated number of unique objects, thereby reducing computing cycles and processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional linear-time methods are used to determine cardinality, then accuracy is maintained, but computing resources and processing time increase significantly
Solution Approach 1:
The patent replaces traditional mechanical linear-time comparison methods with a probabilistic algorithmic approach using hash functions and logarithmic counting. Instead of comparing every data element sequentially, the system uses hash-based fingerprinting and logarithmic moment calculations to estimate cardinality in sub-linear time, achieving both speed improvement and maintained accuracy
Solution Approach 2:
The patent transforms the cardinality estimation problem by changing the parameter representation from direct counting to logarithmic moment calculations. By computing logarithmic moments of the hash value distributions and using these transformed parameters in estimation formulas, the system achieves accurate cardinality estimates across the full spectrum of cardinality ranges with reduced computational complexity
2Productivity
If newer approximation methods are used to reduce computing resources, then processing time decreases, but accuracy deteriorates for low-range and mid-range cardinalities
Solution Approach 1:
The patent implements a dynamic estimation system that automatically adapts its calculation depth and methodology based on the detected cardinality range. The system performs initial rapid estimation, detects whether the cardinality falls in low-range, mid-range, or high-range categories, and then dynamically adjusts the processing level and formula selection to optimize accuracy for each specific range while maintaining overall efficiency
Solution Approach 2:
The patent creates a universal estimation framework that handles all cardinality ranges (low, mid, and high) through a single integrated algorithmic approach. The system uses a unified hash-based logging structure combined with logarithmic moment calculations that can accurately estimate cardinalities across the entire spectrum, eliminating the need for separate specialized methods for different ranges
3Measurement precision
If multiple successive operations are executed to compensate for inaccuracy, then accuracy improves for specific ranges, but computing cycles and processing time increase
Solution Approach 1:
The patent performs preliminary action by executing a single pass of hash computation and logarithmic moment calculation that captures sufficient statistical information for accurate estimation across all cardinality ranges. This preliminary computation of logarithmic moments from the hash value distribution provides enough data to directly estimate cardinality without requiring subsequent corrective operations, thereby eliminating time loss while maintaining accuracy
Data Source
AI summary
Systems and methods are disclosed for optimizing full-spectrum cardinality approximations on big data utilizing an optimized LogLog counting technique. To accomplish the foregoing, a multiset of objects that each corresponds to one of a plurality of objects associated with a resource are obtained. A compound data object is populated at least in part with data that is derived based on generated hash values that correspond to each object in the obtained multiset. The populated compound data object is processed with a full-spectrum harmonic mean estimation operation that can accurately determine a cardinality estimate for the obtained multiset using less resources and time when compared to traditional techniques. The determination is further made without the need to employ linear counting or bias correction operations on low or high cardinalities. An estimated number of unique objects in the obtained multiset is determined as a result of the processing, and subsequently provided for display or further manipulation.