Cardinality Estimation Using Sample Frequency Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for estimating the number of distinct values (NDV) in large datasets are costly due to scanning and in-memory requirements, and existing estimators often fail to provide accurate results, especially with small sample sizes, leading to unrepresentative samples and poor error bounds.
Innovation Solution
The approach divides cardinality estimation into estimating high-frequency and low-frequency values using histogram normalization and singleton correction techniques, providing lower and upper bounds, and employing binomial probabilities to improve accuracy without increasing sample size.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the entire multiset is scanned to calculate NDV, then measurement precision is improved, but productivity deteriorates due to excessive scanning cost
Solution Approach 1:
The patent segments the NDV estimation problem into two distinct components: high-frequency values (appearing multiple times in the sample) and low-frequency values (appearing once or twice). This segmentation allows different estimation techniques to be applied to each segment, improving overall accuracy while maintaining efficiency. The high-frequency component is estimated using standard methods, while the low-frequency component uses correction techniques based on sample size and data distribution characteristics.
2Measurement precision
If in-memory structures are maintained to remember distinct values, then measurement precision is improved, but device complexity increases due to memory requirements
Solution Approach 1:
The patent extracts and removes the need for complex in-memory structures by deriving NDV estimates from sample data statistics alone. Instead of maintaining hash tables or other memory-intensive structures to track distinct values, the method uses mathematical formulas that operate on aggregated sample characteristics (frequency counts, sample size, data distribution parameters), thereby eliminating the need for complex memory structures while preserving estimation accuracy.
Solution Approach 2:
The patent replaces expensive, long-lived in-memory structures with cheap, disposable statistical calculations. Rather than maintaining persistent data structures throughout processing, the method uses lightweight statistical moments and frequency distributions that can be computed from samples and discarded after estimation, significantly reducing memory overhead and structure complexity.
3Productivity
If small samples are used for estimation, then productivity is improved, but measurement precision deteriorates due to sampling error
Solution Approach 1:
The patent changes the parameters used in estimation by incorporating corrections based on the frequency distribution of values in the sample. Specifically, it adjusts the estimation formulas to account for the expected number of low-frequency values that may be missed in small samples, using parameters such as sample size, observed frequency counts, and data distribution characteristics to compensate for sampling errors and improve accuracy.
4Measurement precision
If machine learning approaches are used for NDV estimation, then measurement precision may be improved, but device complexity increases due to training requirements
Solution Approach 1:
The patent enables the estimation system to be self-sufficient by deriving all necessary information directly from the sample data itself, without requiring external training datasets or pre-trained models. The method uses the sample's own frequency distribution and statistical characteristics to train and apply estimation formulas on-the-fly, eliminating the need for separate training phases, training data storage, and model management infrastructure.
Data Source
AI summary
Herein are quantitative analytics to increase the accuracy of cardinality estimation without increasing sample size. In an embodiment, a computer selects a few sample values from a multiset. A high-frequency exact count of distinct values that have at least a threshold amount of occurrences in the sample values is counted. A low-frequency exact count of distinct values in the sample that do not have at least the threshold amount of occurrences in the sample is counted. Based on multiple binomial probabilities, an upper bound of a count of missing distinct values in the multiset that are not in the sample is calculated. A total count of distinct values (NDV) in the multiset is estimated based on: a) the high-frequency exact count of distinct values, b) the low-frequency exact count of distinct values, and c) the upper bound of the count of missing distinct values in the multiset that are not in the sample.