Distinct Value Estimation via Probabilistic Sampling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distinct value estimation techniques in database technology are impractical for large datasets due to high computational resource requirements and inaccuracy, especially when dealing with ever-changing data, as they often need to scan all values or require excessive memory.
Innovation Solution
A probabilistic approach that uses multiple intermediate distinct value estimates based on varying samples, plots these estimates against sample size, fits a function, and extrapolates to estimate the total number of distinct values, allowing for accurate estimation with bounded memory and reduced processing, accommodating additions and deletions without reprocessing all data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional distinct value estimation techniques are used to scan all values in large datasets, then measurement precision is improved, but use of energy and computational resources increase excessively
Solution Approach 1:
The patent applies partial action by scanning only a sample of values rather than all values in the dataset. The query planner selects a sample size that provides sufficient accuracy for query optimization without requiring full dataset scanning, thus reducing computational resource consumption while maintaining adequate measurement precision for distinct value estimation.
Solution Approach 2:
The patent changes the parameter of sample size to balance between estimation accuracy and computational cost. By adjusting the sample size parameter, the system can achieve acceptable distinct value estimates with reduced scanning effort, resolving the contradiction between precision and energy consumption.
2Measurement precision
If traditional distinct value estimation techniques scan all data values, then measurement precision is improved, but loss of time increases due to processing all data
Solution Approach 1:
The patent implements partial action by processing only a sample of data values rather than the complete dataset. This allows the query planner to obtain sufficient distinct value statistics for optimization without the time cost of scanning all data, thereby reducing processing time while maintaining adequate measurement precision.
Solution Approach 2:
The patent applies preliminary action by pre-computing distinct value estimates on sampled data before query execution. This preliminary estimation allows the query planner to make optimization decisions without waiting for full dataset processing, reducing the time loss during query execution.
3Measurement precision
If traditional distinct value estimation is performed on ever-changing data, then measurement precision is maintained, but device complexity increases due to need to reprocess all data
Solution Approach 1:
The patent applies partial action by maintaining distinct value estimates on sampled data rather than requiring reprocessing of all data when changes occur. This allows the system to update statistics with reduced complexity by only considering changes in the sample or using incremental updates, maintaining measurement precision without full reprocessing.
4Measurement precision
If traditional distinct value estimation techniques are used, then measurement precision is improved, but memory requirements increase excessively
Solution Approach 1:
The patent applies partial action by storing and processing only a sample of data values rather than the complete dataset. This significantly reduces the quantity of data that must be held in memory during distinct value estimation, allowing adequate precision to be achieved with minimal memory requirements.
Data Source
AI summary
The problem of distinct value estimation has many applications, but is particularly important in the field of database technology where such information is utilized by query planners to generate and optimize query plans. Introduced is a novel technique for estimating the number of distinct values in a given dataset without scanning all of the values in the dataset. In an example embodiment, the introduced technique includes gathering multiple intermediate probabilistic estimates based on varying samples of the dataset, 2) plotting the multiple intermediate probabilistic estimates against indications of sample size, 3) fitting a function to the plotted data points, and 4) determining an overall distinct value estimate by extrapolating the objective function to an estimated or known total number of values in the dataset.