Quantile Computation via Dynamic Bin Splitting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computing exact quantiles for distributed data systems and big data environments is challenging due to the large volume of unsorted data, which prevents efficient sorting and leads to issues like non-convergence and high memory usage.
Innovation Solution
A method that computes maximum and minimum values, bin values, and frequency counters for unsorted data, allowing for efficient quantile computation by identifying the bin where the quantile lies, and dynamically adjusts the number of bins to manage memory usage, thereby avoiding expensive sorting and non-convergence.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If sorting is used to compute exact quantiles, then measurement precision is improved, but productivity deteriorates due to high computational cost and inability to handle distributed big data
Solution Approach 1:
The data range is segmented into multiple bins based on maximum and minimum values. Frequency counters are computed for each bin independently, allowing parallel processing and avoiding the need to sort the entire dataset. This segmentation enables exact quantile computation through selective bin analysis rather than full sorting.
2Measurement precision
If the number of bins is increased to improve quantile precision, then measurement precision is improved, but memory usage increases
Solution Approach 1:
The bin configuration is made dynamic through an adaptive refinement process. The system initially uses a coarse binning scheme with fewer bins to minimize memory usage. Based on the computed frequency counters and quantile requirements, the system dynamically refines only the necessary bins, increasing precision where needed while maintaining memory efficiency elsewhere.
Solution Approach 2:
Different bins are treated with different levels of refinement based on local data characteristics. Bins containing quantile values require higher precision and are subdivided, while bins not containing quantiles maintain coarser granularity. This local quality adjustment optimizes the balance between precision and memory consumption.
3Measurement precision
If multiple passes through the data are performed to compute frequency counters, then measurement precision is improved, but loss of time increases
Solution Approach 1:
Maximum and minimum values are computed in a preliminary pass before the main frequency counting process. This preliminary action establishes the bin boundaries and enables efficient frequency counter computation in subsequent passes, reducing the overall processing time by avoiding repeated maximum-minimum calculations.
Solution Approach 2:
The algorithm maintains continuous useful action by computing frequency counters for all bins in each data pass without interruption. Rather than performing separate maximum-minimum calculations for each bin, the system continuously processes data through predefined bins, maximizing the utility of each data pass.
Data Source
AI summary
A computing device computes a quantile value. A maximum value and a minimum value are computed for unsorted variable values to compute an upper bin value and a lower bin value for each bin of a plurality of bins. A frequency counter is computed for each bin by reading the unsorted variable values a second time. A bin number and a cumulative rank value are computed for a quantile. When an estimated memory usage value exceeds a predefined memory size constraint value, a subset of the plurality of bins are split into a plurality of bins, the frequency counter is recomputed for each bin, and the bin number and the cumulative rank value are recomputed. Frequency data is computed using the frequency counters. The quantile value is computed using the frequency data and the cumulative rank value for the quantile and output.