Distributed Quantile Calculation Using Local Histograms
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for determining exact quantiles, such as the median, in large distributed datasets are resource-intensive and inefficient, particularly in distributed systems where data is stored across multiple machines, due to the high cost of sorting and memory requirements.
Innovation Solution
A distributed computing system that uses a central driver program and multiple nodes to allocate memory, load data blocks, aggregate numeric data, determine histogram bins, and initiate a histogram-building process to efficiently calculate exact quantiles, leveraging Apache Hadoop and Apache Spark for scalable and parallel processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If exact methods are used to determine median and quantiles in distributed datasets, then measurement precision is improved, but use of energy and loss of time increase due to multiple iterations and data reading
Solution Approach 1:
The patent divides the distributed dataset into multiple data blocks stored across different nodes, and processes each block independently using local histograms. This segmentation allows parallel computation while maintaining exact quantile results, resolving the contradiction between accuracy and runtime performance.
Solution Approach 2:
The system performs preliminary actions by pre-computing histograms for each data block during initial data loading, and by pre-determining the quantile rank k before processing. This allows the main computation to proceed more efficiently without requiring multiple full data reads, thus improving runtime while maintaining precision.
2Measurement precision
If exact methods are used to determine median and quantiles in distributed datasets, then measurement precision is improved, but use of energy by moving object increases due to resource intensive operations
Solution Approach 1:
By segmenting the large dataset into smaller blocks processed in parallel across multiple nodes, the system reduces the energy consumption of individual operations while maintaining overall computational accuracy. Each node processes only its local data block, reducing per-node energy usage compared to centralized processing.
Solution Approach 2:
The patent uses histogram structures that store summarized data representations (copies of data distributions) rather than the full original data. These histograms capture the essential information needed for quantile calculation without requiring processing of all原始 data, thus reducing energy consumption while maintaining measurement precision.
3Measurement precision
If simple algorithms are used to collect data into one machine and sort globally, then measurement precision is improved, but device complexity and loss of time increase due to expensive sorting operation
Solution Approach 1:
Instead of collecting all data to a single machine for sorting, the patent segments the data across multiple nodes and computes local histograms independently. This eliminates the need for complex global sorting operations while maintaining the ability to accurately determine quantiles through aggregation of local results.
Solution Approach 2:
The patent introduces histogram structures as intermediary representations that summarize data distributions. These histograms serve as mediators between the raw data and the final quantile calculation, replacing the need for complex global sorting operations while maintaining measurement precision.
4Productivity
If partial sorting strategies and partitioning strategies are used, then productivity is improved, but memory requirements increase proportionally to the amount of input data
Solution Approach 1:
The patent uses histogram structures that create compressed representations (copies) of data distributions. These histograms store only the essential information (bucket counts and value ranges) needed for quantile calculation, significantly reducing memory requirements compared to storing or sorting the full input data, while maintaining productivity through parallel processing.
Solution Approach 2:
The patent changes the data representation parameters from raw data values to histogram bucket counts and ranges. This parameter transformation reduces the amount of data that must be stored in memory while preserving the information needed for accurate quantile calculation, thus improving productivity without proportionally increasing memory usage.
Data Source
AI summary
A computing system for big data processing includes: a first node, configured to execute a central driver program; and a plurality of data and/or computing nodes, configured to store a plurality of data blocks corresponding to a data set. The first node and the plurality of data and/or computing nodes form a distributed computing environment configured for determining an exact value for one or more desired quantiles for the data set.