Dynamic Key Range Adjustment for Distributed Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In parallel computing, efficiently assigning large data items to multiple processors to achieve balanced processing and minimize overall processing time is challenging, especially with MapReduce jobs where Reducers often have uneven loads due to uneven data distribution across processors.
Innovation Solution
Analyzing statistical properties of data segments to dynamically adjust key range boundaries, ensuring a more even distribution of data items across processors by calculating mean and standard deviation, and adjusting boundaries to equalize the probability of each processor finishing last, thereby reducing overall processing time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data items are assigned to processors using simple alphabetical key ranges, then the assignment process is simple and fast, but the load distribution across processors becomes uneven
Solution Approach 1:
The patent performs preliminary statistical analysis on a sample of data blocks before the actual processing begins. By calculating mean and standard deviation of key distributions in advance, the system determines optimal key range boundaries that will balance the load across processors. This preliminary action ensures that when processing starts, the load distribution is already optimized, resolving the contradiction between simple assignment and balanced load.
Solution Approach 2:
The patent changes the parameters of key range boundaries from fixed alphabetical divisions to statistically optimized boundaries. By adjusting the boundaries based on mean and standard deviation calculations from sampled data, the system transforms the key range parameters to achieve both simplicity in assignment and balance in load distribution across processors.
2Ease of operation
If a perfect map is created by aggregating all data items, then load distribution becomes even, but the time required exceeds the entire job processing time
Solution Approach 1:
The patent applies partial action by analyzing only a sample of data blocks rather than aggregating all data items. By selecting a representative subset of blocks for statistical analysis, the system achieves sufficient load balancing information without the time cost of processing the entire dataset. This partial analysis provides the necessary mean and standard deviation values to optimize key ranges while keeping the assignment phase much faster than full aggregation.
3Device complexity
If key range boundaries are fixed before processing, then the assignment plan is simple to implement, but processing time increases due to uneven load distribution
Solution Approach 1:
The patent performs preliminary statistical analysis on sampled data blocks to determine optimal key range boundaries before processing begins. By calculating mean and standard deviation in advance and setting boundaries accordingly, the system achieves balanced load distribution without adding complexity during the actual processing phase. The boundaries are determined once beforehand, maintaining simplicity while optimizing processing time.
Data Source
AI summary
Dynamic generation and implementation of assignment mappings of data items in large data files to distributed processors to achieve objectives such as reduced overall processing time like. Any appropriate key (e.g., character string) can be identified or obtained for each data item in a data file and the file can be segmented into sequential data blocks, where each data block includes a set of data items. The data items in each of a first plurality of the blocks (e.g., sampled block set) may be initially sorted into one of a plurality of key ranges of a search space (each corresponding to a different respective processor) and analyses conducted on the data items totals in each key range. The key range boundaries can be adjusted by accounting for uncertainty in the sample estimates to more evenly distribute data items from all blocks sent to each processor and thereby achieve the objective.