MapReduce Aggregator Eliminates Sorting via Hash Tables
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current MapReduce frameworks require sorting of intermediate key/value pairs before the reduce operation, which is resource-intensive and inefficient, especially when dealing with large volumes of data and a small number of distinct keys.
Innovation Solution
Implementing a generic aggregator that uses a hash table on both the map and reduce sides, allowing for non-sorted aggregation of intermediate key/value pairs with common keys being handled by the same reducer, thereby eliminating the need for sorting and reducing memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If sorting is performed on intermediate key/value pairs before reduce operation, then correct grouping of common keys is achieved, but resource consumption and processing time increase significantly
Solution Approach 1:
The patent applies preliminary action by having mappers pre-aggregate and pre-sort key/value pairs before transmitting them to reducers. This preliminary processing ensures that when data arrives at the reducer, it is already organized by key, eliminating the need for reducers to perform sorting operations and thereby improving overall processing efficiency while maintaining correct grouping
Solution Approach 2:
The patent introduces an intermediary sorting mechanism at the mapper level that acts as a mediator between the map and reduce phases. This intermediary sorting process organizes data locally at each mapper before transmission, serving as a bridge that prepares data for efficient reduce operations without requiring global sorting at the reducer stage
2Stability of the object's composition
If sorting is performed on intermediate key/value pairs, then data is properly organized for reduce operation, but memory usage and computational resources increase
Solution Approach 1:
The patent applies segmentation by dividing the sorting task into smaller segments performed independently at each mapper node rather than requiring centralized sorting at the reducer. This segmentation allows data to be organized in distributed, manageable chunks that consume less memory at any single point while maintaining proper key grouping across the system
3Reliability
If traditional MapReduce sorting is used, then all values for the same key are guaranteed to go to the same reduce call, but processing speed decreases
Solution Approach 1:
The patent uses preliminary action by performing aggregation and sorting at the mapper level before data leaves the map phase. This preliminary organization guarantees that all values for the same key are properly grouped and directed to the appropriate reducer, maintaining reliability while avoiding the need for slow sorting operations during the reduce phase
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A computer-implemented method for processing input data in a mapreduce framework includes: receiving, in the mapreduce framework, a data processing request for input data; initiating, based on the data processing request, a map operation on the input data by multiple mappers in the mapreduce framework, each of the mappers using an aggregator to partially aggregate the input data into one or more intermediate key/value pairs; initiating a reduce operation on the intermediate key/value pairs by multiple reducers in the mapreduce framework, wherein, without sorting the intermediate key/value pairs, those of the intermediate key/value pairs with a common key are handled by a same one of the reducers, each of the reducers using the aggregator to aggregate the intermediate key/value pairs into one or more output values; and providing the output values in response to the data processing request.