MapReduce Shuffler Pipeline Policy for Shuffle-Ahead Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
MapReduce processes face performance issues due to shufflers passively responding to fetch requests from reducers, leading to slower data retrieval from disk and inefficient use of scheduling information, particularly when intermediate results are flushed from memory.
Innovation Solution
Implementing a pipeline policy that prioritizes keeping map task outputs in memory and using 'shuffle-ahead' to proactively move data to nodes where reduce tasks will be executed, reducing the need for disk access and optimizing data transfer within the distributed computing grid.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If shufflers passively respond to fetch requests from reducers, then system simplicity is maintained, but data retrieval speed deteriorates due to disk access delays
Solution Approach 1:
The shuffler proactively transmits intermediate results to reducers before the reducers request them, based on predicted timing and scheduling information. This preliminary action ensures data is already in memory when needed, eliminating disk access delays while maintaining system simplicity through automated prediction-based transmission
2Quantity of substance
If intermediate results are written to disk to free memory, then memory capacity is preserved, but processing performance deteriorates due to slower disk access
Solution Approach 1:
The system predicts when intermediate results will be needed by reducers and proactively keeps them in memory during that predicted time window. By using scheduling information to anticipate future access patterns, the system maintains data in memory just-in-time, preserving both memory capacity utilization and processing performance without requiring premature disk writes
3Speed
If more memory is allocated to hold intermediate results, then data access speed improves, but memory resource utilization deteriorates due to wasted capacity
Solution Approach 1:
The shuffler dynamically adjusts which intermediate results remain in memory based on real-time predictions of when reducers will need them. Using scheduling information, the system continuously optimizes memory allocation by keeping only the necessary data in memory at any given time, achieving fast access speeds while maximizing memory resource utilization through adaptive, time-based management
Data Source
AI summary
A shuffler receives information associated with partition segments of map task outputs and a pipeline policy for a job running on a computing device. The shuffler transmits to an operating system of the computing device a request to lock partition segments of the map task outputs and transmits an advisement to keep or load partition segments of map task outputs in the memory of the computing device. The shuffler creates a pipeline based on the pipeline policy, wherein the pipeline includes partition segments locked in the memory and partition segments advised to keep or load in the memory, of the computing device for the job, and the shuffler selects the partition segments locked in the memory, followed by partition segments advised to keep or load in the memory, as a preferential order of partition segments to shuffle.