MapReduce Fault Tolerance via Local Reduce and Non-Block Compression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional MapReduce frameworks face significant challenges in efficiently processing extremely large data sets due to high computational costs and limited fault tolerance, particularly when dealing with failures in processing nodes.
Innovation Solution
The system employs a custom mapper script that outputs results directly to the reduce process, eliminating the need for a shuffle step. Results are written into multiple compressed files, which are then processed by map-only reduce jobs to aggregate and reduce data, ensuring that each file is handled by a single mapper, thereby improving fault tolerance and reducing computational overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional MapReduce framework is used to process extremely large data sets, then parallel processing capability is achieved, but fault tolerance is limited and computational costs are high
Solution Approach 1:
The patent segments the processing workflow into distinct phases: Map phase, local Reduce phase, and final Reduce phase. Each phase operates independently with its own fault tolerance mechanisms. The local Reduce phase processes data locally without requiring shuffle, creating isolated processing units that can fail independently without affecting the entire system.
Solution Approach 2:
The patent performs preliminary local reduction of data before the final reduce phase. By pre-processing and aggregating data locally at each node before shuffling, the system reduces the amount of data that needs to be transferred and processed in the final phase, while also creating intermediate results that can be recovered independently if failures occur.
2Productivity
If shuffle step is included in MapReduce process, then data redistribution is achieved, but computational overhead and processing time increase
Solution Approach 1:
The patent extracts and eliminates the shuffle step from the traditional MapReduce workflow by implementing local reduce phases that process data immediately after the map phase without requiring data redistribution. This removal of the shuffle step directly reduces processing time and computational overhead while maintaining processing throughput.
3Reliability
If all Map nodes are re-run to recover from Reduce node failure, then data consistency is maintained, but computational resources and time are wasted
Solution Approach 1:
The patent implements prior cushioning by creating local reduce phases that produce intermediate results stored independently before the final reduce phase. These intermediate results act as a cushion or buffer that can be recovered independently if failures occur, preventing the need to re-run entire Map phases and preserving both data consistency and computational resources.
4Productivity
If multiple Reduce nodes are used to process large data sets, then parallel processing is achieved, but system complexity increases
Solution Approach 1:
The patent segments the reduce operation into local reduce phases distributed across multiple nodes and a final aggregate reduce phase. This segmentation maintains parallel processing capability while simplifying each individual node's complexity, as each node only needs to implement local reduction logic rather than full MapReduce coordination.
Data Source
AI summary
A method for improving fault tolerance of MapReduce computing systems processing extremely large data sets employs a custom mapper script that outputs the results to the shuffle and reduce processes. The Reducer writes its results into multiple files that are then compressed. The resulting files are processed by a map-only Reducer job that aggregates and reduces the results of a single file into a single output. The non-block compression prevents the file from being split across multiple mappers. The job can recover if a single mapper fails without requiring that each Map node be rerun, thus improving fault tolerance.