M3R Engine Shared Heap State for MapReduce Job Sequences
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The MapReduce programming model, particularly in Hadoop, incurs significant performance overhead due to disk I/O and serialization costs, as it supports only single-job execution and requires data to be written to disk and re-read for subsequent jobs, leading to inefficient data processing and communication between map and reduce tasks.
Innovation Solution
The M3R engine optimizes MapReduce job sequences by running all jobs in a collection of processes with shared heap state, using in-memory caching and reducing serialization overhead, and employing a combiner to process data locally before network transmission, thereby minimizing data transfer and leveraging X10's inter-process communication for efficient data handling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If MapReduce jobs are executed using traditional Hadoop framework with disk I/O and serialization, then data processing can be performed on large datasets, but execution time and performance overhead are significantly increased
Solution Approach 1:
The patent applies preliminary action by pre-loading data into shared memory before map tasks execute. The system reads input data from storage and loads it into a shared heap memory space in advance, so that mappers can directly access data in memory without repeated disk I/O operations during job execution, significantly reducing execution time
Solution Approach 2:
The patent uses copying by creating in-memory copies of data in shared heap space. Instead of repeatedly reading from disk, the system creates copies of data in memory that can be efficiently accessed and processed by multiple mapper tasks simultaneously, eliminating the performance penalty of repeated serialization and disk access
2Adaptability or versatility
If data is written to disk and re-read for subsequent jobs in Hadoop, then jobs can be executed sequentially, but I/O operations and serialization overhead increase execution time
Solution Approach 1:
The patent implements continuity of useful action by maintaining data in shared memory across multiple MapReduce jobs. Instead of writing to disk and re-reading, the system keeps data continuously available in shared heap space, allowing sequential jobs to access the same in-memory data without I/O interruptions, enabling continuous processing without serialization overhead
Solution Approach 2:
The shared memory space serves multiple functions across different jobs. The same in-memory data structure can be read by multiple mappers in sequence, serving as input for multiple different MapReduce jobs without requiring repeated disk I/O, making the memory space a universal input source for job sequences
3Reliability
If mappers and reducers operate in separate processes with full serialization, then job isolation is maintained, but communication overhead and execution speed are reduced
Solution Approach 1:
The shared heap memory space acts as an intermediary between mappers and reducers. Instead of direct serialization and deserialization between processes, both mappers and reducers access data through the shared memory intermediary, which eliminates serialization overhead while maintaining process isolation through controlled memory access interfaces
4Loss of time
If in-memory execution with shared heap state is used, then execution time is reduced, but memory management complexity increases
Solution Approach 1:
The patent merges memory management responsibilities into a unified shared heap space that is jointly managed by all mapper and reducer tasks. Instead of each task managing separate memory spaces, the system combines all memory management into a single shared structure with centralized allocation and deallocation, simplifying management while enabling efficient in-memory access
Data Source
AI summary
Executing a map reduce sequence may comprise executing all jobs in the sequence by a collection of a plurality of processes with each process running one or more mappers, combiners, partitioners and reducers for each job, and transparently sharing heap state between the jobs to improve metrics associated with the job. Processes may communicate among themselves to coordinate completion of map, shuffle and reduce phases, and completion of said all jobs in the sequence.