Remote Shared Partition Map Reduce Data Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The Map/Reduce task execution time is prolonged due to the time-consuming disk read/write operations and network data transmission in the existing Map/Reduce programming model, particularly when mapper nodes need to write results to local disks and then send them over TCP streams to reducer nodes.
Innovation Solution
The proposed solution involves a data processing method where a mapper node executes a map task and stores the data segment in a remote shared partition, allowing a reducer node to directly access and process the data segment from this partition, eliminating the need for disk I/O and network I/O operations by enabling the mapper and reducer nodes to run on the same or different CPUs within a CPU pool, and utilizing a storage pool with higher read/write speeds.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If mapper nodes write results to local disks using disk I/O and then send them over TCP streams to reducer nodes, then data can be persisted and transmitted reliably, but the execution time of the Map/Reduce task is significantly prolonged
Solution Approach 1:
The patent extracts the time-consuming disk I/O and network transmission operations from the data processing path. By allowing reducer nodes to directly access the storage pool where mapper nodes store data segments, the system removes the intermediate steps of writing to local disks and transmitting over TCP streams, thereby eliminating the time loss while maintaining data accessibility through the storage pool's file system interface
Solution Approach 2:
The storage pool acts as an intermediary between mapper nodes and reducer nodes. Instead of direct peer-to-peer communication requiring disk writes and network transmission, both node types access data through the storage pool's standardized interface, which mediates data transfer and eliminates the need for traditional disk I/O and network protocol overhead
2Productivity
If mapper nodes use disk I/O for multiple times to perform disk read/write operations during partitioning, sorting, and merging, then data can be processed in stages, but the execution time is significantly increased
Solution Approach 1:
The patent applies preliminary action by having mapper nodes store data segments directly in the storage pool during the map phase, making data immediately accessible to reducer nodes without requiring subsequent disk reads. This preliminary storage action eliminates the need for repeated disk I/O operations during partitioning, sorting, and merging phases that would otherwise be necessary if data remained in local buffers or temporary files
Data Source
AI summary
A mapper node and a reducer node respectively run on different central processing units (CPUs) in a CPU pool, and a remote shared partition shared by the mapper node and the reducer node is delimited in the storage pool. The mapper node executes a map task to obtain a data segment, and stores the data segment into a remote shared partition. The reducer node directly obtains a to-be-processed data segment from the remote shared partition, and executes a reduce task on the to-be-processed data segment.