Map Reduce Data Processing via Remote Storage Pool
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The Map/Reduce task execution time is prolonged due to time-consuming disk read/write operations and network data transmission in the existing Map/Reduce programming model.
Innovation Solution
A data processing method and system where a mapper node executes a map task and stores the result in a remote storage area, allowing a reducer node to directly access and process the data from this area, eliminating the need for local disk I/O and network transmission, and utilizing a CPU pool and storage pool with communicative controllers for efficient data access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If the mapper node writes execution results to local disk and the reducer node reads from local disk with network transmission, then data can be stored and transferred, but the execution time is prolonged
Solution Approach 1:
The patent merges the storage pool with the CPU pool to create an integrated memory system. The storage pool is directly coupled to the CPU pool, allowing mapper nodes and reducer nodes to access data through shared memory without external disk I/O or network transmission. This merging eliminates the traditional separation between computation and storage, achieving unified memory access for both data writing and reading operations.
Solution Approach 2:
The storage pool acts as an intermediary between the mapper node and reducer node. Instead of direct disk I/O and network transmission, data flows through the storage pool which is directly coupled to the CPU pool. This intermediary provides high-speed access while eliminating the need for traditional disk and network I/O operations.
2Ease of operation
If disk I/O and network I/O are used for data transmission, then data can be transferred between mapper and reducer nodes, but the operations are very time-consuming
Solution Approach 1:
The patent replaces the mechanical disk I/O system with a direct memory access system. Instead of physical disk read/write operations, data is accessed through the storage pool which is directly coupled to the CPU pool via memory buses. This substitution eliminates mechanical delays and provides electronic-level access speeds for both writing execution results and reading processed data.
Solution Approach 2:
The storage pool serves as an intermediary that eliminates the need for network I/O. Data is transferred through the directly-coupled storage pool rather than through network protocols, removing network transmission overhead while maintaining data transfer capability between mapper and reducer nodes.
Data Source
AI summary
A data processing method and apparatus are provided. A system includes a CPU pool and a storage pool, a mapper node and a reducer node respectively run on different CPUs in the CPU pool, and a remote storage area shared by the mapper node and the reducer node is delimited in the storage pool. In this method, the mapper node executes a map task, and stores a data segment obtained by executing the map task, into the remote storage area; and the reducer node directly obtains a data segment to be processed by the reducer node from the remote storage area, and executes a reduce task on the data segment. The method omits some actions in the prior art, and shortens a time for executing the Map/Reduce task.