Data Arrangement in Distributed Processing Nodes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed processing systems, the large amount of communication required during the MPI process leads to processing inefficiencies due to network bandwidth and load dependencies, resulting in potential processing delays.
Innovation Solution
The system acquires size information and data arrangement information to generate a data arrangement instruction that optimizes the storage and retrieval of result data across multiple nodes, reducing remote reading and improving processing efficiency by aligning data chunks within the same node for the MPI process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If data is read via network in MPI process, then distributed processing can be executed, but processing efficiency deteriorates due to large communication amount and network bandwidth dependency
Solution Approach 1:
The patent applies preliminary action by arranging data in HDFS before the MPI process starts. The system predicts the data arrangement required for MPI, performs data movement and reorganization in advance, and stores the arranged data in HDFS. This eliminates the need for large-scale data communication during MPI execution, as data is already positioned locally on slave servers when needed.
Solution Approach 2:
The patent introduces HDFS as an intermediary between the MapReduce process and the MPI process. HDFS serves as a buffer and reorganization layer that decouples the data production (MapReduce) from data consumption (MPI). By using HDFS as the intermediary storage system, the patent enables efficient data transfer and arrangement without direct peer-to-peer communication overhead during MPI execution.
2Quantity of substance
If data is combined on HDFS from local files, then distributed data can be aggregated, but processing delay occurs due to network bandwidth and load constraints
Solution Approach 1:
The system performs data aggregation and arrangement in advance during the MapReduce phase, organizing data in HDFS according to the predicted MPI data requirements. This preliminary arrangement ensures that when MPI needs the data, it is already in the correct location and format, eliminating time-consuming data movement during the critical MPI execution phase.
Solution Approach 2:
The patent changes the data organization parameters in HDFS based on the requirements of the subsequent MPI process. By predicting MPI's data access patterns and reorganizing data accordingly (changing how data is distributed and stored), the system optimizes for fast local access during MPI while maintaining the benefits of distributed storage.
Data Source
AI summary
An apparatus for processing data includes a processor configured to acquire size information on a size of first result data from a first node. The first result data is generated in first distributed processing executed by the first node on a first part of first input data for the first distributed processing executed by a plurality of nodes. The processor is configured to acquire data arrangement information for second distributed processing executed by the plurality of nodes. The processor is configured to generate a data arrangement instruction on basis of the size information and the data arrangement information. The data arrangement instruction defines an arrangement pattern of second result data generated in the first distributed processing executed by the plurality of nodes and stored in the plurality of nodes as the second input data. The processor is configured to transmit the data arrangement instruction to the plurality of nodes.