Parallel Data Processing with Merger Information Sharing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data processing systems in distributed environments face challenges in parallelizing data processing from multiple start points to one or more end points and merging data items after parallel processing, as they lack the necessary mechanisms to share required information for merging processes across information processing apparatuses.
Innovation Solution
The system includes information processing apparatuses with a division function to divide processing information into parallelizable pieces, a determination function to assign processing tasks, and an execution function to execute processes, enabling parallelization and merging of data items across multiple information processing apparatuses in a distributed environment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data processing is parallelized across multiple information processing apparatuses, then data processing speed is improved, but the ability to merge data items after parallel processing deteriorates due to lack of information sharing mechanisms
Solution Approach 1:
The patent applies preliminary action by having information processing apparatuses share merger information (such as merger destination addresses, merger point identifiers, and data format specifications) before the parallel processing begins. This allows the system to maintain both parallelization benefits and merger capability, as the necessary information is prepared in advance and made available to all apparatuses before they execute their parallel processing tasks.
2Productivity
If processing information is divided into multiple pieces for parallel processing, then processing efficiency is improved, but coordination for merging results becomes more complex
Solution Approach 1:
The patent introduces an intermediary mechanism where a designated information processing apparatus (or a centralized coordination system) acts as a merger coordinator. This intermediary receives merger information from all participating apparatuses, manages the consolidation of parallel processing results, and handles the complexity of coordination, thereby allowing individual apparatuses to focus on their parallel processing tasks without burdening themselves with complex merger coordination logic.
3Quantity of substance
If multiple information processing apparatuses execute parallel processes, then data processing capacity is increased, but information sharing for merger purposes is insufficient
Solution Approach 1:
The patent applies segmentation by dividing the overall data processing task into independent parallel processing segments that can be executed by multiple apparatuses simultaneously. Each apparatus processes its assigned segment while holding onto critical merger information (such as how its results should be combined with others). This segmentation approach allows the system to scale processing capacity across multiple apparatuses while maintaining the necessary information for subsequent merger operations.
Data Source
AI summary
One or more information processing apparatuses to process information are provided. The information processing apparatus includes: a division function that divides processing information into a plurality of pieces, under a division condition that designates parallel processing among the information processing apparatuses, the processing information indicating a data processing procedure from a plurality of start points to one or more end points; a determination function that uniquely determines an assignee of each piece of the processing information divided by the division function, as any of the information processing apparatuses; and an execution function that executes a process in the information processing apparatus determined by the determination function.