Map Processing Allocation in Distributed Data Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed processing systems using map-reduce techniques, the simultaneous execution of map processing across all data nodes leads to delayed start times and concentrated execution time slots, causing inefficiencies and potential delays in processing large datasets.
Innovation Solution
A method where a master node allocates map processing to slave nodes sequentially as data is inputted, shifting to shuffle and sort processing only after data with tail information is received, allowing for parallel execution and avoiding time slot concentration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If map processing is executed simultaneously across all data nodes, then processing can be performed in parallel, but the start time is delayed and execution time slots become concentrated causing inefficiency
Solution Approach 1:
The system performs map processing in advance as data is being inputted into the database, rather than waiting until all data is loaded. This preliminary action allows processing to start earlier and overlap with data loading, eliminating idle time and improving overall productivity without extending processing time
Solution Approach 2:
The system maintains continuous useful action by initiating map processing as soon as data chunks are available, keeping processing nodes continuously occupied. This prevents idle time between data loading and processing, ensuring that computational resources are fully utilized throughout the entire data pipeline
2Reliability
If map processing is performed after all data is loaded, then data completeness is ensured, but processing start time is delayed
Solution Approach 1:
The system performs map processing on data chunks as they become available during the loading phase, rather than waiting for complete data loading. This preliminary processing ensures that processing starts immediately on available data while maintaining reliability through the tail information mechanism that identifies when all data has been processed
Solution Approach 2:
The system uses tail information as a feedback mechanism to determine when all data has been inputted. This feedback allows the system to know when to stop processing and when data completeness is achieved, ensuring reliability while enabling early processing of available data chunks
Data Source
AI summary
A non-transitory computer-readable recording medium storing a program causing a processor to execute a process, the process includes detecting input of data into a memory to which data is inputted in sequence, the data being a processing object of first processing; allocating the first processing, of which a processing object is the data, with respect to any node in a communicable node group; determining whether or not the data is provided with tail information, the tail information indicating tail data of a series of data that are processing objects of the first processing, when detecting input of the data; and allocating second processing, of which a processing object is a processing result of the first processing that is executed with respect to each piece of data of the series of data, to any node of the node group when determining that the data is provided with the tail information.