Memory-Based Data Shuffling in Distributed Task Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed systems handling big data, the data shuffling process between nodes consumes significant computing resources, leading to inefficiencies and high resource utilization.
Innovation Solution
A distributed task processing method where sub-tasks are executed by different devices in a distributed system, with result data stored in memory to reduce disk interaction, and transmitted directly between network cards using RDMA technology to minimize CPU overhead and resource consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is stored in disk during distributed task processing, then data reliability is improved, but I/O overhead and computing resource consumption increase
Solution Approach 1:
The patent extracts the data storage function from the traditional disk-based storage system and implements it in memory instead. By taking out the storage operation from disk I/O and placing it in memory, the system eliminates the bottleneck of disk access while maintaining data availability for computation, thus resolving the contradiction between reliability and productivity
Solution Approach 2:
The patent replaces the mechanical disk I/O system with an electronic memory-based storage system. This substitution eliminates the physical limitations of disk access speed and reduces mechanical wear, thereby improving both data access reliability and overall system productivity by removing the I/O overhead associated with mechanical storage
2Adaptability or versatility
If traditional network protocols are used for data transmission, then compatibility is improved, but CPU overhead and resource consumption increase
Solution Approach 1:
The patent extracts the data transmission function from the CPU-managed network protocol stack and implements it through a dedicated network device. By taking out the network communication burden from the CPU, the system reduces CPU overhead significantly while maintaining protocol compatibility through the network device's built-in protocol handling capabilities
Solution Approach 2:
The patent introduces a network device as an intermediary between the memory system and the external network. This intermediary handles all protocol processing, packetization, and transmission tasks, allowing the main system to focus on computation while the network device manages communication, thus reducing CPU overhead without sacrificing compatibility
3Productivity
If data shuffling is performed between nodes, then distributed task execution is improved, but computing resource consumption and execution duration increase
Solution Approach 1:
The patent performs preliminary data preparation by storing required data in memory before the actual computation task begins. This preliminary action ensures that data is readily available when needed during task execution, eliminating delays associated with disk access or network retrieval during computation, thus reducing overall execution duration while maintaining distributed task execution efficiency
Solution Approach 2:
The patent enables continuous data access during distributed task execution by keeping data in memory throughout the computation process. This continuity eliminates the stop-start nature of disk I/O operations and allows for uninterrupted data processing, thereby reducing execution duration while maintaining the productivity benefits of distributed task execution
Data Source
AI summary
A distributed task processing method, a distributed system, and a first device. A distributed task comprises at least two sub-tasks respectively executed by at least two devices in a distributed system, and the at least two devices in the distributed system comprise a first device. A processor of the first device stores in a memory result data of the execution of the first sub-tasks, such that the interaction with a disk is reduced in a data computing stage. Since the result data is stored in the memory, a network card of the first device can directly transmit the result data in the memory to a network card of a second device by means of a network, such that the interaction with the disk and the consumption of computing resources are also reduced in a data shuffling stage.