Global Memory Shuffle Access for Distributed Reduction Servers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed high-concurrency computing frameworks face inefficiencies in data transmission due to limited memory capacity, network transmission delays, and small bandwidth between computing nodes, affecting processing efficiency in the shuffle phase.
Innovation Solution
Implement a global memory system where reduction servers read data directly from a shared memory without copying and transmitting data blocks, and mapping servers store data blocks in a unified global memory to avoid disk reading/writing, using address mapping and asynchronous data partitioning to improve efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is transmitted between computing nodes through network copy-transmission, then data can be accessed by reduction servers, but data transmission efficiency is low due to limited memory capacity, network transmission delay, and small bandwidth
Solution Approach 1:
The patent merges the memory resources of multiple computing nodes into a unified global memory system. Mapping servers and reduction servers share a common address space, allowing reduction servers to directly access data blocks in global memory without network copy-transmission operations, thereby eliminating network transmission delays and bandwidth limitations
Solution Approach 2:
The patent introduces a memory management unit as an intermediary that handles address mapping and data access coordination between reduction servers and mapping servers. The memory management unit translates logical addresses used by reduction servers into physical addresses in the global memory system, enabling efficient direct access while maintaining system coherence
2Productivity
If data blocks are copied and transmitted between computing nodes, then reduction servers can process data from multiple mapping servers, but processing efficiency is limited by memory capacity and network bandwidth
Solution Approach 1:
The patent creates a universal global memory system that serves multiple functions: it acts as shared storage for all mapping servers, provides direct access for reduction servers, and enables in-memory data processing. This multi-functional memory system eliminates the need for separate network transmission infrastructure and improves processing efficiency across the entire distributed system
Data Source
AI summary
Methods and apparatuses for processing data are provided. In an implementation, a method comprising: obtaining, by a reduction server of a plurality of reduction servers in a distributed processing system and from a first storage area, metadata of first data to be read, wherein the first data comprises a target data block in a plurality of data blocks of second data, and wherein the second data comprises a processing result of a mapping server of a plurality of mapping servers for input data, determining a first address of the first data in a global memory based on the metadata, wherein the global memory comprises memories of the plurality of mapping servers and the plurality of reduction servers, and reading the first data from the global memory based on the first address.