MoE Alltoallv Communication With Virtual Expert Addresses
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large-scale distributed training of Mixture-of-Experts (MoE) models, the alltoallv communication process is time-consuming due to the need for memory synchronization and address information exchange among multiple processors, leading to inefficiencies and resource overheads.
Innovation Solution
A Mixture-of-Experts model based collective communication method that utilizes pre-created virtual address spaces and dynamic allocation of actual physical memory based on data size, eliminating the need for address synchronization and optimizing memory usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If memory synchronization and address information exchange are performed among multiple processors during alltoallv communication, then data communication can be completed, but time consumption increases and processing efficiency deteriorates
Solution Approach 1:
The system pre-allocates contiguous physical memory spaces for all processors before the alltoallv communication operation begins. Each processor's buffer address information is predetermined and stored in a buffer address mapping table, eliminating the need for real-time address synchronization during data transmission. This preliminary preparation of memory spaces and address information directly resolves the time consumption issue while maintaining communication completeness.
2Reliability
If address information exchange is performed among processors, then data can be routed correctly, but resource overhead increases
Solution Approach 1:
The system merges the address management functions into a unified buffer address mapping table that stores all processor buffer address information in advance. Instead of each processor maintaining and exchanging separate address information, the centralized mapping table consolidates all address data, reducing memory usage and communication overhead while ensuring accurate data routing through the pre-stored address mappings.
Solution Approach 2:
The buffer address mapping table serves as an intermediary between the data transmission logic and the physical memory spaces. Rather than processors directly exchanging and managing each other's address information, the mapping table mediates the address translation process, storing the correspondence between logical buffer addresses and physical memory locations, thereby reducing direct communication overhead while maintaining routing accuracy.
3Productivity
If dynamic memory allocation is performed based on data size, then memory usage efficiency improves, but allocation complexity increases
Solution Approach 1:
The system performs preliminary allocation of contiguous physical memory spaces for each processor based on their respective buffer sizes before the alltoallv operation. The host CPU calculates and determines the exact memory space requirements for each processor in advance, allocating continuous physical memory regions accordingly. This preliminary sizing and allocation simplifies the process during actual data transmission, as no dynamic reallocation is needed during runtime, thereby maintaining efficiency while managing complexity through advance planning.
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
The present disclosure provides a Mixture-of-Experts model based collective communication method, system and device, and relates to the field of communication technology, and specifically to the fields of technologies such as chip processors, computing power clusters, collective communication operations and large models. The collective communication method is applied to a communication receiving end, and includes: receiving a data write-in instruction, where the data write-in instruction includes to-be-processed data and first address information; accessing a virtual expert address in a pre-created virtual address space according to the first address information; applying for a corresponding actual physical space for the virtual expert address based on a size of the to-be-processed data; and writing the to-be-processed data into the actual physical space. According to the collective communication method provided in the present disclosure, the collective communication can be realized without synchronizing memory information.