MoE Collective Communication via Virtual Expert Address Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large-scale distributed training of Mixture-of-Experts (MoE) models, the many-to-many variable counting communication (alltoallv) process is time-consuming due to the need for memory synchronization and resource allocation during data transmission among multiple processors.
Innovation Solution
A method that eliminates the need for address synchronization by pre-creating virtual address spaces and dynamically allocating actual physical memory based on data size, allowing direct data writing without synchronizing memory information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If memory synchronization and resource allocation are performed during data transmission among multiple processors, then data transmission completeness is ensured, but communication time increases
Solution Approach 1:
The patent pre-creates virtual address spaces and pre-allocates physical memory spaces before data transmission begins. The virtual-to-physical address mapping relationships are established in advance, so that during actual data transmission, processors can directly write data to pre-mapped physical addresses without performing memory synchronization or resource allocation operations, thereby eliminating time overhead while ensuring data transmission completeness
Solution Approach 2:
The patent introduces virtual address spaces as an intermediary layer between processors and physical memory. This virtual address space acts as a mediator that abstracts away the complexity of physical memory management, allowing processors to communicate using virtual addresses while the system automatically handles the mapping to pre-allocated physical addresses, thus eliminating the need for synchronization during communication
2Productivity
If physical memory spaces are pre-allocated for all possible data transmission scenarios, then data transmission speed increases, but memory resource wastage occurs
Solution Approach 1:
The patent segments the memory management into two independent parts: virtual address space (pre-created for all scenarios) and physical memory space (dynamically allocated based on actual needs). This segmentation allows the system to maintain pre-established address mappings for fast access while only allocating physical memory resources when actually needed, preventing both resource wastage and transmission delays
Solution Approach 2:
The patent implements dynamic physical memory allocation based on actual data transmission requirements. While virtual address spaces are pre-created statically, the corresponding physical memory spaces are allocated dynamically only when data needs to be transmitted to those addresses. This dynamic approach ensures that physical memory resources are allocated efficiently without being pre-wasted, while still maintaining fast transmission speeds through the pre-established virtual address infrastructure
Data Source
AI summary
The present disclosure provides a Mixture-of-Experts model based collective communication method, system and device. The collective communication method is applied to a communication receiving end, and includes: receiving a data write-in instruction, where the data write-in instruction includes to-be-processed data and first address information; accessing a virtual expert address in a pre-created virtual address space according to the first address information; applying for a corresponding actual physical space for the virtual expert address based on a size of the to-be-processed data; and writing the to-be-processed data into the actual physical space.