Mixture-of-Experts Load Balancing via 2.5D and Optical Memory Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Load balancing in mixture of experts (MoE) machine learning models is challenging due to the uncertainty of which expert networks will be activated until input is processed, leading to difficulties in pre-loading all networks in memory and inefficient resource utilization.
Innovation Solution
Implementing a 2.5D configuration and/or optical configuration to provide memory dies with direct access to computing dies, along with a synchronization mechanism to optimize data distribution across computing units based on MoE requests, allowing experts to be loaded as-needed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If all expert networks are pre-loaded into memory in anticipation of potential activation, then all experts can be immediately accessed when needed, but memory bandwidth and power consumption increase significantly
Solution Approach 1:
The system performs preliminary identification of which expert networks will be activated based on the input data and routing logic, then pre-loads only those specific experts into memory before processing begins. This avoids loading all possible experts while ensuring the needed ones are ready, resolving the contradiction between fast access and low power consumption.
Solution Approach 2:
Different expert networks are loaded into different memory regions or computing units based on their specific activation requirements. The system optimizes memory allocation locally for each expert rather than uniformly loading all experts, reducing overall memory bandwidth usage and power consumption while maintaining fast access to activated experts.
2Use of energy by moving object
If expert networks are loaded dynamically as needed, then memory bandwidth and power consumption are reduced, but access latency increases due to loading time
Solution Approach 1:
The system identifies and loads required expert networks into memory before the actual processing of input data begins. By performing the loading operation in advance during the routing phase, the system eliminates loading latency during the critical processing phase, thus reducing overall access latency while avoiding the need to keep all experts permanently loaded.
Solution Approach 2:
The system maintains a pool of expert networks in memory that are frequently activated, ensuring continuous availability without repeated loading. The routing mechanism continuously monitors activation patterns and maintains optimal expert availability, balancing memory usage with access speed to minimize both power consumption and latency.
3Adaptability or versatility
If more memory dies are used to store all expert networks, then all experts can be simultaneously accessed, but device complexity and cost increase
Solution Approach 1:
The system segments expert networks into different groups or tiers based on their activation frequency and importance. Frequently activated experts are stored in fast memory, while less frequently used experts are stored in slower memory or on disk. This segmentation allows the system to handle diverse MoE requests effectively while using fewer high-performance memory dies, reducing device complexity.
Solution Approach 2:
The memory architecture is designed to serve multiple functions: storing activated experts, storing routing information, and providing temporary workspace for processing. By making the memory system multi-functional, the system can handle diverse MoE requests with a smaller, more efficient memory configuration, avoiding the need for separate dedicated memory for each expert network.
4Quantity of substance
If expert networks are stored in remote memory, then memory capacity is increased, but access latency and alignment complexity increase
Solution Approach 1:
The system extracts the most frequently activated expert networks from remote memory and places them in local or on-chip memory. By separating the hot experts from cold experts and storing them in different memory locations, the system increases effective memory capacity while minimizing access latency for the critical activated experts, avoiding the penalty of remote memory access.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Aspects of the disclosure are directed to improving load balancing for serving mixture of experts (MoE) machine learning models. Load balancing is improved by providing memory dies increased access to computing dies through a 2.5D configuration and/or an optical configuration. Load balancing is further improved through a synchronization mechanism that determines an optical split of batches of data across the computing die based on a received MoE request to process the batches of data. The 2.5D configuration and/or optical configuration as well as the synchronization mechanism can improve usage of the computing die and reduce the amount of memory dies required to serve the MoE models, resulting in less consumption of power and lower latencies and complexity in alignment associated with remotely accessing memory.