Mixture-of-Experts Load Balancing with 2.5D Optical Memory Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Mixture of experts (MoE) models face challenges in load balancing due to the uncertainty of which expert networks will be activated until the input is processed, leading to difficulties in pre-loading all networks in memory and inefficient resource utilization.
Innovation Solution
Implementing a 2.5D configuration and/or optical configuration to provide direct access to computing dies from memory dies, along with a synchronization mechanism to determine data splits across computing units based on MoE requests, allowing experts to be loaded as-needed and reducing the number of memory dies required.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If all expert networks are pre-loaded into fast memory, then processing speed is improved, but memory bandwidth and power consumption increase
Solution Approach 1:
The system performs preliminary actions by pre-computing routing information and expert activation patterns during training, storing this information in compact data structures. During inference, this pre-computed information enables rapid expert selection without requiring all experts to be pre-loaded into fast memory, thus maintaining high processing speed while reducing memory bandwidth requirements and power consumption.
2Productivity
If all expert networks are loaded into memory, then processing capability is improved, but device complexity and memory requirements increase
Solution Approach 1:
The system segments the expert network architecture into multiple independent expert modules, each capable of being independently loaded and unloaded from memory. Instead of requiring all experts to reside in fast memory simultaneously, the system loads only the subset of experts needed for each specific input, reducing memory requirements and device complexity while maintaining full processing capability through dynamic expert selection.
3Loss of energy
If expert networks are dynamically loaded as-needed, then memory bandwidth is reduced, but access latency increases
Solution Approach 1:
The system performs preliminary actions by pre-computing and storing routing information that identifies which experts should be activated for different input types. This pre-computed routing information is stored in compact data structures that can be quickly accessed during inference, enabling rapid expert selection and loading without significant access latency, thus reducing memory bandwidth requirements while maintaining low latency.
4Quantity of substance
If remote memory access is used, then memory capacity is increased, but access complexity and latency increase
Solution Approach 1:
The system introduces an intermediary component in the form of a routing layer that sits between the input and the expert networks. This routing layer contains pre-computed information about which experts to activate and manages the dynamic loading/unloading of expert modules. The intermediary routing layer simplifies access complexity by providing a unified interface for expert selection while enabling efficient utilization of remote memory capacity for storing expert networks.
Data Source
AI summary
Aspects of the disclosure are directed to improving load balancing for serving mixture of experts (MoE) machine learning models. Load balancing is improved by providing memory dies increased access to computing dies through a 2.5D configuration and/or an optical configuration. Load balancing is further improved through a synchronization mechanism that determines an optical split of batches of data across the computing die based on a received MoE request to process the batches of data. The 2.5D configuration and/or optical configuration as well as the synchronization mechanism can improve usage of the computing die and reduce the amount of memory dies required to serve the MoE models, resulting in less consumption of power and lower latencies and complexity in alignment associated with remotely accessing memory.