MoE Router Acceleration via Hierarchical Softmax Approximation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for routing in Mixture of Experts (MoE) ensembles are computationally expensive and energy-inefficient, especially as the number of parameters in neural networks increases to billions.
Innovation Solution
The proposed solution involves accelerating routing in MoE ensembles using analogue in-memory computing (AIMC) and a hierarchical Softmax approximation. This approach includes receiving input data at a router comprising matrix-vector-multiplier (MVM) tiles, generating output vectors by applying a routing function, determining sub-vectors, merging them into an element vector, applying a Softmax function, and configuring the router based on the largest elements of the probability distribution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the number of parameters in neural network is increased to billions, then the model capacity and performance are improved, but the computation amount required for routing increases linearly
Solution Approach 1:
The patent divides the routing computation into two stages: (1) computing top-k scores for each expert independently, and (2) computing the normalized probability distribution only for the selected top-k experts. This segmentation reduces the Softmax computation from O(N) to O(k) where k << N, thereby resolving the linear scaling problem while maintaining model capacity.
Solution Approach 2:
Instead of computing the full Softmax distribution over all N experts, the patent computes the Softmax only for the top-k selected experts (partial action). This approximation is sufficient for routing purposes and dramatically reduces computational complexity from O(N) to O(k), resolving the contradiction between model capacity and computation amount.
2Adaptability or versatility
If the number of parameters in neural network is increased to billions, then the model capacity and performance are improved, but the energy consumption increases
Solution Approach 1:
The routing process is segmented into independent top-k score computation followed by selective Softmax normalization. This allows the system to avoid energy-intensive full Softmax computations over all N experts, reducing energy consumption while maintaining the ability to handle billion-parameter models through the reduced O(k) computation.
Solution Approach 2:
The patent performs partial Softmax computation only for top-k experts rather than all N experts. This partial action reduces the energy consumption proportionally to the ratio k/N, enabling scalable deployment of billion-parameter models with controlled energy usage.
3Adaptability or versatility
If the number of parameters in neural network is increased to billions, then the model capacity and performance are improved, but the routing speed decreases
Solution Approach 1:
The routing computation is segmented into two independent phases: (1) efficient top-k score computation that can be parallelized across experts, and (2) reduced Softmax computation over only k experts. This segmentation enables better parallelization and reduces the critical path, thereby maintaining routing speed despite increased model capacity.
Solution Approach 2:
By computing Softmax only for top-k experts instead of all N experts, the patent reduces the computational bottleneck in the routing process. This partial computation approach maintains routing speed by reducing the O(N) operation to O(k), enabling fast inference even with billion-parameter models.
Data Source
AI summary
According to one embodiment, a method, computer system, and computer program product for routing acceleration in mixture of experts ensembles is provided. The present invention may include receiving input data at a router; generating a plurality of output vectors by applying a routing function to the input data, wherein each output vector is associated with one or more respective tiles or pairs of tiles in a plurality of MVM tiles; determining a plurality of sub-vectors in the output vectors, wherein each sub-vector in the plurality of sub-vectors is associated with a respective output vector in the plurality of output vectors, and merging the sub-vectors into an element vector; generating a probability distribution vector by applying a Softmax function to the element vector and determining the largest elements of the probability distribution; and configuring the router based on the one or more largest elements of the probability distribution.