AI Core Token Routing for Sub-Network Load Balancing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional processors face inefficiencies when handling AI workloads due to load imbalances between sub-networks in neural networks, reducing the performance advantages of parallelism.
Innovation Solution
Implementing probability distribution-based load balancing to rearrange workload distribution among AI-accelerating cores, measuring hardware occupancy, and generating a routing plan to optimize token distribution across sub-networks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional processors are used to execute general-purpose computations for AI workloads, then they can handle diverse computational tasks, but they face significant inefficiencies due to load imbalances between sub-networks
Solution Approach 1:
The patent changes the routing parameters dynamically based on measured hardware occupancy. The system adjusts which sub-network processes which tokens by modifying routing decisions in real-time, transforming static processor allocation into dynamic parameter-based routing that adapts to actual hardware utilization states
Solution Approach 2:
The system implements dynamic load balancing where the routing of tokens to sub-networks is not fixed but changes based on measured occupancy. The control mechanism continuously monitors hardware state and adjusts token routing decisions, making the system adaptive rather than static
2Speed
If sub-networks are implemented in parallel by different computational devices to achieve performance advantages, then processing speed increases, but load imbalances between sub-networks reduce the performance advantages
Solution Approach 1:
The system implements a feedback mechanism where hardware occupancy is measured and used to inform subsequent routing decisions. The control unit monitors the state of computational devices and adjusts token routing based on this feedback, ensuring that load imbalances are corrected in real-time to maintain effective parallelism
Solution Approach 2:
The system performs preliminary measurement of hardware occupancy before making routing decisions for subsequent tokens. By measuring the state of computational devices in advance and using this information to guide future routing, the system proactively prevents load imbalances rather than reacting to them after they occur
Data Source
AI summary
Aspects of this disclosure relate to load balancing for artificial intelligence (AI) accelerating cores. A control unit may cause the cores to perform computations of a neural network for a first set of tokens. The control unit may measure hardware occupancy of sub-networks of the network in each of the cores for the computations of the first set of tokens. The control unit generates a load distribution based on the measured hardware occupancy. The control unit re-arranges the load distribution to generate a routing plan that determines how a token selected to be processed by one of the sub-networks is routed among the cores. The control unit may route a second set of tokens to the cores according to the routing plan.