Multi-Chip Neural Network Routing for Efficient LLM Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing demand for high-performance large language models (LLMs) leads to exponential growth in data computation and model parameter size, resulting in higher costs and time requirements, necessitating a need for efficient resource utilization.
Innovation Solution
A neural computing apparatus comprising multiple chips, cores, or nodes with self-attention, layer normalization, expert, and routing units, where computation outputs are shared through data buses or interfaces, and data is stored in a shared memory, allowing specialized neural computations to be performed efficiently across these units.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If large language models are used to increase natural language processing performance, then processing accuracy is improved, but data computation amount and model parameter size increase exponentially
Solution Approach 1:
The neural network model is divided into multiple expert units, each handling specific computations. The routing unit directs different parts of the computation to different expert units, enabling parallel processing and reducing the computational burden on any single unit while maintaining overall model performance.
Solution Approach 2:
The patent introduces a routing dimension that selects which expert units to activate based on the input data characteristics. This adds a selection layer to the traditional feedforward architecture, allowing the system to dynamically configure computation paths and reduce unnecessary computations.
2Measurement precision
If more model parameters are used to maintain performance, then processing accuracy is improved, but cost and time requirements increase
Solution Approach 1:
Instead of activating all expert units for every computation, the routing unit selects only the necessary expert units based on the input data. This partial action approach reduces computation time while maintaining accuracy by applying full model capacity only when needed.
3Productivity
If multiple chips are used to perform neural computations, then processing capability is improved, but device complexity increases
Solution Approach 1:
Each chip contains the full set of functional units (self-attention unit, layer normalization unit, expert units, and routing unit), making them universally capable of performing any neural computation task. This multi-functionality allows for simplified system architecture where identical modules can be deployed across multiple chips.
Solution Approach 2:
The patent combines multiple chips into a unified system where each chip operates as an independent but coordinated unit. The routing units across chips work together to distribute computations, and shared memory enables data exchange, effectively merging the capabilities of individual chips into a cohesive system.
Data Source
AI summary
A neural computing apparatus includes a plurality of chips, a data bus for data transmission and reception between the plurality of chips, a memory that is accessible to the plurality of chips and that stores data, and a controller that controls the plurality of chips, in which each of the plurality of chips includes a self-attention unit that computes an attention score for an input sequence, a layer normalization unit that performs a layer-level normalization computation, an expert unit that performs a neural computation, and a routing unit that selects an expert unit suitable for a specific neural computation.