Multi-Die Neural Network IC With EMIB Interconnect for Low-Latency AI
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AI inference technologies face challenges with high computation costs, power consumption, latency issues, and physical space constraints when deploying large neural network models, particularly in edge computing and IoT applications, due to limitations in current hardware architectures like GPUs, NPUs, CPUs, dedicated accelerators, and FPGAs.
Innovation Solution
Embedding a neural network on an IC device with distinct dies and a fabric that connects them, utilizing an EMIB fabric for high-bandwidth, low-latency interconnects, separating computations into embedding, attention, and base dies, optimizing each for specific tasks and integrating memory and computation within the device.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If DNNs are deployed on existing hardware architectures (GPUs, NPUs, CPUs, dedicated accelerators, FPGAs), then high accuracy AI processing is achieved, but computation cost, power consumption, and latency increase significantly
Solution Approach 1:
The patent segments the neural network processing into distinct functional blocks distributed across multiple dies: embedding layer die, attention layer die, and base die. Each die is optimized for specific computational tasks, reducing unnecessary data movement and computation. The embedding layer die handles embedding lookups and transformations, the attention layer die processes attention mechanisms, and the base die handles remaining computations, thereby reducing overall power consumption while maintaining accuracy.
Solution Approach 2:
The patent transitions from a single-die or few-die architecture to a multi-die stacked architecture connected via high-bandwidth interconnects. This dimensional expansion allows parallel processing across multiple specialized dies, reducing computation time and power consumption per operation while maintaining the required AI processing accuracy through coordinated multi-die operation.
2Measurement precision
If DNNs are deployed on existing hardware architectures, then high accuracy is achieved, but physical space constraints are exceeded
Solution Approach 1:
The patent divides the large-scale neural network into modular functional blocks that can be distributed across multiple smaller dies. Each die contains only the necessary computational units for its specific function (e.g., embedding lookups, attention computations), significantly reducing the area required on each individual die compared to implementing the entire network on a single chip.
Solution Approach 2:
The patent implements a hierarchical nested structure where multiple specialized dies are stacked and interconnected to form a complete neural network processing system. Each die is nested within the overall system architecture, with smaller functional units nested within larger computational blocks, allowing efficient space utilization across the multi-die package.
3Measurement precision
If DNNs are deployed on existing hardware architectures, then high accuracy is achieved, but latency increases
Solution Approach 1:
By segmenting the neural network into specialized functional blocks across multiple dies, the patent enables parallel processing of different neural network operations simultaneously. The embedding layer, attention layer, and base computations can occur in parallel across different dies, significantly reducing the overall processing latency compared to sequential processing on a single processor.
Solution Approach 2:
The patent introduces high-bandwidth interconnects as intermediaries between the multiple dies, enabling fast data transfer and communication. These specialized interconnect structures reduce the communication overhead and latency associated with data movement between functional blocks, maintaining low latency despite the distributed architecture.
4Productivity
If DNNs are deployed on existing hardware architectures, then computation is performed, but computation cost and power consumption increase
Solution Approach 1:
The patent optimizes each die with specialized computational units and data structures tailored to its specific function. The embedding layer die uses optimized lookup tables and transformation units, while the attention layer die uses specialized attention computation units. This local optimization reduces the computational energy cost by performing only necessary operations in each location rather than general-purpose computation everywhere.
Solution Approach 2:
The patent merges computational units and data storage within each functional die to minimize data movement between memory and processing units. By combining related computational functions and their associated data structures within the same die, the system reduces the energy cost of data transfer while maintaining high computation throughput.
Data Source
AI summary
An integrated circuit (IC) device may implement a neural network model. The IC device may include stacked embedding dies, stacked attention dies, and a base die. The embedding dies may perform embedding computations in the model. Each embedding die may have an embedding dot unit that includes memories for storing precomputed embedding vectors, multiply units for performing multiplication operations on embeddings, add units for summing the results of the multiplication operations. The attention dies may perform attention computations in the model. Each attention die may have an attention dot unit that includes memories for storing intermediate values, multiply units for performing multiplication operations for attention mechanisms, add units for summing the results of the multiplication operations. The base die may coordinate the overall operation of the model and perform preprocessing, embedding, normalization, activation, and final output generation. Micro-bumps may provide electrical connections between the stacked dies, facilitating inter-die communication.