Digital In-Memory Compute Chiplets for Low-Latency LLM Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional processing units and accelerators fall short in addressing the high compute throughput, memory capacity, memory bandwidth, low precision numeric support, and scalability with low-latency, high-bandwidth interconnects required by Large Language Models (LLMs) and reasoning models, leading to under-utilization and high system footprint, which negatively impact economic viability and user experience.
Innovation Solution
The use of chiplet devices with digital in-memory compute (DIMC) engines, block floating point numerics, and large high-bandwidth on-chip memories, along with efficient die-to-die interconnects, to accelerate neural network model computations, optimizing for latency-bounded throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional processing units and accelerators are used for LLM inference, then the system can process workloads with standard architecture, but compute throughput is insufficient and memory bandwidth cannot meet the intense needs of auto-regressive workloads
Solution Approach 1:
The system is divided into multiple chiplet devices that can be independently manufactured and then assembled together. Each chiplet contains specialized in-memory compute engines that process LLM inference workloads, allowing the system to scale throughput by adding more chiplets while maintaining a manageable individual component size.
Solution Approach 2:
The patent combines memory and compute functions into integrated in-memory compute engines within each chiplet. This merging eliminates the traditional von Neumann bottleneck by allowing data to be processed directly where it is stored, dramatically improving memory bandwidth utilization and compute throughput for LLM inference workloads.
2Productivity
If conventional accelerators are used, then the system can operate with standard memory architecture, but memory bandwidth is insufficient for the intense memory access requirements of LLM inference
Solution Approach 1:
The memory subsystem and compute engines are merged into integrated in-memory compute blocks within each chiplet. This allows data to remain in memory while being processed, eliminating frequent data transfers between separate memory and compute units, thereby achieving high memory bandwidth utilization without requiring overly complex external memory architectures.
Solution Approach 2:
Each chiplet is designed with localized in-memory compute engines that process data locally where it is stored. This local processing approach optimizes memory bandwidth utilization for each compute task without requiring global memory architecture changes, balancing performance gains with architectural manageability.
3Productivity
If conventional processing units are used for LLM inference, then the system can use standard precision arithmetic, but low precision numeric support is insufficient leading to under-utilization of resources
Solution Approach 1:
The in-memory compute engines are designed to dynamically adapt their numeric precision based on the specific LLM inference workload being executed. The system can switch between different precision modes (e.g., low precision for certain operations, higher precision for others) to optimize resource utilization while maintaining adequate accuracy for the auto-regressive nature of LLM inference.
Solution Approach 2:
The system changes numeric precision parameters adaptively during inference operations. By adjusting precision levels based on workload requirements, the in-memory compute engines achieve better resource utilization without sacrificing the measurement precision needed for accurate LLM output generation.
4Productivity
If conventional accelerators are used, then the system can maintain simple interconnect architecture, but scalability with low-latency, high-bandwidth interconnects is insufficient for distributed LLM processing
Solution Approach 1:
The system uses segmented chiplet devices that can be independently scaled and assembled. Each chiplet maintains a standardized interface, allowing the overall system to scale by adding more identical or heterogeneous chiplets without redesigning the entire interconnect architecture, thus achieving scalability with manageable complexity.
Solution Approach 2:
The chiplet interconnect architecture is designed with universal interfaces that can handle multiple functions including data transfer, synchronization, and coordination of distributed LLM inference operations. This multi-functional approach enables scalability without requiring separate specialized interconnect structures for each function.
Data Source
AI summary
An apparatus using in-memory compute (IMC) chiplet devices for inference-time compute acceleration. The apparatus is configured to accelerate the workload computations for neural network models, such as those for Large Language Models (LLMs) and reasoning models. The apparatus achieves high throughput and low latency using a chiplet design, digital IMC (DIMC) based engines, efficient die-to-die (D2D) interconnects, block floating point (BFP) numerics, and large high bandwidth on-chip memories. With modular chiplets and efficient interconnects, the accelerator apparatus can be easily scaled to accelerate workloads for models of different sizes. The DIMC configuration within the chiplet slices also improves computational performance and reduces power consumption by integrating computational functions and memory fabric. And by dynamically switching between precision levels based on real-time analysis of a target workload, computational efficiency can be optimized while maintaining the necessary level of accuracy for each step of the workload computation.