Stacked IMC Chiplets for Low-Latency LLM Inference Scaling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional processing units and accelerators fall short in addressing the high compute throughput, memory capacity, memory bandwidth, low precision numeric support, and scalability with low-latency, high-bandwidth interconnects required by Large Language Models (LLMs) and reasoning models, leading to under-utilization and high deployment costs.
Innovation Solution
The use of chiplet devices with digital in-memory compute (DIMC) engines, block floating point numerics, and large high-bandwidth on-chip memories, coupled through efficient die-to-die interconnects, to accelerate neural network model computations, optimizing for latency-bounded throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional processing units and accelerators are used, then device compatibility is maintained, but compute throughput and memory bandwidth are insufficient for LLM workloads
Solution Approach 1:
The system is divided into multiple chiplet devices, each containing compute engines and memory structures. These chiplets can be independently manufactured and then interconnected through an interposer substrate, allowing the system to achieve high compute throughput while maintaining modular complexity management
Solution Approach 2:
The patent transitions from conventional two-dimensional planar architectures to three-dimensional stacked architectures using interposer substrates. This vertical stacking enables significantly higher memory bandwidth and compute throughput by reducing data access distances and enabling parallel data flows across multiple layers
2Quantity of substance
If memory capacity is increased to support large models, then model size is improved, but memory bandwidth and access latency worsen
Solution Approach 1:
Memory is segmented into multiple memory structures distributed across different chiplet devices. Each chiplet has its own local memory structures, reducing the need for long-distance memory accesses and improving bandwidth by enabling parallel access to multiple memory banks simultaneously
Solution Approach 2:
The interposer substrate acts as an intermediary that provides high-speed interconnects between memory structures on different chiplets. This intermediary enables fast data transfer between distributed memory structures without the bottlenecks of conventional memory controllers
3Measurement precision
If precision is maintained for accurate inference, then computational accuracy is improved, but power consumption increases
Solution Approach 1:
Different chiplet devices can operate at different precision levels based on their specific computational tasks. Some chiplets maintain full precision for critical operations while others use lower precision for less sensitive computations, optimizing the overall power-accuracy tradeoff of the system
Data Source
AI summary
A stacked apparatus using in-memory compute (IMC) chiplet devices for inference-time compute acceleration. The apparatus is configured to accelerate the workload computations for neural network models, such as those for Large Language Models (LLMs) and reasoning models. The apparatus achieves high throughput and low latency using a chiplet design, digital IMC (DIMC) based engines, efficient die-to-die (D2D) interconnects, block floating point (BFP) numerics, and large high bandwidth on-chip memories. With modular chiplets in stacked configurations with memory devices and efficient interconnects, the accelerator apparatus can be easily scaled to accelerate workloads for models of different sizes. The DIMC configuration within the chiplet slices also improves computational performance and reduces power consumption by integrating computational functions and memory fabric. And by dynamically switching between precision levels based on real-time analysis of a target workload, computational efficiency can be optimized while maintaining accuracy.