Autoregressive Inference Pipeline With Near-Compute Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Autoregressive inference is challenged by its inherently serial structure, which limits high performance, low latency, and high throughput, and memory bandwidth issues exacerbate processing time, compromising computational accuracy.
Innovation Solution
A pipeline execution model using compute accelerators with a near-compute memory architecture, where model layers are sequentially dependent and memory is available locally to each processing element, reducing communication path lengths and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If autoregressive inference uses traditional serial processing architecture, then computational accuracy is maintained, but processing throughput is limited and latency increases
Solution Approach 1:
The patent segments the model processing into multiple independent stages (e.g., different model layers or processing steps) that can be executed in parallel across multiple compute accelerators. Each stage processes a portion of the computational workload, allowing simultaneous execution and thereby increasing throughput while maintaining accuracy through proper sequential dependency management between stages.
Solution Approach 2:
The patent transitions from a single-processor serial execution model to a multi-processor parallel execution model, adding the dimension of spatial distribution across multiple compute accelerators. This dimensional expansion enables concurrent processing of different model stages, significantly improving throughput and reducing latency while preserving computational accuracy through coordinated data flow management.
2Quantity of substance
If off-site memory is used for model data storage, then memory capacity is sufficient, but memory bandwidth limits processing performance
Solution Approach 1:
The patent implements a nested memory hierarchy where on-chip memory (fast, high bandwidth) is nested within each compute accelerator, and off-site memory (larger capacity) is nested at the system level. Frequently accessed model data is cached in the on-chip memory of relevant compute accelerators, providing high-speed access for critical operations while maintaining sufficient total capacity through the hierarchical structure.
Solution Approach 2:
The patent introduces on-chip memory as an intermediary between the compute accelerators and off-site memory. This intermediary layer buffers and caches model data, reducing the frequency and volume of accesses to off-site memory. The on-chip memory acts as a high-speed mediator that satisfies bandwidth requirements for active model data while leveraging the large capacity of off-site memory for complete model storage.
Data Source
AI summary
The present disclosure includes systems and methods for autoregressive inference using one or more compute accelerators. A method includes configuring one or more compute accelerators to implement a processing sequence of a machine learning (ML) model, wherein the ML model includes a plurality of model layers, and wherein the configuring includes mapping, based at least in part on the processing sequence, the plurality of model layers to a plurality of processing regions of the one or more compute accelerators and arranging connections between the plurality of processing regions to form a processing pipeline corresponding to the processing sequence. The method includes, based at least in part on receiving one or more queries, processing, using the ML model, the one or more queries through the processing pipeline.