LLM Layer Buffer Prefetching for Memory Bandwidth Utilization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language models (LLMs) face inefficiencies in inference processes due to memory bandwidth limitations, leading to low average system utilization of both compute and memory bandwidth, particularly during the token decode phase where memory bandwidth utilization is low and compute utilization is high.
Innovation Solution
Implementing a layer buffer that prefetches data from external memory into a buffer before processing, allowing compute engines to read from the buffer during the attention phase, thereby reducing latency and increasing memory bandwidth utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is fetched from external memory during processing, then memory bandwidth utilization is improved, but latency increases due to memory access time
Solution Approach 1:
The system performs preliminary action by prefetching data from external memory into the layer buffer before the compute engine actually needs it. The prefetching operation is initiated in advance during periods when compute resources are available, so that when the compute engine requires the data, it is already resident in the fast layer buffer, eliminating latency without reducing overall productivity.
Solution Approach 2:
The system maintains continuity of useful action by overlapping the prefetching operation with compute operations. While the compute engine is processing data, the prefetching mechanism continuously loads the next batch of data from external memory into the layer buffer. This ensures that the compute engine never waits for memory access, maintaining continuous productive action while maximizing memory bandwidth utilization.
2Productivity
If a layer buffer is introduced to prefetch data, then system utilization is improved, but device complexity increases
Solution Approach 1:
The layer buffer serves as an intermediary component between external memory and the compute engine. It mediates the data flow by receiving prefetched data from external memory and supplying it to the compute engine on demand. This intermediary layer simplifies the overall system architecture by providing a clear separation between memory management functions and compute functions, making the system easier to manage despite the added component.
Solution Approach 2:
The layer buffer is designed with multi-functionality to justify its addition to the system. It serves multiple purposes: (1) as a prefetch target for incoming data, (2) as a fast storage location for the compute engine, (3) as a buffer that can be cleared and reused for subsequent data, and (4) as a mechanism for managing data flow timing. This multi-functionality maximizes the utility of the added component, improving system utilization without proportionally increasing complexity.
Data Source
AI summary
Examples described herein relate to a processor to process constant weight values and key value entries associated with a first transformer kernel of a large language model (LLM) neural network and a circuitry. The circuitry is to: during processing of the constant weight values and key value entries associated with the first transformer kernel of the LLM neural network, pre-fetch constant weight values and key value entries associated with a second transformer kernel of the LLM neural network into a buffer.