Near-Memory Token Generation for Parallel LLM Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computing systems that implement large language models face inefficiencies in processing resources due to serial processing of output tokens, which limits parallel processing capabilities, especially when using graphics processing units (GPUs) or multi-threaded processors.
Innovation Solution
Implementing near-memory computing (NMC) by storing base parameters of the large language model in a memory system and using an assistant model with lower fidelity to generate predicted tokens, which are then processed in parallel by the memory system, reducing the host system's processing load and improving efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If serial processing of output tokens is used in large language models, then processing simplicity is maintained, but processing speed and resource utilization are limited
Solution Approach 1:
The patent segments the parameter set into multiple subsets, each stored in separate memory devices. This allows parallel processing of different parameter subsets across multiple memory devices, transforming the serial processing bottleneck into a parallel architecture that can generate multiple output tokens simultaneously, thereby improving token generation speed while managing complexity through structured division
Solution Approach 2:
The patent transitions from a single-dimensional serial processing approach to a multi-dimensional parallel processing architecture by distributing parameter subsets across multiple memory devices and processing channels. This dimensional expansion enables simultaneous processing of multiple tokens through different parameter subsets, significantly increasing throughput without linearly increasing system complexity
2Speed
If full-precision parameters are used in memory system, then model accuracy is maintained, but memory bandwidth and processing speed are reduced
Solution Approach 1:
The patent applies different precision levels to different parameter subsets stored in various memory devices. Certain memory devices store full-precision parameters for critical computations, while others store lower-precision approximations for less critical operations. This local quality differentiation allows the system to optimize for speed in non-critical paths while maintaining accuracy where needed, resolving the contradiction between processing speed and parameter fidelity
Solution Approach 2:
The patent dynamically adjusts parameter precision levels based on computational requirements. By changing the precision parameter of stored values in memory devices according to the specific computation being performed, the system can switch between high-precision and low-precision modes, enabling faster processing when full precision is not required while maintaining accuracy when it is, thus balancing speed and fidelity
Data Source
AI summary
In some implementations, a memory system may obtain, from a host system, a first command indicating a prompt associated with a large language model. The memory system may generate, based on the prompt, one or more first tokens using one or more first parameters, the one or more first parameters having a first fidelity and the one or more first parameters based on one or more second parameters associated with the large language model, the one or more second parameters having a second fidelity. The memory system may provide the one or more first tokens to the host system.