LLM Inference Acceleration Using Parallel Submodels and Accelerators
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing size of large language models (LLMs) leads to significant computational demands, resulting in prolonged inference times that adversely affect task execution efficiency.
Innovation Solution
Deploy LLMs on multiple hardware accelerators and segment them into submodels, performing first processing on input data through the LLMs and subsequent processing on the results through submodels until a stop inference condition is met, utilizing data and model parallelism to accelerate inference.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the size of the LLM model is increased to improve task execution capability, then the model's processing power and accuracy are improved, but the computational demands increase and inference time is prolonged
Solution Approach 1:
The patent divides the LLM into multiple submodels and deploys them across multiple hardware accelerators. During the prefilling stage, input data is segmented and processed in parallel by multiple LLMs. During the decoding stage, the model is segmented into submodels that process data in parallel. This segmentation enables concurrent processing without sacrificing model accuracy, thereby reducing inference time while maintaining task execution capability.
Solution Approach 2:
The patent transitions from single-model sequential processing to multi-model parallel processing across multiple hardware accelerators. By introducing the dimension of parallelism across multiple devices, the system achieves faster inference speeds while maintaining the computational power of large models. The staged hybrid approach combines data parallelism and model parallelism to optimize inference across different stages.
2Productivity
If LLMs are deployed on multiple hardware accelerators to increase processing capacity, then inference speed is improved, but system complexity increases
Solution Approach 1:
The patent segments the LLM into submodels and distributes them across multiple hardware accelerators, with each accelerator handling specific submodels or data portions. This segmentation enables parallel processing that improves inference speed while managing system complexity through structured distribution of computational tasks across independent units.
Solution Approach 2:
The patent implements dynamic task distribution where the system adapts its processing strategy based on the inference stage. During prefilling, data parallelism is used with multiple LLMs processing segmented input data. During decoding, model parallelism is used with submodels processing results sequentially. This dynamic adaptation optimizes performance while managing complexity through stage-specific strategies.
3Speed
If data parallelism is used during prefilling stage to process input data in parallel, then prefilling speed is improved, but communication overhead between hardware accelerators increases
Solution Approach 1:
The patent segments input data into multiple portions and distributes them across multiple hardware accelerators for parallel prefilling processing. Each accelerator processes its assigned data segment independently, then results are aggregated. This segmentation enables faster prefilling while managing communication overhead through structured data distribution and result collection protocols.
Solution Approach 2:
The patent performs data segmentation and distribution before the actual prefilling computation begins. By preparing the data distribution structure in advance, the system minimizes communication overhead during the computationally intensive prefilling stage, as data is already positioned for parallel processing without requiring dynamic communication during computation.
4Speed
If model parallelism is used during decoding stage to process data through submodels in parallel, then decoding speed is improved, but memory requirements and system coordination complexity increase
Solution Approach 1:
The patent divides the LLM model into multiple submodels and distributes them across different hardware accelerators for parallel decoding processing. Each submodel handles specific portions of the decoding task, enabling faster inference while managing memory requirements through distributed storage and computation across multiple devices.
Data Source
Figure 1
Figure 2~3
Figure 4
AI summary
A large language model (LLM)-based inference acceleration method and apparatus, a medium, and a device are disclosed. The method includes: determining (201) LLMs deployed on a plurality of first hardware accelerators, multiple submodels obtained through segmenting the LLMs and respectively deployed on a plurality of second hardware accelerators, and input data of a to-be-executed task; performing (202) first processing on the input data through the plurality of LLMs to obtain a first processing result; performing (203) second processing on the first processing result through the multiple submodels to obtain a second processing result; and determining (204) an execution result of the to-be-executed task in response to the second processing result meeting a stop inference condition.