LLM Inference Acceleration Using Segmented Submodels and Parallel Hardware
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As large language models (LLMs) increase in size, they face significant computational demands leading to prolonged inference times, adversely affecting task execution efficiency.
Innovation Solution
Deploy LLMs on multiple hardware accelerators and segment them into submodels, performing first and second processing stages in parallel to accelerate inference, utilizing data and model parallelism to improve efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the size of the LLM model increases to handle complex tasks, then the model's processing capability and task coverage are improved, but the computational demands increase and inference time becomes longer
Solution Approach 1:
The patent segments the large language model into multiple submodels and distributes them across multiple hardware accelerators. This segmentation allows the model to process different parts of the input data in parallel, significantly reducing inference time while maintaining the overall processing capability of the original large model.
Solution Approach 2:
The patent transitions from a single-model sequential processing approach to a multi-model parallel processing architecture. By adding the dimension of parallelism across multiple hardware accelerators, the system achieves both high task coverage and reduced inference time simultaneously.
2Adaptability or versatility
If the model size increases to handle more complex tasks, then the processing capability is improved, but the computational demands result in longer inference time
Solution Approach 1:
The large model is divided into multiple submodels that can be executed in parallel on different hardware accelerators. This segmentation maintains the processing capability of the original large model while improving task execution efficiency through parallel computation.
Solution Approach 2:
The patent merges multiple submodel execution results to produce the final output, combining the computational power of multiple smaller models to achieve the processing capability of a single large model with improved efficiency.
3Speed
If parallel processing is implemented across multiple hardware accelerators, then processing speed is increased, but the system complexity increases
Solution Approach 1:
The system segments both the model and the computation tasks across multiple hardware accelerators. This segmentation enables parallel processing and increased speed, while the modular structure makes the complexity manageable through clear division of responsibilities.
Data Source
AI summary
A large language model (LLM)-based inference acceleration method, a medium, and a device are disclosed. The method includes: determining LLMs deployed on a plurality of first hardware accelerators, multiple submodels obtained through segmenting the LLMs and respectively deployed on a plurality of second hardware accelerators, and input data of a to-be-executed task; performing first processing on the input data through the plurality of LLMs to obtain a first processing result; performing second processing on the first processing result through the multiple submodels to obtain a second processing result; and determining an execution result of the to-be-executed task in response to the second processing result meeting a stop inference condition.