Batch Inference System for Transformer Model Parameter Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Super-large transformer models consume significant storage resources, making it difficult to deploy them in service systems that need to support multiple tasks in parallel, leading to poor system performance.
Innovation Solution
A batch computing system that includes a memory for storing shared and task-specific model parameters, a task manager, and an inference computer with preprocessing, batch assignment, and inference execution modules, which enables common and task-specific computations to be performed in batches, reducing storage needs and improving performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If super-large transformer models are deployed to support multiple tasks in parallel, then task processing capability is improved, but storage resource consumption increases significantly
Solution Approach 1:
The patent segments the model parameters into shared parameters (common to all tasks) and task-specific parameters (unique to each task). This segmentation allows the system to store only one copy of the shared parameters while maintaining support for multiple tasks, thereby reducing overall storage requirements while preserving task processing capability.
Solution Approach 2:
The shared model parameters serve multiple functions by being common to all tasks. The architecture enables a single set of shared parameters to support various downstream tasks through fine-tuning, making the storage resources more efficient and allowing the system to handle diverse tasks without proportionally increasing storage consumption.
2Speed
If traditional inference service systems process requests from different users in parallel, then system responsiveness is improved, but computation efficiency deteriorates due to inability to batch different tasks
Solution Approach 1:
The patent introduces a dynamic batch assignment mechanism that can flexibly group task requests into batches based on their characteristics. The system can adaptively determine batch sizes and compositions to optimize both responsiveness and computation efficiency, allowing the system to switch between batch processing modes as needed.
Solution Approach 2:
The patent merges multiple task requests into computation batches, combining batched common computations with batched task-specific computations. This merging approach enables the system to process multiple requests simultaneously in an efficient manner, improving both responsiveness and computation efficiency by leveraging parallel processing capabilities.
3Quantity of substance
If parameter-efficient transformer models are used to reduce storage resources, then storage consumption is reduced, but system performance deteriorates due to difficulty in processing different tasks in parallel
Solution Approach 1:
By segmenting parameters into shared and task-specific components, the system achieves both low storage consumption (through shared parameter reuse) and high system performance (through efficient batched processing of task-specific computations). This segmentation resolves the contradiction by enabling compact storage while maintaining parallel processing capability.
Solution Approach 2:
The patent ensures continuous useful action by maintaining the shared model parameters in memory and reusing them across all tasks. The system continuously performs batched common computations and task-specific computations without interruption, ensuring that the reduction in storage resources does not compromise processing performance or parallel task execution capability.
Data Source
AI summary
The present disclosure discloses a batch computing system and an associated method. The batch computing system includes a memory, a task manager and an inference computer. The memory stores a shared model parameter set common to a plurality of tasks that is generated by fine tuning a shared model and a task-specific parameter set of each task. The inference computer receives a plurality of task requests, derives a data length and a designated task of each task request, and enables the task manager to read a task-specific parameter set and a shared model parameter set corresponding to each task request. The inference computer further assigns task requests corresponding to the shared model to a plurality of computation batches, performs, in batch, the common computation of designated tasks in each batch computation, and performs task-specific computation operations of the designated tasks of each batch computation.