Selective Batching for Transformer Inference Variable Lengths
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer neural network models face challenges in processing batches with variable input or target lengths due to existing batching methods that require all requests to have the same length, leading to inefficiencies and unnecessary resource usage.
Innovation Solution
The proposed solution involves selective batching, where the inference system processes a subset of operations individually, such as attention operations, while performing batch operations for other parts, allowing for dynamic modification of batches during execution and enabling processing of requests with variable lengths.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If requests are processed in batches with same length requirement, then processor utilization is improved, but requests with variable lengths cannot be processed efficiently
Solution Approach 1:
The patent segments the processing operations into two categories: batch operations (applied to all requests in the batch) and individual operations (applied to each request separately). This segmentation allows the system to process requests with variable lengths by applying individual operations to each request while maintaining batch operations for common processing steps, thereby resolving the contradiction between batch processing efficiency and variable length adaptability.
Solution Approach 2:
The patent introduces dynamic batch modification capabilities that allow the batch composition to change during execution. Requests can be added to or removed from batches dynamically based on their length requirements and processing status. This dynamic approach enables the system to adapt to variable length requests while maintaining the efficiency benefits of batch processing for compatible requests.
2Adaptability or versatility
If workarounds are used to handle variable length requests, then variable length processing is enabled, but resource usage increases
Solution Approach 1:
The patent merges the processing of multiple requests with variable lengths into a unified batch operation framework. By combining batch operations (applied to all requests) with individual operations (applied to each request), the system achieves variable length processing without the need for separate processing pipelines or redundant computations, thereby reducing overall resource usage compared to traditional workarounds.
Solution Approach 2:
The patent changes the parameter of batch composition from static (fixed at the beginning) to dynamic (modifiable during execution). This parameter change allows the system to optimize resource usage by adjusting the batch to include only compatible requests at each stage, avoiding the computation of padding or placeholder data for variable length requests.
3Loss of time
If batches are modified dynamically, then response times are improved, but batch management complexity increases
Solution Approach 1:
The patent performs preliminary actions by pre-identifying compatible requests for batching and preparing the batch composition before execution begins. This preliminary organization reduces the complexity of dynamic modifications during execution, as the system only needs to manage additions or removals rather than complete reorganization, thereby achieving fast response times with manageable complexity.
Data Source
AI summary
An inference system applies a machine-learning transformer model to a batch of requests with variable input length or variable target length or variable internal state length by selectively batching a subset of operations in the transformer model but processing requests in the batch individually for a subset of operations in the transformer model. In one embodiment, the operation to be processed individually is an attention operation of an encoder or a decoder of the transformer model. By selective batching, the inference system can allow batching operations to be performed for a batch of requests with variable input or target length or internal state length to utilize the parallel computation capabilities of hardware accelerators while preventing unnecessary computations that occur for workarounds that restrain the data of a batch of requests to a same length.