Selective Transformer Batching for Variable-Length Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transformer models struggle with batching requests of variable lengths, leading to inefficient resource usage and increased computational costs due to the need for workarounds that either pad requests to a fixed length or process them individually, which reduces throughput and utilization of hardware accelerators.
Innovation Solution
Implement selective batching for transformer models, where operations like attention are processed individually for requests with variable lengths, allowing dynamic modification of batches during execution to accommodate requests with different input or internal state lengths, thereby optimizing hardware utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If requests are batched together for processing, then throughput and hardware utilization improve, but requests with variable lengths cannot be processed efficiently
Solution Approach 1:
The patent segments the processing of batched requests by separating attention operations from other transformer operations. Within a batch, attention operations are further segmented to process each request individually based on its specific length, while non-attention operations process the entire batch together. This segmentation allows mixed-length requests to be handled efficiently without requiring all requests to have uniform length.
Solution Approach 2:
The patent introduces dynamic batching where the batch composition can change during execution. Requests can be added to or removed from a batch dynamically based on their length requirements and the current processing state. This dynamic approach allows the system to adapt to variable length requests while maintaining high hardware utilization through parallel processing of compatible operations.
2Adaptability or versatility
If requests are processed individually, then variable length requests can be handled accurately, but hardware utilization and throughput decrease
Solution Approach 1:
The patent merges multiple requests into a single batch for processing, but selectively applies different processing strategies to different operations within the batch. Compatible operations (non-attention) are merged and processed together to maximize hardware utilization, while attention operations are processed with individual request characteristics preserved. This selective merging maintains both throughput and adaptability to variable lengths.
3Productivity
If workarounds like padding to fixed length are used, then batching becomes possible, but unnecessary computations increase resource usage
Solution Approach 1:
The patent extracts the attention operation from the uniform batch processing pipeline and handles it separately for each request based on its actual length. This extraction eliminates the need to pad requests to a fixed length, as attention operations naturally accommodate variable lengths. Other non-attention operations continue to process the batch uniformly, maintaining efficiency without unnecessary computations.
Data Source
AI summary
An inference system applies a machine-learning transformer model to a batch of requests with variable input length or variable target length or variable internal sate length by selectively batching a subset of operations in the transformer model but processing requests in the batch individually for a subset of operations in the transformer model. In one embodiment, the operation to be processed individually is an attention operation of an encoder or a decoder of the transformer model. By selective batching, the inference system can allow batching operations to be performed for a batch of requests with variable input or target length or internal state length to utilize the parallel computation capabilities of hardware accelerators while preventing unnecessary computations that occur for workarounds that restrain the data of a batch of requests to a same length.