Text Inference Batching Without Padding for Lower GPU Load
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The consumption of GPU resources is high due to the need to invoke a large number of model parameters for each piece of text data during inference, particularly in large language models like GPT3, and the use of padding characters to align text data lengths leads to invalid calculations and inefficient resource utilization.
Innovation Solution
A method that concatenates initial token sequences from multiple pieces of text data into a single concatenated token sequence without padding, allowing inference on this combined sequence to generate reply data for each piece of text data, thereby reducing the need for redundant GPU resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If padding characters are added to align text data lengths, then text data can be processed in batches, but invalid calculations occur and GPU resource utilization efficiency decreases
Solution Approach 1:
The patent segments the concatenated token sequence by inserting special separator tokens between different text inputs. This allows the model to process multiple texts in a batch without requiring padding, as each text is clearly delimited. The segmentation principle resolves the contradiction by enabling batch processing (improving productivity) while avoiding the invalid calculations associated with padding (reducing energy loss).
Solution Approach 2:
The patent changes the parameter of sequence length handling by abandoning the fixed-length padding approach in favor of variable-length sequences separated by special tokens. This parameter change allows the system to process texts of different lengths efficiently in batches without performing unnecessary calculations on padded positions, thus improving both productivity and reducing GPU resource consumption.
2Productivity
If multiple pieces of text data are processed in the same batch, then GPU resource utilization efficiency improves, but the complexity of managing different text lengths increases
Solution Approach 1:
The patent introduces special separator tokens as intermediaries between different text inputs in a batch. These separator tokens act as delimiters that clearly mark the boundaries of each text input, allowing the model to process variable-length sequences without complex padding management. This intermediary approach simplifies batch management while maintaining high GPU resource utilization efficiency.
3Stability of the object's composition
If padding characters are supplemented to relatively short text data, then all text data in a batch have consistent lengths, but a large number of invalid calculations are performed
Solution Approach 1:
The patent extracts and removes the harmful effect of padding by using variable-length sequences with special separator tokens instead. This extraction principle eliminates the invalid calculations generated by padding characters while maintaining the necessary structure for batch processing. The system achieves length consistency not through padding but through uniform separator placement, thereby removing the harmful factor of invalid calculations.
Data Source
AI summary
The present disclosure discloses a text data inference method and apparatus, a storage medium, and an electronic device. The method includes the following operations: acquiring initial token sequences corresponding to N pieces of text data, N being an integer greater than 1, and one token in each initial token sequence characterizing one character in corresponding text data; concatenating N initial token sequences into a concatenated token sequence; and performing inference on the concatenated token sequence to obtain N reply token sequences, the N reply token sequences being configured for generating reply data of the N pieces of text data, and a starting token in each reply token sequence being obtained by: performing, in the concatenated token sequence, inference on tokens belonging to the same initial token sequence.