Storage Block Rearrangement for Transformer Decoder IO Bottleneck
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The transformer model's decoder experiences an IO bottleneck during the rearrangement of key (K) and value (V) information due to its large sizes, which affects processing efficiency and time in natural language processing tasks.
Innovation Solution
The solution involves dividing the storage unit into N storage blocks, each associated with successive time steps, and rearranging intermediate variables within these blocks based on candidate output sequences, reducing the amount of IO required for rearrangement and eliminating the need for additional storage space.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the decoder caches and rearranges key (K) and value (V) information for beam search decoding, then the decoding accuracy is improved, but the IO bottleneck worsens due to large sizes of K and V
Solution Approach 1:
The patent divides the storage unit into multiple storage blocks, where each block stores intermediate variables for a specific range of time steps. This segmentation allows the system to process and rearrange smaller subsets of data independently, reducing the overall IO burden while maintaining the necessary caching functionality for accurate beam search decoding.
2Measurement precision
If the decoder rearranges K/V information according to best beams, then the decoding performance is improved, but the IO time increases due to reading and writing large data sizes
Solution Approach 1:
The patent performs preliminary organization of intermediate variables by storing them in time-step-range-based storage blocks during the encoding phase. This preliminary action ensures that when beam search decoding requires rearrangement, the data is already partially organized, reducing the time needed for rearrangement operations.
3Adaptability or versatility
If additional storage space is allocated for rearrangement operations, then the rearrangement flexibility is improved, but the memory requirements increase
Solution Approach 1:
The storage blocks serve multiple functions: they cache intermediate variables during encoding, store data for beam search decoding, and facilitate rearrangement operations. This multi-functionality eliminates the need for separate dedicated storage spaces for each operation, reducing overall memory requirements while maintaining operational flexibility.
Data Source
AI summary
The present disclosure discloses a data processing apparatus, a method, and related products. The data processing apparatus is used as a computing apparatus and is included in a combined processing apparatus. The combined processing apparatus further includes an interface apparatus and other processing apparatus. The computing apparatus interacts with other processing apparatus to jointly complete a computing operation specified by a user. The combined processing apparatus further includes a storage apparatus. The storage apparatus is respectively connected to the computing apparatus and other processing apparatus and is configured to store data of the computing apparatus and other processing apparatus. The solution of the present disclosure reduces IO time at runtime and memory requirements by means of block local rearrangement.