Transformer Model Training Resource Allocation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing transformer model training methods fail to fully utilize computing resources and off-chip memory bandwidth due to varying operation per byte ratios, leading to underutilization of resources during training processes.
Innovation Solution
The method involves dividing batches into micro-batches with no dependency, forming layer pairs, determining resource allocation based on operation per byte ratios, and assigning layers to resource cores or unified vector units to minimize idle time and total operation execution time, thereby optimizing resource utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If batches are processed as single large units in transformer model training, then memory bandwidth utilization may be improved, but computing resource utilization deteriorates due to idle time between operations
Solution Approach 1:
The patent segments large batches into multiple micro-batches that can be processed independently and in parallel. This segmentation allows different micro-batches to be assigned to different computing resources simultaneously, eliminating idle time and improving computing resource utilization while maintaining efficient memory bandwidth usage through coordinated processing of the segmented batches.
2Speed
If computing resources are allocated to layers with high operation per byte ratio, then computing speed is improved, but memory bandwidth utilization deteriorates
Solution Approach 1:
The patent applies local quality by allocating different computing resources to different layers based on their specific operation per byte ratio characteristics. Layers with high operation per byte ratio are assigned to computing-optimized resources, while layers with low operation per byte ratio are assigned to memory-optimized resources, ensuring optimal utilization of both computing speed and memory bandwidth for each specific layer.
3Reliability
If layers are processed sequentially in transformer training, then dependency constraints are satisfied, but total operation execution time increases
Solution Approach 1:
The patent introduces dynamic processing by allowing the processing order and resource allocation to adapt based on layer characteristics and availability. While maintaining the required sequential dependency constraints between layers, the system dynamically assigns different micro-batches to different resources and adjusts processing schedules to minimize idle time, thereby reducing total execution time without violating dependency constraints.
Data Source
AI summary
A device including processors configured to execute instructions and memories storing the instructions, which when executed by the processors configure the processors to perform an operation for training a transformer model having a plurality of encoders and a plurality of decoders by configuring the processors to identify the batches of training data into a plurality of micro-batches, select layer pairs for the plurality of micro-batches, assemble a processing order of the layer pairs, determining resource information to be allocated to the layer pairs, and allocate resources to the layer pairs based on the determined resource information to be allocated to the layer pairs, dependent con the processing order of the layer pairs.