Vision Transformer Training With Decoupled Weight Decay
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing Vision Transformer (ViT) neural networks face challenges in achieving state-of-the-art performance on downstream tasks while maintaining computational efficiency, particularly when scaled up with increased parameters.
Innovation Solution
Modifications to the ViT architecture and training process, including decoupling weight decay values, generating classification outputs from all output elements without extra positional inputs, and storing momentum values with reduced precision, allow for effective training of large-scale ViT networks that improve performance on downstream tasks without excessive resource consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If ViT model size is increased to achieve better downstream task performance, then classification accuracy improves, but computational resource consumption and memory usage increase excessively
Solution Approach 1:
The patent changes key training parameters including decoupling weight decay values for different parameter groups, using reduced precision (e.g., bfloat16) for momentum values in optimizers, and adjusting learning rate schedules. These parameter changes enable larger model sizes to be trained efficiently without proportionally increasing computational resource consumption, thus improving classification accuracy while controlling energy usage.
2Measurement precision
If ViT model size is increased to achieve better downstream task performance, then classification accuracy improves, but memory usage increases excessively
Solution Approach 1:
The patent employs reduced precision storage for momentum values in optimizer states, using formats like bfloat16 instead of full precision floating point. This parameter change reduces the memory footprint of training state variables, enabling larger model sizes that improve classification accuracy without proportionally increasing memory usage.
3Stability of the object's composition
If weight decay is applied uniformly to all parameters, then training stability is maintained, but performance on downstream tasks deteriorates
Solution Approach 1:
The patent applies different weight decay values to different parameter groups within the model. Specifically, it decouples weight decay for transformer block parameters versus other parameters, allowing each group to have optimized regularization strength. This local differentiation maintains training stability for core transformer operations while improving downstream task performance by adjusting regularization appropriately for different parameter types.
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training Vision Transformer (ViT) neural networks.