Vision Transformer Head Pruning for Lightweight Deployment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Vision transformers require a large number of parameters, making them unsuitable for deployment on lightweight devices due to increased model size and computational resources, which existing pruning methods for other models are ineffective for.
Innovation Solution
Structured pruning of vision transformers using multi-granularity sparsity across attention heads and MLP layers, employing a graph-based approach to rank importance and a soft-pruning strategy with optimization techniques to maintain model accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If vision transformers use many parameters to define the model, then model accuracy and performance are improved, but model size and computational resources increase significantly
Solution Approach 1:
The patent segments the vision transformer model into multiple attention heads and MLP layers, applying structured pruning at different granular levels. Each attention head can be independently pruned, allowing selective removal of less important parameters while preserving the overall model architecture and performance.
Solution Approach 2:
The patent applies local quality by differentiating the importance of different parameters within the model. Not all parameters are treated equally - instead, parameters are ranked by importance using graph-based methods, and pruning is applied selectively to less important parameters while retaining critical ones.
2Reliability
If vision transformers use many parameters to define the model, then model accuracy and performance are improved, but deployment on lightweight devices becomes unsuitable
Solution Approach 1:
The patent segments the vision transformer model into multiple attention heads and MLP layers, applying structured pruning at different granular levels. Each attention head can be independently pruned, allowing selective removal of less important parameters while preserving the overall model architecture and performance.
Solution Approach 2:
The patent changes the parameter configuration of the vision transformer by reducing the number of parameters through structured pruning. The model transitions from a dense parameter configuration to a sparse one, where only the most important parameters are retained, making it suitable for deployment on lightweight devices.
3Quantity of substance
If existing pruning methods for other models are applied to vision transformers, then model size is reduced, but pruning effectiveness is insufficient
Solution Approach 1:
The patent applies local quality by differentiating the importance of different parameters within the model. Not all parameters are treated equally - instead, parameters are ranked by importance using graph-based methods, and pruning is applied selectively to less important parameters while retaining critical ones.
Solution Approach 2:
The patent introduces a dynamic pruning strategy where the pruning process is iterative and adaptive. The model is pruned in stages with fine-tuning between stages, allowing the pruning to adapt to the specific characteristics of vision transformers and maintain performance while reducing size.
Data Source
AI summary
In one embodiment, a method includes accessing a batch B of a plurality of images, wherein each image in the batch is part of a training set of images used to train a vision transformer comprising a plurality of attention heads. The method further includes determining, for each attention head A, a similarity between (1) the output of the attention head evaluated using each image in the batch and the (2) output of each attention head evaluated using each image in the batch. The method further includes determining, based on the determined similarities, an importance score for each attention head; and pruning, based on the importance scores, one or more attention heads from the vision transformer.