Frequency-Domain ViT Token Pruning with FFT for Lower Power
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Vision transformer (ViT) models face challenges in efficiently utilizing limited hardware resources due to high computation and power consumption, making them difficult to implement in mobile devices and autonomous vehicles without compromising network accuracy.
Innovation Solution
Perform token pruning in the frequency domain by converting tokens from a spatial domain to a frequency domain using a Fast Fourier Transform (FFT)-based algorithm, pruning high-frequency components, and converting the remaining tokens back to the spatial domain to reduce computation and power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If ViT models use more parameters and computation amount to improve performance, then accuracy is improved, but power consumption increases severely
Solution Approach 1:
The patent extracts and removes high-frequency token components from the transformer model through FFT-based frequency domain conversion. By identifying and eliminating tokens with high-frequency characteristics that contribute less to overall model performance, the method reduces computation amount and power consumption while preserving the essential low-frequency components that maintain accuracy.
Solution Approach 2:
The patent transforms tokens from spatial domain to frequency domain using FFT, changing the representation parameters of the input data. This parameter transformation enables frequency-based pruning where tokens are selected and removed based on their frequency characteristics rather than spatial position, allowing efficient reduction of computation while maintaining performance.
2Use of energy by moving object
If pruning tokens is performed to reduce computation amount, then power consumption is reduced, but accuracy may decrease
Solution Approach 1:
The patent changes the domain parameter from spatial to frequency domain through FFT transformation. This enables pruning decisions to be made based on frequency importance rather than spatial position, allowing the model to remove computationally redundant high-frequency tokens while preserving the low-frequency tokens that carry the most important information for maintaining accuracy.
Solution Approach 2:
The patent applies different treatment to different frequency components of tokens. Low-frequency tokens are preserved as they contain essential information, while high-frequency tokens are pruned as they contribute less to performance. This local quality differentiation based on frequency characteristics enables selective pruning that maintains accuracy while reducing power consumption.
3Productivity
If high frequency components are removed through frequency conversion, then computation amount is reduced, but image information may be lost
Solution Approach 1:
The patent extracts only the essential low-frequency components from the token representation while removing high-frequency components. This selective extraction preserves the most important image information contained in low-frequency tokens while discarding the high-frequency tokens that contribute less to overall performance, thereby reducing computation amount without significant information loss.
Solution Approach 2:
The patent transforms the token representation from spatial domain to frequency domain, changing the parameter space in which information is represented. This parameter change enables the model to identify and remove redundant high-frequency information while preserving essential low-frequency information, achieving faster inference with minimal information loss.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach reduces computation and power consumption while maintaining accuracy, enabling efficient use of ViT models in resource-constrained devices with minimal accuracy loss and reduced CO2 emissions.
Implementation Method 1
converting tokens based on the plurality of patches from a spatial domain to a frequency domain through a fast Fourier transform (FFT)-based frequency domain conversion algorithm
Data Source
AI summary
According to various embodiment of the present disclosure, an electronic device for performing token pruning in a frequency domain may include a processor, and the processor may configured to divide an image frame into a plurality of patches, convert tokens based on the plurality of patches from a spatial domain to a frequency domain through a fast Fourier transform (FFT)-based frequency domain conversion algorithm, the tokens being output from a predetermined transformer block among a plurality of transformer blocks, after performing patch embedding on the plurality of patches, and convert the remaining tokens other than specific tokens from the frequency domain to the spatial domain, after pruning specific tokens from the tokens based on frequency information of the tokens converted into the frequency domain. Various other embodiments are also possible.