Attention Token Pruning in Diffusion Image Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Generative machine learning models for image generation are inefficient in addressing the challenges of high computational efficiency and latency, particularly in real-time applications, due to their high computational complexity and latency, particularly in real-time applications.
Innovation Solution
The proposed solution is a method and apparatus for efficient image generation using diffusion models that leverage attention mechanisms to identify and prune less important tokens during the denoising process, utilizing attention maps to assign importance scores and a similarity-based copy mechanism to maintain image quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If generative machine learning models are used for image generation, then image quality and diversity are maintained, but computational cost and latency increase
Solution Approach 1:
The patent extracts and removes less important tokens from the set of tokens processed by the attention layer. By identifying and eliminating redundant or low-importance tokens based on attention scores, the model reduces computational overhead while preserving the essential information needed for high-quality image generation.
Solution Approach 2:
The patent applies different processing strategies to different tokens based on their importance. High-importance tokens are processed in full detail, while low-importance tokens are pruned or processed with reduced complexity. This local differentiation allows the model to maintain image quality where needed while reducing computational cost in less critical areas.
2Reliability
If generative machine learning models are used for image generation, then image quality and diversity are maintained, but latency increases
Solution Approach 1:
The patent removes less important tokens from processing in real-time during image generation. By extracting and eliminating redundant tokens before they enter the computationally intensive denoising process, the model significantly reduces latency while preserving the quality of generated images through selective token retention.
Solution Approach 2:
The patent performs token pruning as a preliminary action before the main denoising process. By identifying and removing low-importance tokens in advance, based on attention map analysis, the model prepares a reduced set of tokens that will be processed in subsequent steps, thereby reducing overall generation latency without compromising image quality.
3Productivity
If token pruning is applied to reduce computational cost, then computational efficiency improves, but image quality may deteriorate
Solution Approach 1:
The patent applies selective pruning based on local importance metrics derived from attention maps. Rather than uniformly pruning tokens, the model identifies regions and tokens with high importance to image quality and preserves them, while pruning only those with low importance. This localized quality preservation ensures computational efficiency gains without deteriorating overall image quality.
Solution Approach 2:
The patent uses attention maps as feedback to guide the pruning process. The attention mechanism provides information about which tokens are most important for maintaining image quality, and this feedback is used to make informed decisions about which tokens to prune. This feedback loop ensures that pruning decisions are based on actual impact on image quality, preventing deterioration.
Data Source
AI summary
A method, apparatus, non-transitory computer readable medium, apparatus, and system for image processing include obtaining an input prompt; generating a plurality of tokens for an attention layer of a generative machine learning model based on an intermediate noise map; generating, using the attention layer, an attention map based on the plurality of tokens; pruning the plurality of tokens based on the attention map to obtain a pruned set of tokens; denoising the intermediate noise map based on the pruned set of tokens to obtain a denoised map; and generating a synthetic image based on the denoised map.