Lightweight Text-to-Image Model via Self-Attention Knowledge Distillation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Recent large text-to-image generation models improve performance but result in increased model size and processing time, necessitating a technology to alleviate these burdens.
Innovation Solution
A method and apparatus for generating a lightweight model by pruning and changing transformer block groups in a text-to-image generation model, using self-attention knowledge distillation to train the lightweight model, and transferring feature maps from a teacher model to a compression model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If the model size is increased to improve image generation quality, then the image generation quality is improved, but the processing time increases
Solution Approach 1:
The patent segments the U-Net model into multiple transformer block groups that can be independently pruned. By dividing the model into manageable blocks, the patent enables selective removal of less critical blocks while preserving essential functionality, thus reducing overall model size and processing time while maintaining acceptable image generation quality.
Solution Approach 2:
The patent changes the depth parameter of transformer block groups from 10 to a reduced value (e.g., 6 or 5) to create lightweight variants. This parameter modification directly reduces the number of parameters in the model, decreasing both model size and processing time while the knowledge distillation process ensures quality is preserved.
2Manufacturing precision
If the number of parameters is increased to enhance image generation quality, then the image generation quality is improved, but the model size increases
Solution Approach 1:
The patent extracts and removes less critical transformer block groups from the U-Net model structure. By identifying and removing redundant or less important blocks (particularly in DOWN-2, DOWN-3, UP-1, and UP-2 stages), the patent significantly reduces model size while the knowledge distillation from the full model ensures the remaining structure retains essential generation capabilities.
Solution Approach 2:
The patent creates lightweight copy variants of the original SDXL model by pruning specific transformer blocks. These copied models (KD-SDXL-1B, KD-SDXL-700M) replicate the essential functionality of the full model with reduced parameters, providing a space-efficient alternative that maintains image generation quality through knowledge distillation training.
3Productivity
If the processing speed is increased by reducing model size, then the processing speed is improved, but the image generation quality may deteriorate
Solution Approach 1:
The patent performs preliminary knowledge distillation training where the lightweight model learns from the full teacher model before deployment. This preliminary training phase ensures the pruned model acquires essential knowledge and patterns from the complete model, guaranteeing that when the lightweight model runs faster, it maintains high image generation quality without requiring the full model's computational resources.
4Quantity of substance
If transformer block groups are pruned to reduce model size, then the model size is reduced, but the complexity of model construction increases
Solution Approach 1:
The patent segments the model construction process into standardized stages (DOWN-1, DOWN-2, DOWN-3, MID, UP-1, UP-2, UP-3) with clear rules for which transformer blocks to prune in each stage. This segmentation provides a systematic framework that simplifies the complexity of model construction by offering a repeatable pattern rather than requiring ad-hoc decisions for each pruning operation.
Data Source
AI summary
Disclosed herein are a method for lightweighting a text-to-image generation model based on self-attention knowledge distillation and an apparatus for the same. A method for lightweighting a text-to-image generation model based on self-attention knowledge distillation is performed by an apparatus for lightweighting a text-to-image generation model, and includes constructing a lightweight model by pruning and changing a part of blocks in the text-to-image generation model, and training the lightweight model based on self-attention knowledge distillation using a teacher model.