Lightweight Generative Model via Strong Lottery Tickets
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing artificial intelligence-based generative models are limited in their application to edge devices due to high computing costs, and current compression methods are unstable and costly, leading to performance degradation.
Innovation Solution
The use of a subnetwork (strong lottery tickets, SLTs) algorithm to find a lightweight network within a generative model, which stabilizes compression and maintains performance without additional networks or complex learning processes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If large and complex structures are used for generative models, then performance is improved, but computing cost increases making application to edge devices limited
Solution Approach 1:
The patent segments the dense network into multiple subnetworks by randomly initializing and selecting a subset of weights. This segmentation allows the model to maintain the architectural complexity needed for high performance while reducing the actual computational load by only activating a portion of the network parameters during inference.
Solution Approach 2:
The patent extracts a lightweight subnetwork from the dense network by selecting and retaining only the most important weights while setting others to zero. This extraction process removes unnecessary computational components while preserving the essential functionality and performance characteristics of the original model.
2Weight of stationary object
If training-pruning-retraining process is used to lighten generative models, then model size is reduced, but additional networks and complex learning processes are required increasing cost and instability
Solution Approach 1:
The patent performs preliminary random initialization and selection of weights before the actual pruning process. By pre-assigning random weights and identifying a candidate subnetwork structure in advance, the method simplifies the subsequent pruning operation and avoids the need for complex iterative retraining processes.
Solution Approach 2:
The patent enables the network to self-select its optimal subnetwork configuration through random initialization and weight selection without requiring external guidance or complex pruning algorithms. The network autonomously identifies important weights and prunes less important ones, reducing the need for additional networks and complex learning processes.
3Weight of stationary object
If compression methods are applied to generative models, then model size is reduced, but performance degradation occurs
Solution Approach 1:
The patent changes the parameter configuration by randomly initializing weights and selectively retaining only the top k% of weights based on their importance. This parameter change approach allows the model to be compressed while maintaining performance by ensuring that the most critical parameters are preserved in the subnetwork.
Data Source
AI summary
Disclosed is a device for lightening an artificial intelligence-based generative model including a memory that stores data for lightening the artificial intelligence-based generative model and a processor that perform operations related to lighten the generative model. The processor assigns a randomly initialized score(s) to each of weights for a dense network based on an edge-popup algorithm, finds a random subnetwork, sorts the assigned scores in each forward path, and updates the scores using backpropagation, while leaving a weight with a preset top k % score.