Neural Network Model Compression Without Original Training Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network model compression techniques often result in performance deterioration, and fine-tuning requires access to the original training dataset, which may not be feasible due to privacy or legal constraints.
Innovation Solution
An electronic apparatus that compresses a neural network model using a compression algorithm, generates training data pairs based on the original model, and trains the compressed model to recover performance without relying on the original training data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the neural network model is compressed using compression algorithms, then memory usage and computational complexity are reduced, but performance deterioration occurs
Solution Approach 1:
The patent applies preliminary action by generating synthetic training data pairs before the fine-tuning stage. The generation module creates input-output data pairs based on the original neural network model's processing of sample data, preparing the necessary training materials in advance. This allows the compressed model to be fine-tuned without requiring access to the original training dataset, thereby resolving the contradiction between compression and performance maintenance.
Solution Approach 2:
The patent employs copying by creating synthetic training data pairs that replicate the essential characteristics of the original training data. The generation module produces copies of input-output relationships by processing sample data through the original model and capturing its transformations. These copied data pairs serve as substitutes for the original training data, enabling fine-tuning of the compressed model while maintaining performance without requiring the original dataset.
2Reliability
If fine tuning is performed using actual training data, then model performance is recovered, but access to original training data is required which may not be available due to privacy or legal constraints
Solution Approach 1:
The patent implements self-service by enabling the system to generate its own training data pairs from sample data and the original model's processing behavior. The generation module autonomously creates the necessary training materials by processing sample inputs through the original model and capturing the transformations. This self-generated data allows the compressed model to be fine-tuned independently, without requiring external access to the original training dataset, thereby resolving the contradiction between performance recovery and data availability.
Solution Approach 2:
The patent introduces an intermediary mechanism in the form of the data generation module, which acts as a mediator between the original model and the compressed model training process. This intermediary generates synthetic training data pairs that bridge the gap between the unavailable original training data and the compressed model's fine-tuning requirements. By using this intermediary-generated data, the system can fine-tune the compressed model while respecting privacy and legal constraints on accessing the original training dataset.
3Reliability
If the neural network model scale is increased to improve performance, then accuracy improves, but device resource limitations are exceeded
Solution Approach 1:
The patent applies parameter changes by systematically adjusting the compression parameters of the neural network model to find the optimal balance between model size and performance. The compression module varies parameters such as pruning ratios, quantization levels, and layer reduction rates to compress the model while maintaining acceptable performance. This allows deploying larger, more accurate models on resource-constrained devices by transforming the model's parameters and structure rather than reducing its fundamental capability.
Solution Approach 2:
The patent employs segmentation by dividing the neural network model into different components and layers that can be selectively compressed. The compression module applies different compression techniques to different segments of the model, such as pruning less important connections in certain layers while preserving critical structures in others. This segmented approach allows maintaining high accuracy in essential functions while reducing overall model scale to fit device resource limitations.
Data Source
AI summary
An electronic apparatus for compressing a neural network model may acquire training data pairs based on an original, trained neural network model and train a compressed neural network model compressed from the original, trained neural network model using the acquired training data pairs.