Neural Network Model Compression via Joint Optimization Target
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional knowledge distillation methods for compressing neural network models for machine translation result in low information migration accuracy and poor translation quality, particularly in mobile terminals, due to reduced model capacity.
Innovation Solution
The method involves training two teacher models based on N training samples, determining guide components for each sample, and using a joint optimization target to train a student model, which improves knowledge migration and translation quality by leveraging both teacher models' predictions and probability distributions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If conventional knowledge distillation method is used to compress neural network model, then model complexity is reduced, but information migration accuracy deteriorates
Solution Approach 1:
The patent segments the knowledge distillation process into multiple independent distillation modules, each handling specific aspects of knowledge transfer. This segmentation allows for more precise control over different types of information migration while maintaining overall model compression, thereby improving information migration accuracy without significantly increasing model complexity.
Solution Approach 2:
The patent merges multiple teacher models (including bidirectional teacher models and auxiliary teacher models) into a comprehensive distillation framework. By combining the strengths of different teacher models and using ensemble distillation, the system achieves higher information migration accuracy while the student model remains compact through efficient knowledge integration.
2Quantity of substance
If model compression is applied to improve translation quality, then model size is reduced, but translation quality deteriorates
Solution Approach 1:
The patent employs preliminary actions by training multiple teacher models to high accuracy before distillation, and by pre-processing training data to enhance quality. The bidirectional teacher models are trained extensively beforehand to capture comprehensive translation knowledge, which is then distilled into the smaller student model, ensuring the compressed model maintains high translation quality.
Solution Approach 2:
The patent introduces auxiliary teacher models as intermediaries that bridge the gap between large teacher models and the small student model. These auxiliary teachers facilitate more effective knowledge transfer by providing intermediate guidance during distillation, enabling the student model to achieve better translation quality despite its reduced size.
3Device complexity
If single teacher model is used for distillation, then training process is simplified, but information migration completeness deteriorates
Solution Approach 1:
The patent segments the teacher model ensemble into different functional components: bidirectional teacher models for capturing forward and backward translation patterns, and auxiliary teacher models for providing specialized knowledge. This segmentation allows comprehensive information migration while keeping the training process organized and manageable through modular architecture.
Solution Approach 2:
The patent creates a universal distillation framework that can accommodate multiple types of teacher models with different functions. The system universally handles various teacher model outputs through a unified distillation mechanism, enabling complete information migration from diverse teacher sources while maintaining a relatively simple and flexible training process.
Data Source
AI summary
A method for compressing a neural network model includes: obtaining a first trained teacher model and a second trained teacher model based on N training samples, N being a positive integer greater than 1; for each of the N training samples, determining a first guide component of the first teacher model and a second guide component of the second teacher model respectively, determining a sub optimization target corresponding to the training sample and configured to optimize a student model according to the first guide component and the second guide component, and determining a joint optimization target based on each of the N training samples and a sub optimization target corresponding to the training sample; and training the student model based on the joint optimization target.