Matching Modeling Units for Knowledge Distillation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Pre-trained models with different granularities cannot perform knowledge distillation with service models, limiting the optimization and improvement of service model performance.
Innovation Solution
A data processing method that matches modeling units of different granularities between pre-trained and initial service models, allowing for knowledge distillation by determining target modeling units with consistent fully-connected layer output sequence lengths, enabling the transfer of knowledge from pre-trained models to optimize initial service models.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If knowledge distillation is performed between pre-trained models and service models with different granularities, then the application scope of knowledge distillation is expanded, but the sequence lengths of fully-connected layer outputs are inconsistent making distillation impossible
Solution Approach 1:
The patent segments the fully-connected layer outputs into multiple parts corresponding to different modeling units. By dividing the output sequences into matching units from both pre-trained and service models, it enables granular matching even when overall sequence lengths differ, thus resolving the contradiction between adaptability and precision
Solution Approach 2:
The patent applies local quality by ensuring that corresponding modeling units from different granularities have matching output dimensions. Instead of requiring global sequence length consistency, it allows local matching at the modeling unit level, where each unit's output is adjusted to be compatible for knowledge distillation
2Reliability
If pre-trained models with different granularities are used to optimize service models, then service model performance is improved, but the modeling units have different granularities preventing direct knowledge transfer
Solution Approach 1:
The patent introduces an intermediary matching mechanism that aligns modeling units from different granularities. This intermediary process maps between the pre-trained model's modeling units and the service model's modeling units, enabling knowledge transfer despite granularity differences while maintaining system complexity at an acceptable level
Data Source
AI summary
This application discloses a data processing method performed by a computer device. Obtained service data is inputted into a pre-trained model and an initial service model to obtain first fully-connected layer outputs of the pre-trained model and second fully-connected layer outputs of the initial service model. The pre-trained model and the initial service model construct modeling units with different granularities, first modeling units of the pre-trained model are matched with second modeling units of the initial service model, fully-connected layer outputs of the second modeling units have the same sequence length as fully-connected layer outputs of matched target first modeling units so that with the assistance of the pre-trained model, the target service model can be obtained through the initial service model, the knowledge distillation of the models including modeling units with different granularities is realized, thereby effectively improving the service performance of the service model.