Entropy-Guided Model Training for Robust Speech Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models face challenges in balancing model size reduction and accuracy maintenance, particularly in applications like speech recognition, where overfitting and expertise combination issues arise from using single trained models.
Innovation Solution
A model training apparatus selects previously trained models based on entropy to train a target model using a knowledge distillation scheme, combining output values with weighted averages to enhance accuracy and robustness across different conditions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single previously trained model is used for speech recognition, then the model size is small and training time is reduced, but accuracy and robustness deteriorate due to overfitting and limited expertise
Solution Approach 1:
The patent combines multiple previously trained models (first teacher model and second teacher model with different characteristics) into a unified training framework for the target model. By merging the expertise of multiple specialized models through knowledge distillation, the system achieves improved accuracy and robustness while managing model complexity through selective combination based on entropy metrics.
2Reliability
If multiple previously trained models are combined to improve accuracy, then reliability and robustness are improved, but device complexity and computational overhead increase
Solution Approach 1:
The patent introduces dynamic model selection based on entropy calculation. The system dynamically determines which teacher models to use and their weighting ratios based on the entropy of their outputs for each input. This dynamic approach allows the system to adapt to different input conditions, using only the necessary model complexity for each specific case rather than always deploying the full ensemble.
Solution Approach 2:
The patent changes the parameter of model selection by using entropy values as a dynamic parameter. The entropy of each teacher model's output is calculated, and these entropy values directly determine the weighting ratios in the knowledge distillation process. This parameter change enables flexible adaptation of system complexity based on the confidence and agreement of different teacher models.
3Adaptability or versatility
If knowledge distillation from multiple teacher models is used, then generalization capability is enhanced, but training time and computational resources increase
Solution Approach 1:
The patent applies partial action by selectively using only the necessary teacher models for each training instance based on entropy thresholds and ratios. Rather than always using all available teacher models, the system performs partial knowledge distillation from the most relevant teachers, reducing unnecessary computational overhead while maintaining generalization benefits.
Data Source
AI summary
A processor-implemented model training method and apparatus are provided. The method calculates an entropy of each of a plurality of previously trained models based on training data, selects a previously trained model from the plurality of previously trained models based on the calculated entropy, and trains a target model, distinguished from the plurality of previously trained models, based on the training data and the selected previously trained model.