Mixed-Domain Model Training via Teacher-Student Soft Label Distillation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models face performance degradation when attempting to support multiple domains with a single shared model, particularly evident in ASR systems and image recognition models, where narrowband and broadband systems or color modes require separate models, leading to increased computational resources and maintenance workload.
Innovation Solution
A computer-implemented method using teacher models specialized for different domains to generate soft labels, which are then used to train a student model with a unified feature extractor, allowing the mixed-domain model to support multiple domains without performance deterioration, by leveraging matched feature extractors and hybrid normalization parameters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple separate acoustic models are constructed for different sampling frequencies (narrowband and broadband), then speech recognition performance for each domain is optimized, but computational resources and system maintenance workload increase significantly
Solution Approach 1:
The patent merges multiple domain-specific acoustic models into a single mixed-band acoustic model that can handle both narrowband and broadband speech. The student model integrates knowledge from multiple teacher models trained on different sampling frequencies, enabling one model to serve multiple domains while maintaining recognition performance.
Solution Approach 2:
The mixed-band acoustic model is designed with universal functionality to process both narrowband and broadband speech inputs. The unified model structure with shared parameters allows it to adapt to different sampling frequencies without requiring separate specialized models for each domain.
2Device complexity
If a single mixed-band acoustic model is used for both narrowband and broadband systems, then computational resources and maintenance workload are reduced, but speech recognition performance deteriorates in both domains
Solution Approach 1:
The patent applies preliminary action by first training multiple domain-specific teacher models on their respective domains before creating the mixed-band student model. This preliminary training ensures that the student model learns from high-performance specialized models, preserving recognition accuracy while achieving model consolidation.
Solution Approach 2:
The student model acts as an intermediary that synthesizes knowledge from multiple teacher models. It mediates between the specialized narrowband and broadband teacher models, integrating their strengths into a unified model that maintains performance across both domains while reducing system complexity.
3Measurement precision
If domain-specific feature extractors are used for each domain, then feature extraction accuracy is optimized, but the system requires separate feature extraction processes for each domain
Solution Approach 1:
The patent employs a unified feature extractor that serves multiple domains simultaneously. This single feature extraction process handles both narrowband and broadband speech inputs, eliminating the need for separate domain-specific feature extraction pipelines while maintaining extraction accuracy through adaptive processing.
Data Source
AI summary
A technique for constructing a model supporting a plurality of domains is disclosed. In the technique, a plurality of teacher models, each of which is specialized for different one of the plurality of the domains, is prepared. A plurality of training data collections, each of which is collected for different one of the plurality of the domains, is obtained. A plurality of soft label sets is generated by inputting each training data in the plurality of the training data collections into corresponding one of the plurality of the teacher models. A student model is trained using the plurality of the soft label sets.