Fusion DNN Training via Knowledge Distillation for Facial Expression Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing facial expression recognition methods, such as those described in PTL 1, can be improved in terms of precision when using neural networks for identifying facial expressions from captured images.
Innovation Solution
A training device that utilizes a deep neural network (DNN) model trained with multi-modal information items and ground truth labels, incorporating knowledge distillation techniques to fuse outputs from multiple DNNs, thereby enhancing inference precision without the need for pre-training each modal information item.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If knowledge distillation is performed using a teacher model and student model, then the precision of inference results is improved, but the training time and computational resources increase
Solution Approach 1:
The patent applies preliminary action by pre-training a teacher model with multi-modal information items before training the student model. The teacher model is trained first to extract and fuse features from multiple modalities (e.g., facial images, audio), and this pre-trained knowledge is then transferred to the student model through knowledge distillation. This preliminary training of the teacher model enables the student model to achieve high precision without requiring extensive training time itself.
Solution Approach 2:
The patent uses copying by creating a student model that replicates the architecture and functionality of the teacher model but at a smaller scale. The student model copies the essential learning patterns and feature extraction mechanisms from the teacher model through knowledge distillation, allowing it to achieve comparable precision with reduced computational resources and training time.
2Measurement precision
If multiple modal information items are used for training, then the precision of facial expression recognition is improved, but the complexity of the training process increases
Solution Approach 1:
The patent applies segmentation by dividing the training process into separate modules for different modalities (e.g., image processing module, audio processing module) and then integrating them through the fusion mechanism. Each modality is processed independently by dedicated DNNs, and the results are combined in the fusion layer. This modular segmentation reduces the overall complexity by making each component manageable and independent.
Solution Approach 2:
The patent uses merging (combining) by integrating multiple DNNs that process different modalities through a fusion mechanism. The fusion layer combines the outputs from image-processing DNNs, audio-processing DNNs, and other modality-specific DNNs into a unified representation. This merging approach simplifies the training process by providing a clear framework for integrating multiple data sources without requiring complex interactions between all modalities simultaneously.
3Reliability
If pre-training is performed for each single modal information item, then the model can handle each modality effectively, but the number of learning steps and training time increase
Solution Approach 1:
The patent applies universality by training a single teacher model that handles multiple modalities simultaneously rather than separate pre-training for each modality. The teacher model is designed with multi-functional capabilities to process images, audio, and other modalities within a unified framework. This universal approach maintains effectiveness for each modality while reducing the total number of learning steps by eliminating redundant separate pre-training processes.
Data Source
AI summary
A training device or the like according to the present disclosure trains a fusion deep neural network (DNN) model by (i) using training data that includes two or more modal information items and ground truth labels of the two or more modal information items and (ii) performing knowledge distillation that is a technique in which knowledge obtained as a result of a teacher model being trained is used to train a student model. The fusion DNN model includes: two or more DNNs; and a fusion that includes a configuration in which portions of the two or more DNNs are fused and that receives an input of features that are outputs of the two or more DNNs.