Facial Expression Recognition Using Speech Map Suppression
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Facial expression recognition performance is degraded by shape changes around the mouth area, even when the user's emotion remains unchanged, due to the lack of adequate models for differentiating key areas associated with speech.
Innovation Solution
A method that extracts feature information from key areas using a speech map and a facial expression model, allowing for accurate recognition by applying adequate models for facial expression recognition, which involves a neural network-based approach to invert the speech map and reflect it on the facial expression feature, generating a target vector to identify the facial expression.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If facial expression recognition is performed on the entire face, then comprehensive emotion detection is achieved, but recognition accuracy is degraded by shape changes around the mouth area during speech
Solution Approach 1:
The face is divided into multiple key areas (eyes, eyebrows, nose, mouth, cheek) with different importance weights for emotion recognition. The speech map is segmented to identify mouth-related regions, and these regions are selectively suppressed in the fused feature map to eliminate their harmful effect on emotion recognition accuracy.
Solution Approach 2:
Different regions of the face are assigned different quality weights based on their relevance to emotion expression. Eye and eyebrow regions are given higher weights as they are more stable and indicative of emotion, while mouth regions are given lower weights during speech to reduce their negative impact on recognition accuracy.
2Reliability
If the speech map is inverted and reflected on the facial expression feature, then robustness against speech-induced changes is improved, but the complexity of the recognition system increases
Solution Approach 1:
A speech map generated by a speech detection model serves as an intermediary to identify and suppress mouth-related regions. This intermediary map enables the system to dynamically adjust the contribution of different face regions based on speech activity, improving robustness without requiring complex manual intervention.
Solution Approach 2:
The speech map is generated in advance to predict mouth-related regions before they interfere with emotion recognition. By pre-identifying these regions and preparing suppression weights, the system avoids the need for complex real-time adjustments during emotion recognition, thereby managing system complexity effectively.
3Measurement precision
If feature information from all facial areas is used, then complete facial expression analysis is achieved, but speech-related noise reduces recognition performance
Solution Approach 1:
Speech-related noise is extracted and removed from the facial expression features by identifying mouth-related regions through the speech map and suppressing their contribution. This extraction process separates harmful speech-induced variations from useful emotion-related features, improving recognition precision.
Solution Approach 2:
The speech map, which initially identifies harmful mouth-related regions, is inverted and used as a beneficial suppression mechanism. The same information that reveals the source of noise (speech activity) is converted into a tool to eliminate that noise, thereby improving emotion recognition precision.
Data Source
Figure 1
Figure 2
Figure 3A
AI summary
A facial expression recognition apparatus and method and a facial expression training apparatus and method are provided. The facial expression recognition apparatus generates a speech map indicating a correlation between a speech and each portion of an object based on a speech model, extracts a facial expression feature associated with a facial expression based on a facial expression model, and recognizes a facial expression of the object based on the speech map and the facial expression feature. The facial expression training apparatus trains the speech model and the facial expression model.