Facial Emotion Recognition Mitigating Artifacts via Action Unit Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing facial emotion recognition systems fail to accurately recognize emotions in the presence of facial artifacts such as eyeglasses, facial hair, scars, and birthmarks due to sensitivity to scaling, rotation, and internal variations, leading to false or no identification.
Innovation Solution
A method and system that detect Action Units (AUs) in frames with facial artifacts, analyze and register neutral facial expressions, predict AUs in subsequent frames, and determine emotions by comparing registered neutral expressions with predicted AUs, while mitigating weights of AUs with artifacts to improve accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing facial emotion recognition systems use geometric patterns from eyes, eyebrows, and mouth region, then emotion recognition is performed, but the system becomes sensitive to scaling and rotation of face in image plane leading to reduced robustness
Solution Approach 1:
The patent segments the face into multiple regions of interest (eyes, eyebrows, mouth, nose, cheeks, forehead) and analyzes Action Units independently in each region. This segmentation allows the system to focus on local muscular movements rather than global geometric patterns, making it less sensitive to overall face scaling and rotation while maintaining emotion recognition accuracy.
Solution Approach 2:
The patent applies local quality analysis by examining specific Action Units in different facial regions with region-specific weighting. Each facial region is analyzed with appropriate importance weights assigned to different Action Units based on their emotional significance, allowing the system to maintain robustness while achieving precise local emotion detection.
2Reliability
If existing systems use fiducial point approaches to handle occlusions, then some occlusion robustness is achieved, but the system remains sensitive to internal facial variations such as facial hair, scars, and birth marks causing false identification
Solution Approach 1:
The patent extracts and separately analyzes Action Units from occluded or artifact-affected regions, taking out the problematic areas from the overall emotion recognition process. By focusing on Action Units in non-occluded regions and using temporal modeling to infer occluded regions, the system achieves robustness to both occlusions and internal facial variations without causing false identifications.
Solution Approach 2:
The patent implements feedback through temporal modeling where the system continuously updates its understanding of facial artifacts and occlusions over time. By analyzing Action Unit patterns across multiple frames and providing feedback to the recognition process, the system adapts to individual facial characteristics and reduces false identification while maintaining occlusion robustness.
3Measurement precision
If the system analyzes all Action Units equally, then comprehensive emotion analysis is performed, but facial artifacts result in incorrect analysis due to distorted features
Solution Approach 1:
The patent applies local quality by assigning different importance weights to different Action Units based on their reliability and emotional significance. Action Units in regions affected by facial artifacts are down-weighted or excluded, while Action Units in clear regions are given higher weight, allowing comprehensive emotion analysis while maintaining correctness despite the presence of artifacts.
Solution Approach 2:
The patent dynamically changes parameters by adjusting the weight of different Action Units based on detected facial artifacts and occlusions. The system modifies the contribution of each Action Unit to the final emotion classification in real-time, changing the analysis parameters adaptively to maintain both comprehensiveness and correctness in the presence of artifacts.
Data Source
AI summary
A method for determining a facial emotion of a user in the presence of a facial artifact includes detecting Action Units (AUs) for a first set of frames with the facial artifact; analyzing the AUs with the facial artifact after the detection; registering the analyzed AUs for a neutral facial expression with the facial artifact in the first set of frames; predicting the AUs in a second set of frames; and determining the facial emotion by comparing the registered neutral facial expression with the predicted AUs in the second set of frames.