Facial Expression Annotation Using Color and Deep Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing algorithms fail to reliably recognize Action Units (AUs) and their intensities across databases, are computationally demanding, and are limited in their ability to handle large datasets, making it impractical to annotate large databases of facial expressions efficiently.
Innovation Solution
A computer vision and machine learning process that recognizes AUs, their intensities, and emotion categories across databases in real-time, using color features and a Global-Local loss function for Deep Neural Networks, enabling automatic annotation of large datasets and independent database recognition.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual annotation by expert coders is used, then annotation accuracy is improved, but productivity deteriorates (too slow for large databases)
Solution Approach 1:
The patent replaces the mechanical manual annotation process with an automated computer vision system using deep neural networks. The system processes facial images through convolutional neural networks and recurrent neural networks to automatically detect Action Units and annotate emotional expressions, eliminating the need for manual expert coding while maintaining high accuracy across large databases.
Solution Approach 2:
The system enables self-service annotation by training the neural network models to autonomously identify and annotate facial expressions without human intervention. The automated pipeline processes images through multiple neural network stages, performing detection, classification, and annotation tasks independently, thus achieving both high productivity and maintained annotation quality.
2Reliability
If existing algorithms are used, then some AUs are recognized, but reliability deteriorates (cannot recognize all AUs reliably across databases)
Solution Approach 1:
The patent creates a universal annotation system using deep neural networks that can reliably recognize Action Units across multiple different databases. The trained models are designed to generalize beyond training data, enabling consistent and reliable AU detection whether images come from the training database or independent test databases, thus achieving cross-database reliability.
Solution Approach 2:
The system employs feedback mechanisms through the neural network training process, where models are trained on annotated data and then evaluated on independent databases. The feedback from performance metrics guides further training and optimization, improving the reliability of AU recognition across different database sources through iterative refinement.
3Measurement precision
If comprehensive AU recognition is implemented, then measurement precision is improved, but device complexity deteriorates (computational demanding)
Solution Approach 1:
The patent segments the complex AU recognition task into multiple specialized neural network components. Convolutional neural networks handle feature extraction from facial images, while recurrent neural networks process temporal sequences for video data. This segmentation allows each component to focus on specific aspects of AU detection, improving precision while managing computational complexity through modular architecture.
4Productivity
If real-time processing is implemented, then productivity is improved, but measurement precision deteriorates (difficulty in maintaining accuracy at high speed)
Solution Approach 1:
The system performs preliminary action by pre-training deep neural network models on large annotated databases before deployment. The models are pre-computed and optimized during the training phase, storing learned features and patterns that enable rapid real-time inference. This preliminary training allows the system to achieve both high processing speed and maintained accuracy during actual real-time operation without performing complex computations during inference.
Data Source
AI summary
The innovation disclosed and claimed herein, in aspects thereof, comprises systems and methods of identifying AUs and emotion categories in images. The systems and methods utilized a set of images that include facial images of people. The systems and methods analyze the facial images to determine AUs and facial color due to facial blood flow variations that are indicative of an emotion category. In aspects, the analysis can include Gabor transforms to determine the AUs, AU intensities and emotion categories. In other aspects, the systems and method can include color variance analysis to determine the AUs, AU intensities and emotion categories. In further aspects, the analysis can include deep neural networks that are trained to determine the AUs, emotion categories and their intensities.