Emotion Recognition in Video Conferencing via Facial Mesh Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Video conferencing systems face challenges in detecting and managing negative emotions, such as anger or annoyance, in customers, as these emotions can be difficult for service representatives to recognize, potentially leading to unresolved issues and strained interactions.
Innovation Solution
A computer-implemented method for video conferencing that analyzes facial and speech emotions using convolution neural networks and state vector machines to identify negative emotions, allowing for real-time reporting and intervention by a third party, such as a supervisor, to address customer concerns.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If video conferencing is used to provide direct communication with customers, then personalized attention and service quality are improved, but the ability to detect and manage negative emotions becomes more difficult
Solution Approach 1:
The patent introduces an emotion recognition system as an intermediary between the customer service representative and the customer. This system automatically analyzes facial expressions, voice tone, and body language to detect negative emotions, serving as a mediator that assists human operators in situations where emotional detection is difficult.
Solution Approach 2:
The patent replaces the mechanical/manual process of emotional detection by human operators with an automated computer vision and audio analysis system. This substitution uses algorithms to process visual and auditory data, identifying negative emotions without requiring human interpretation of subtle emotional cues.
2Device complexity
If manual emotion detection by service representatives is used, then system complexity is kept low, but detection accuracy and timeliness deteriorate
Solution Approach 1:
The patent implements a feedback mechanism where the emotion recognition system continuously monitors the interaction and provides real-time alerts to service representatives when negative emotions are detected. This feedback loop enables timely intervention and allows representatives to adjust their approach based on automated emotional analysis.
Solution Approach 2:
The emotion recognition system operates autonomously without requiring manual configuration or intervention. It automatically processes video and audio streams, identifies negative emotions, and generates notifications, enabling the system to serve itself in detecting and reporting emotional states throughout the interaction.
3Measurement precision
If automated emotion recognition is implemented, then emotion detection accuracy is improved, but system complexity and computational requirements increase
Solution Approach 1:
The patent segments the emotion recognition system into distinct functional modules: facial expression analysis, voice tone analysis, body language detection, and integration/decision-making components. Each module processes specific aspects of the interaction independently, then combines results to determine overall emotional state, reducing the complexity of any single component.
Data Source
Figure 1A
Figure 1B
Figure 2
AI summary
Methods and systems for videoconferencing include recognition of emotions related to one videoconference participant such as a customer. This ultimately enables another videoconference participant, such as a service provider or supervisor, to handle angry, annoyed, or distressed customers. One example method includes the steps of receiving a video that includes a sequence of images, detecting at least one object of interest (e.g., a face), locating feature reference points of the at least one object of interest, aligning a virtual face mesh to the at least one object of interest based on the feature reference points, finding over the sequence of images at least one deformation of the virtual face mesh that reflect face mimics, determining that the at least one deformation refers to a facial emotion selected from a plurality of reference facial emotions, and generating a communication bearing data associated with the facial emotion.