Talking Face Detection via Entropy Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current emotion recognition systems fail to accurately classify emotions in 'talking faces' due to the complexity of lip movements and lack of temporal segmentation, leading to erroneous classifications.
Innovation Solution
An unsupervised temporal segmentation method using integrated gradient histograms (IGH) and entropy analysis to distinguish talking and non-talking segments, which normalizes and localizes face regions, derives IGH from Local Binary Pattern (LBP) values, and employs online K-Means clustering to quantify uncertainties in mouth movements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If facial expression recognition systems use the mouth region for emotion detection, then emotion recognition capability is enhanced, but accuracy deteriorates during talking segments due to complex lip movements
Solution Approach 1:
The patent segments the video sequence into talking and non-talking phases using entropy analysis of integrated gradient histograms. By dividing the temporal sequence into distinct phases, the system can apply different recognition strategies: excluding mouth region during talking phases and including it during non-talking phases, thereby resolving the contradiction between utilizing mouth information for emotion detection and avoiding its misleading effects during talking.
2Productivity
If static methods analyze frames independently for emotion recognition, then processing speed is improved, but temporal context information is lost
Solution Approach 1:
The patent performs preliminary action by computing integrated gradient histograms and entropy values for each frame before the actual emotion recognition process. This preprocessing step captures temporal variations in mouth region appearance, preserving contextual information that can be used during subsequent emotion classification without requiring complex temporal modeling during the main processing phase.
3Ease of operation
If current systems determine active speakers in multi-person environments, then speaker identification is achieved, but temporal segmentation of lip activities into talking and non-talking phases is not performed
Solution Approach 1:
The patent extracts the specific temporal segmentation information regarding talking and non-talking phases from the video sequence, separating this function from the speaker identification task. By extracting the temporal phase information through entropy analysis of mouth region histograms, the system can use this segmented information to guide emotion recognition, taking out the problematic talking segments where mouth movements would mislead emotion classification.
Data Source
AI summary
A method and system for detecting temporal segments of talking faces in a video sequence using visual cues. The system detects talking segments by classifying talking and non-talking segments in a sequence of image frames using visual cues. The present disclosure detects temporal segments of talking faces in video sequences by first localizing face, eyes, and hence, a mouth region. Then, the localized mouth regions across the video frames are encoded in terms of integrated gradient histogram (IGH) of visual features and quantified using evaluated entropy of the IGH. The time series data of entropy values from each frame is further clustered using online temporal segmentation (K-Means clustering) algorithm to distinguish talking mouth patterns from other mouth movements. Such segmented time series data is then used to enhance the emotion recognition system.