Facial Feature Tracking via Spatial Regularization and Calibration
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for measuring facial expressions are limited by the need for constrained environments, specialized hardware, and trained operators, making them impractical for consumer applications, and generic trackers often fail to generalize across individuals due to the complexity and time required for training.
Innovation Solution
A system that robustly locates facial features using a calibration phase with neutral expressions, incorporates spatial regularization in feature locator update models, and employs optical flow to tailor trackers to specific individuals, allowing for accurate and generic expression measurement across a wide range of subjects without additional computational cost.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a generic tracker is trained to work on the entire population, then the system can be applied broadly, but it fails or underperforms on many specific individuals due to inability to generalise
Solution Approach 1:
The system performs a calibration phase before actual expression measurement, where it captures neutral expression images from each individual and pre-computes appearance templates. This preliminary action allows the tracker to be adapted to specific individuals without requiring full retraining, resolving the contradiction between broad applicability and individual accuracy.
Solution Approach 2:
The system creates individual-specific appearance templates during calibration that are tailored to each person's facial characteristics. These localised templates are then used during tracking to improve accuracy for that specific individual while the overall system remains applicable to the entire population through the calibration process.
2Measurement precision
If the tracker is retrained to include specific individuals, then tracking accuracy improves for those individuals, but the process is impractical due to the length of time and complexity of training
Solution Approach 1:
The system extracts only the essential calibration information needed for individual-specific tracking during a brief neutral expression capture phase. Instead of performing full retraining, it extracts appearance templates and initial feature locations that are sufficient for accurate tracking, dramatically reducing the time and complexity while maintaining individual accuracy.
Solution Approach 2:
The calibration phase performs preliminary adaptation to each individual by capturing neutral expressions and computing appearance templates in advance. This preliminary action eliminates the need for time-consuming retraining during actual use, allowing the system to achieve individual-specific accuracy efficiently.
3Measurement precision
If professional facial expression measurement methods are used, then measurement accuracy is high, but the system requires constrained environments, specialized hardware, and trained operators making it impractical for consumer applications
Solution Approach 1:
The system performs automatic calibration and tracking without requiring trained operators or specialized hardware. The calibration process automatically captures neutral expressions and computes templates, and the tracking algorithm automatically adapts to each individual, eliminating the need for human expertise and complex equipment while maintaining professional-grade accuracy.
Solution Approach 2:
The system uses a standard video camera already present in consumer devices rather than specialized hardware. The same camera serves both for calibration and for actual expression measurement, making the system universally applicable to consumer electronics without requiring additional specialized equipment or constrained environments.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Enables accurate and robust facial expression analysis in unconstrained consumer environments, improving feature tracking and expression measurement, and decoupling measurements from individual identity, thus increasing the system's applicability to a broader population.
Implementation Method 1
The use of optical flow between a particular image of a subject showing a neutral expression and each image within a video sequence, in order to tailor a generic feature tracker
Implementation Method 2
The use of a novel spatial regularization when training a feature locator update model that produces update matrices where the coefficients associated to neighbouring pixels in the image have similar values
Data Source
AI summary
Computer implemented methods for generating a non-transient record of feature locations and/or facial expression parameters characterizing a person's face. A video sequence of a specified individual person is received and a feature locator update model is applied to the video sequence. The feature locator update model is derived by defining a set of training images, generating a set of facial feature displacements for each training image with associated image sample vectors, and training a regularized linear regression which maps from image sample vectors to displacement vectors, wherein the regularization includes a spatial smoothness term within the shape-free sample space. A feature location and/or a facial expression parameter is then extracted, based on the feature update model, characterizing the location, and/or the expression, of the feature of the face of the specified individual person.