Markerless Jaw Tracking via Skin-to-Jaw Motion Mapping
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for facial performance capture, such as manual tracking with markers or naive assumptions about skin motion, are inaccurate and labor-intensive, lacking the ability to accurately track jaw motion without invasive instruments or markers.
Innovation Solution
A system and method that learns a non-linear mapping from skin motion to jaw motion using facial training data, allowing for accurate jaw tracking without markers by predicting jaw poses from facial skin geometry, and can be transferred to new subjects for whom ground-truth jaw motion is not available.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If markers or invasive instruments are used for jaw tracking, then measurement precision is improved, but ease of operation and device complexity worsen
Solution Approach 1:
The patent extracts and removes the markers and invasive instruments from the tracking system, achieving accurate jaw motion capture through purely optical means by tracking facial skin geometry and using a learned mapping to jaw motion, thereby simplifying the setup while maintaining precision
Solution Approach 2:
The patent replaces the mechanical marker-based tracking system with an optical-based system that uses computer vision to capture facial skin geometry and a learned mapping model to infer jaw motion, eliminating the need for physical markers and invasive instruments
2Measurement precision
If markers are used for jaw tracking, then measurement precision is improved, but device complexity worsens
Solution Approach 1:
The patent removes markers and invasive instruments from the system, achieving accurate jaw tracking through optical capture of facial skin geometry and a learned mapping model, thereby reducing device complexity while maintaining measurement precision
Solution Approach 2:
The patent creates a digital model (copy) of jaw motion by learning the mapping from facial skin geometry to jaw motion, allowing accurate reconstruction of jaw poses without physical markers, thus simplifying the overall system
3Ease of operation
If naive assumptions about skin motion are made, then ease of operation is improved, but measurement precision worsens
Solution Approach 1:
The patent performs preliminary action by capturing facial training data and learning the mapping from skin motion to jaw motion before actual tracking, enabling accurate jaw pose prediction from skin geometry without requiring complex real-time analysis or invasive markers
Solution Approach 2:
The patent uses feedback by capturing ground-truth jaw motion data during training and using it to learn and refine the mapping model, ensuring high accuracy in predicting jaw poses from facial skin geometry while maintaining operational simplicity
Data Source
AI summary
Some implementations of the disclosure are directed to capturing facial training data for one or more subjects, the captured facial training data including each of the one or more subject's facial skin geometry tracked over a plurality of times and the subject's corresponding jaw poses for each of those plurality of times; and using the captured facial training data to create a model that provides a mapping from skin motion to jaw motion. Additional implementations of the disclosure are directed to determining a facial skin geometry of a subject; using a model that provides a mapping from skin motion to jaw motion to predict a motion of the subject's jaw from a rest pose given the facial skin geometry; and determining a jaw pose of the subject using the predicted motion of the subject's jaw.