Automatic 3D Face Mesh Tracking with Eye and Mouth Boundary Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing mesh tracking algorithms for 3D face models are semi-automatic and require user intervention, particularly struggling with accurate estimation of eye and mouth contours.
Innovation Solution
A fully automatic mesh tracking method that uses rigid alignment, 3D contour detection, and dense mesh tracking with Wrap3, incorporating a Region Of Interest (ROI) detection and improved active contour fitting snake algorithm to reconstruct eye and mouth boundaries, and includes eyelid correction through mesh deformation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Extent of automation
If semi-automatic mesh tracking algorithms are used, then user intervention can provide manual correction capability, but the tracking process requires manual manipulation and user input
Solution Approach 1:
The system performs automatic mesh tracking without requiring user intervention. The algorithm independently processes 3D scans, performs rigid alignment, detects eye and mouth boundaries, and generates tracked meshes automatically, making the system self-sufficient and eliminating the need for manual manipulation
Solution Approach 2:
The patent replaces manual user manipulation with automated computational algorithms. Specifically, it uses rigid alignment algorithms instead of manual pose correction, automated eye and mouth boundary detection instead of manual contour tracing, and automatic mesh tracking algorithms instead of manual vertex correspondence establishment
2Measurement precision
If traditional contour detection methods are used for eyes and mouths, then the process is simpler, but accurate estimation of eye and mouth contours remains a challenge
Solution Approach 1:
The patent segments the face mesh into distinct regions of interest (eyes and mouths) and applies specialized detection algorithms to each. The eye and mouth boundary detection is separated from general mesh tracking, allowing for more accurate and targeted contour estimation in these critical facial regions
Solution Approach 2:
The patent introduces intermediate steps in the boundary detection process, including region of interest identification, key point detection, and contour fitting algorithms. These intermediary processes bridge the gap between simple detection and accurate contour estimation, improving precision while managing complexity through structured processing
3Reliability
If dense mesh tracking is performed on entire video sequences, then comprehensive tracking coverage is achieved, but drifting occurs over time
Solution Approach 1:
The patent divides the video sequence into smaller clips or segments for processing. By performing dense mesh tracking on shorter segments rather than entire video sequences, the system reduces cumulative drifting errors while maintaining comprehensive coverage through sequential processing of multiple segments
Solution Approach 2:
The patent implements feedback mechanisms in the tracking process, where previous tracking results are used to inform and correct subsequent tracking operations. This feedback loop helps maintain tracking stability by continuously refining vertex correspondences and correcting drift accumulation over time
Data Source
AI summary
The mesh tracking described herein involves mesh tracking on 3D face models. In contrast to existing mesh tracking algorithms which generally require user intervention and manipulation, the mesh tracking algorithm is fully automatic once a template mesh is provided. In addition, an eye and mouth boundary detection algorithm is able to better reconstruct the shape of eyes and mouths.