Facial Landmark Tracking via Optical Flow and Detection Fusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing landmark tracking methods in video streams are computationally inefficient and often result in inaccurate or jittery movements of facial landmarks, leading to incorrect determination of facial expressions and mask locations in augmented and virtual reality applications.
Innovation Solution
A system comprising a landmark detection engine, an optical flow landmark engine, a landmark difference engine, and a weighted landmark determination engine that uses facial detection and optical flow models to determine accurate and smooth landmark positions across images, incorporating backward optical flow for improved accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional landmark tracking methods are used, then computational efficiency is improved, but tracking accuracy and smoothness deteriorate
Solution Approach 1:
The patent combines multiple landmark detection methods (detection engine and optical flow engine) into a unified tracking system. The detection engine identifies landmarks in the current frame while the optical flow engine predicts landmark positions based on previous frames, and both results are merged through a weighting mechanism to produce the final tracked landmark positions, achieving both accuracy and smoothness.
Solution Approach 2:
The system implements feedback by using previously tracked landmark positions to inform current and future tracking. The optical flow engine uses historical landmark data to predict current positions, and the system continuously refines predictions by comparing with detection results and adjusting weights based on tracking quality metrics, creating a closed-loop feedback system.
2Device complexity
If traditional landmark tracking methods are used, then device complexity is reduced, but landmark position stability deteriorates
Solution Approach 1:
The patent performs preliminary actions by pre-calculating and storing landmark positions from previous frames, and pre-processing optical flow data before final landmark determination. The system prepares multiple candidate landmark positions in advance (from detection and optical flow) and selects the optimal combination, avoiding the need for complex real-time optimization during frame processing.
3Ease of operation
If simple tracking methods are used, then ease of operation is improved, but facial expression determination accuracy deteriorates
Solution Approach 1:
The patent introduces an intermediary weighting mechanism that automatically reconciles results from the detection engine and optical flow engine. Instead of requiring complex manual tuning or semi-automatic methods, the system uses intermediate weighted combinations of detection and optical flow results, with weights determined by tracking quality metrics, to produce accurate landmark positions for facial expression analysis.
Data Source
AI summary
An example system includes: a landmark detection engine to detect landmark positions of landmarks in images based on facial detection; an optical flow landmark engine to determine the landmark positions in the images based on optical flow of the landmarks between the images; a landmark difference engine to determine, for a landmark in a given image: a distance between a detected landmark position and an optical flow landmark position of the landmark; and a weighted landmark determination engine to determine, for a first and second image, a position for the landmark in the second image based on: a respective detected landmark position and a respective optical flow position of the landmark in the second image; and respective distances, determined with the landmark difference engine, between a first detected landmark position of the landmark in the first image and respective optical flow landmark positions for the first and second images.