Face Landmark Coordinates via Spatially Preserving Cascaded Decoders
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing face landmark detection methods face challenges such as lower accuracy due to non-differentiable post-processing operations in heat map regression-based approaches and loss of spatial structure in coordinate regression-based methods, leading to quantization errors and decreased localization performance.
Innovation Solution
A multi-stage feature map generation using a staged convolutional network, followed by a cascaded decoder network with self-attention and deformable attention mechanisms, where initial query embeddings are derived from meaningful image features, reducing the number of decoder layers and utilizing a regression loss function for training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If heat map regression-based approach is used, then detection accuracy is improved, but non-differentiable post-processing operations prevent end-to-end training
Solution Approach 1:
The patent replaces the non-differentiable post-processing operations (peak detection, coordinate extraction) with a differentiable coordinate regression module. This substitution allows the entire network to be trained end-to-end using gradient-based optimization, eliminating the need for separate post-processing steps while maintaining detection accuracy.
2Extent of automation
If coordinate regression-based approach with fully connected prediction layer is used, then end-to-end training is enabled, but spatial structure of image features is destroyed
Solution Approach 1:
The patent segments the feature processing into two distinct paths: a spatial structure preservation path that maintains the convolutional feature map structure for accurate localization, and a coordinate prediction path that enables end-to-end training. This segmentation allows both goals to be achieved simultaneously without compromising either.
Solution Approach 2:
The patent introduces an intermediary coordinate prediction module that acts as a bridge between the spatial feature maps and the final landmark coordinates. This intermediary preserves the spatial structure information from the convolutional features while enabling differentiable coordinate regression, thus maintaining both localization performance and end-to-end training capability.
3Device complexity
If heat map resolution is reduced, then computational complexity is decreased, but quantization errors occur and prevent further improvement
Solution Approach 1:
The patent replaces the heat map regression mechanism with direct coordinate regression. This substitution eliminates the need for high-resolution heat maps and subsequent quantization, allowing the model to predict landmark coordinates directly from the feature maps without suffering from resolution-related quantization errors.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method and apparatus with landmark coordinate prediction are provided. The method includes generating a multi-stage feature map for landmarks of a face image through a staged convolutional network, generating an initial query matrix by fully connecting a last-stage feature map in the multi-stage feature map using a fully connected network, where a total number of feature elements in the initial query matrix is equal to a total number of predicted landmarks of the face image, generating a memory feature matrix by flattening and connecting the multi-stage feature map, generating the predicted landmark coordinates by inputting the memory feature matrix and the initial query matrix to a decoder network of plural cascaded decoder networks.