Facial Landmark Detection With Multi-Style Decoder Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for facial landmark detection in images are limited to predicting a single style or type of landmark, requiring multiple models for different datasets and wasting time and memory, and are not adaptable across different annotation types.
Innovation Solution
A method and apparatus using a convolutional neural network to generate multi-level feature maps, followed by a cascaded decoder system with mask, self-attention, and transformable attention processing elements to predict coordinates of multiple landmark styles in a single model, allowing for flexible landmark detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a single model is used to detect multiple landmark styles, then model versatility and adaptability improve, but the model complexity increases
Solution Approach 1:
The patent implements a single landmark detection model that can detect multiple landmark styles (e.g., 68-point, 106-point, 214-point annotations) through a unified architecture. The model uses a backbone network (ResNet-50) shared across all landmark styles, with style-specific projection layers that map to different annotation types. This universal design allows one model to handle multiple detection tasks without requiring separate models for each landmark style, directly resolving the contradiction between versatility and complexity.
Solution Approach 2:
The patent segments the landmark detection task into style-independent and style-dependent components. The backbone network and feature extraction layers are shared across all styles (style-independent), while only the projection layers specific to each landmark style are different (style-dependent). This segmentation allows the model to maintain simplicity in the shared components while adapting to different styles through lightweight style-specific layers, effectively managing model complexity while achieving versatility.
2Measurement precision
If multiple models are trained for different datasets and landmark styles, then detection accuracy for specific styles improves, but training time and memory consumption increase
Solution Approach 1:
The patent trains a single multi-style landmark detection model that can handle multiple datasets and annotation types simultaneously. The model architecture includes a backbone network trained on multi-style data, with projection layers that can accommodate different landmark styles. This universal model eliminates the need to train separate models for each style or dataset, significantly reducing training time and memory consumption while maintaining high detection accuracy across all supported styles through the shared feature extraction capabilities.
3Measurement precision
If multiple models are trained for different landmark styles, then style-specific detection performance improves, but memory consumption increases
Solution Approach 1:
The patent employs a single memory-efficient model architecture that stores shared feature maps and representations for all landmark styles in one unified memory structure. The backbone network processes images once and generates style-independent features, which are then projected to different landmark styles through lightweight layers. This approach requires significantly less memory than training and storing multiple separate models, as the shared components are loaded only once and reused across all style-specific detection tasks.
Solution Approach 2:
The patent segments memory usage into style-independent shared memory (for backbone features) and style-dependent projection memory (for style-specific layers). The shared memory is loaded once and reused for all styles, while only the lightweight style-specific projection layers are activated during inference for each particular style. This segmentation dramatically reduces peak memory consumption compared to loading multiple complete models simultaneously.
Data Source
AI summary
A processor-implemented method including obtaining a multi-level feature map of a facial image through a convolutional neural network layer, generating an initial query matrix by fully connecting a feature map of a last level of the multi-level feature map through a fully connected layer, generating a memory feature matrix by flattening and concatenating the multi-level feature map, and determining, based on an input specifying at least one landmark style among a plurality of landmark styles, the memory feature matrix, and the initial query matrix, coordinates of a first landmark corresponding to the at least one landmark style of the facial image by using at least one decoder layer of one or more cascaded decoder layers.