Video Face Clustering via Motion Tracker and Fine-Tuned Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Video face clustering in movie and TV datasets is challenging due to variations in facial pose, expression, appearance, lighting, size, and cinematic styles, which hinder the performance of pre-trained face identification models and make hand labeling tedious and error-prone.
Innovation Solution
A computer-implemented method that uses a motion tracker to determine face tracks from image frames, fine-tunes a pre-trained face identification model to generate embeddings with high similarity, and groups face tracks into common identity clusters based on these embeddings, leveraging self-supervised learning to optimize the model for clustering without negative training examples.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If pre-trained face identification models are used directly, then the model can be applied immediately without additional training, but the model performs badly due to variations in facial pose, expression, appearance, lighting, and cinematic styles
Solution Approach 1:
The patent applies preliminary action by fine-tuning the pre-trained face identification model on a subset of the movie dataset before actual clustering. This preliminary fine-tuning adapts the model to cinematic variations in lighting, pose, and expression, allowing it to perform reliably on the full dataset without requiring complete retraining or manual labeling
2Reliability
If hand labeling of characters is performed to train supervised models, then the model can be trained on movie-specific data, but the process is tedious, time-consuming, costly and error-prone
Solution Approach 1:
The patent applies self-service by implementing an unsupervised clustering algorithm that automatically groups face tracks without requiring manual labels. The system uses a pre-trained face identification model to generate embeddings and then clusters them using a loss function, eliminating the need for time-consuming hand labeling while still achieving movie-specific adaptation through automated fine-tuning
3Reliability
If a common loss function is used for fine-tuning and clustering, then the fine-tuned model is optimized for clustering, but the model requires additional fine-tuning before use
Solution Approach 1:
The patent applies partial action by fine-tuning the model on a subset of the movie dataset rather than the entire dataset. This partial fine-tuning using a common loss function optimizes the model for clustering while requiring significantly less time and computational resources than complete retraining, achieving a balance between optimization and efficiency
Data Source
AI summary
A method includes determining, using a motion tracker, a plurality of face tracks from one or more sequences of image frames. Each face track corresponds to a respective instance of a respective face and includes a respective sequence of image frame crops. The method includes fine-tuning, using the determined plurality of face tracks, a pre-trained face identification model to generate, for image frame crops of a common face track, respective embeddings that have a mutually high degree of similarity as measured by a loss function. The method then includes grouping the plurality of face tracks into common identity clusters based at least in part on similarities, as measured by the loss function, between respective embeddings generated using the fine-tuned face identification model for image frame crops within different face tracks.