Distilled Video Object Segmentation for Real-Time On-Device Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing semi-supervised video object segmentation (SVOS) methods are not suitable for on-device deployment due to high memory requirements and lack of real-time performance, especially on resource-constrained devices like mobile phones, and they do not effectively leverage user data or interaction.
Innovation Solution
A method for locally training a machine learning model on user devices using a high-resolution and low-resolution configuration, employing knowledge distillation to reduce memory usage and improve accuracy, utilizing user-generated video clips and interaction for training, and combining locally-trained models to enhance performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If memory-based matching methods are used to improve SVOS accuracy, then segmentation accuracy is improved, but memory usage increases making it unsuitable for on-device deployment
Solution Approach 1:
The model is divided into two configurations: a high-resolution teacher model for accurate segmentation and a low-resolution student model for efficient on-device operation. The student model processes low-resolution video clips to reduce memory usage while the teacher model provides guidance for maintaining segmentation accuracy through knowledge distillation.
Solution Approach 2:
The patent changes the resolution parameter of the input video clips processed by the student model. By processing low-resolution video clips instead of full-resolution clips, the memory requirements are significantly reduced while the knowledge distillation from the teacher model compensates for the loss in detail, enabling on-device deployment.
2Manufacturing precision
If high-resolution processing is used to maintain segmentation quality, then segmentation quality is maintained, but processing speed decreases preventing real-time performance
Solution Approach 1:
The system segments the processing task between two models with different resolution requirements. The student model operates on low-resolution clips to achieve real-time processing speeds, while the teacher model periodically processes high-resolution clips to maintain segmentation quality through knowledge distillation.
Solution Approach 2:
The patent changes the resolution parameter based on the processing stage and model type. Student model uses low-resolution processing for speed-critical real-time operation, while teacher model uses high-resolution processing for quality-critical knowledge generation, optimizing both speed and quality across the system.
Data Source
AI summary
The present techniques provide methods for training machine learning, ML, models to track an object through frames of a video even when the object may change shape, orientation, position, proximity and angle to a camera that captured the video, and so on.