Online Segmentation Adaptation Using Keypoints and Bounding Regions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Segmentation neural networks perform poorly in situations with large domain-shifts, such as from indoor to outdoor or daylight to nighttime, and rely on unreliable pixel pseudo-labels for online adaptation, leading to inaccurate segmentation.
Innovation Solution
Utilize bounding regions and keypoints for generating pseudo-labels during online adaptation, leveraging their reliability for fine-tuning segmentation models to improve segmentation accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If pixel pseudo-labels are used for online adaptation, then the segmentation model can be adapted to new environments, but the segmentation accuracy deteriorates due to unreliability of pixel pseudo-labels under large domain-shifts
Solution Approach 1:
The patent introduces bounding box predictions and keypoint predictions as intermediary elements that serve as more reliable pseudo-labels compared to pixel-wise pseudo-labels. These intermediaries provide coarse-grained supervision signals that are less sensitive to domain shifts, thereby maintaining segmentation accuracy while enabling online adaptation.
Solution Approach 2:
The patent utilizes readily available bounding box and keypoint predictions from the segmentation model itself as disposable pseudo-labels for online adaptation. These predictions are generated on-the-fly during inference and used to update the model parameters, eliminating the need for expensive annotated datasets while maintaining adaptation effectiveness.
2Manufacturing precision
If the segmentation model is trained on source domain data, then it achieves good performance on source data, but performance deteriorates when applied to target domain data with large domain-shifts
Solution Approach 1:
The patent performs preliminary online adaptation by updating model parameters using bounding box and keypoint pseudo-labels generated from target domain data before final segmentation inference. This preliminary adaptation step prepares the model for target domain characteristics, improving generalization capability while maintaining accuracy.
Solution Approach 2:
The patent dynamically changes model parameters during online adaptation by updating weights and biases based on self-supervised learning objectives derived from bounding box and keypoint predictions. This parameter adaptation allows the model to transition from source domain optimization to target domain performance without retraining from scratch.
Data Source
AI summary
Techniques and systems are provided for performing online adaptation of machine learning model(s). For example, a process may include obtaining features extracted from a image by a machine learning model during inference and determining, by the machine learning model based on the features during inference, a plurality of keypoint estimates in the image and/or a bounding region estimate associated with an object in the image. The process may further include generating pseudo-label(s) based on the plurality of keypoint estimates and/or the bounding region estimate. The process may include determining at least one self-supervised loss based on the plurality of keypoint estimates and/or the bounding region estimate. The process may further include adapting, based on the at least one self-supervised loss, parameter(s) of the machine learning model. The process may include generating, using the machine learning model with the adapted parameter(s), a segmentation mask for the image (or another image).