Stereo Human Segmentation Training with Cross-View Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The challenge of training image segmentation models is exacerbated by the limited availability of human-labeled datasets, especially for diverse environments, leading to potential noise from pseudo-labels and suboptimal performance in applications like autonomous driving and XR.
Innovation Solution
Utilizing stereoscopic images from multiple cameras to calculate a loss value by projecting and converting segmentation masks between viewpoints, thereby enhancing the training of segmentation models without relying on additional human-labeled data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If pseudo-labels are used on unlabeled data to expand training dataset, then the coverage of data distributions is improved, but noise is introduced into the training process
Solution Approach 1:
The patent introduces stereo consistency as an intermediary mechanism to mediate between pseudo-labels and training. By projecting segmentation masks between stereo views and computing consistency losses, the system creates an intermediate validation layer that filters out unreliable pseudo-labels while preserving useful ones, thus expanding data coverage without proportionally increasing noise
Solution Approach 2:
The patent implements feedback loops where segmentation predictions are projected to stereo views, consistency losses are computed, and these losses feed back into model retraining. This iterative feedback process allows the model to progressively learn from unlabeled data while using stereo consistency as a quality control mechanism to prevent noise accumulation
2Reliability
If human-labeled datasets are used for training, then training data quality is improved, but the cost and availability are worsened due to limited labeled data
Solution Approach 1:
The patent creates virtual copies of labeled data through stereo projection. By projecting segmentation masks from one stereo view to another and using them as pseudo-labels, the system effectively multiplies the value of each human-labeled image, reducing the total number of labeled images needed while maintaining training quality
Solution Approach 2:
The system enables self-service training by automatically generating pseudo-labels from unlabeled stereo images and filtering them through consistency checks. This self-service mechanism reduces dependency on expensive human annotation while maintaining acceptable training data quality through automated stereo consistency validation
3Loss of time
If segmentation models are trained with limited diverse environments, then training time is reduced, but performance in deployment scenarios is worsened
Solution Approach 1:
The patent performs preliminary action by pre-computing stereo projections and consistency losses for unlabeled data before actual model training. This preparation creates a ready-to-use training pipeline that can quickly adapt to diverse environments without requiring extensive retraining, as the stereo consistency framework is already established and can process new environments efficiently
Data Source
AI summary
A system stores first and second images generated by first and second cameras; applies a segmentation model to the first image to generate a first segmentation mask identifying object instances; applies the segmentation model to the second image to generate a second segmentation mask identifying the object instances; projects the first segmentation mask to a viewpoint of the second camera to generate a first projected segmentation mask; converts the first projected segmentation mask and the second segmentation mask to first and second semantic masks, respectively; and computes a first similarity value based on the first and second semantic masks. This may be repeated exchanging the first and second images to compute a second similarity value. The system determines a loss value based on the first similarity value and the second similarity value and trains the segmentation model based on the loss value.