Video Label Augmentation via Bidirectional Propagation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Semantic segmentation in autonomous driving faces challenges due to limited publicly available datasets and the time-consuming process of manual labeling, which hinders the development of accurate models for complex scenarios.
Innovation Solution
A method for targeted retraining of models using automatically generated high-quality training data from a small number of preselected ground-truth labeled video frames, employing forward and backward probabilistic models for label propagation and pixel-level analysis to increase the quantity of available ground truth labels, handle occlusions, and reduce erroneous labeling.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual labeling is used to create ground-truth labels for semantic segmentation, then labeling accuracy is high, but the time and cost required increases significantly
Solution Approach 1:
The patent applies preliminary action by performing manual labeling only on a small subset of video frames (e.g., every 10th frame) before automatically propagating these labels through the entire video sequence using optical flow and warping techniques. This preliminary manual labeling provides the foundation for subsequent automated label generation, significantly reducing the overall time and cost while maintaining acceptable accuracy through the propagation process.
2Adaptability or versatility
If the number of semantic classes in the model is increased to cover more situations, then model coverage and accuracy improve, but the amount of training data required increases substantially
Solution Approach 1:
The patent applies copying by automatically generating synthetic training data through label propagation from a small set of manually labeled frames. The system creates numerous additional labeled frames by warping and transforming the original labels according to optical flow and geometric transformations, effectively copying and adapting the limited ground-truth data to generate a large training dataset that supports models with many semantic classes without requiring proportional manual labeling effort.
3Measurement precision
If pixel-level analysis is performed instead of super-pixel level analysis, then labeling accuracy improves, but the computational complexity and processing time increase
Solution Approach 1:
The patent applies segmentation by first dividing the video frames into super-pixels or patches, performing label propagation at this coarser level where computations are more efficient, and then refining the results to pixel-level precision in a second pass. This two-stage segmentation approach maintains high accuracy while reducing computational complexity by avoiding direct pixel-level processing throughout the entire pipeline.
4Measurement precision
If a first classifier training step is performed before label propagation, then initial label quality improves, but processor time and energy consumption increase
Solution Approach 1:
The patent applies taking out by removing the preliminary classifier training step from the traditional pipeline and directly proceeding with label propagation from manually labeled frames. This extraction of the unnecessary intermediate step reduces processor time and energy consumption while maintaining label quality, as the manual labels provide sufficient initial quality without requiring additional automated training iterations.
Data Source
AI summary
A method for processing video data comprising a plurality of image frames, the method comprising: obtaining a forward model and a backward model of the plurality of image frames; processing the forward model and the backward model to propagate at least one label in the region or patch to at least one other image frame of the video sequence, using a probabilistic method for estimating the label in the at least one other image frame in forward and backward correspondences, wherein, during the processing, a pixel having a most likely label with a probability lower than a threshold value is assigned a predetermined generic label; and generating a labelled result for any given image frame by applying an image label difference, based on label uncertainty between the forward and backward correspondences, to the given image frame.