Semantic Segmentation Network Training for Temporal Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks for semantic image segmentation lack stability and temporal consistency, particularly in applications like automated driving, leading to inconsistent predictions across consecutive frames, which is a critical safety concern.
Innovation Solution
A method that combines frame-based and sequence-based evaluations to optimize both accuracy and temporal consistency of semantic segmentation predictions, using a weighted sum of cross-entropy and temporal consistency losses to stabilize predictions over time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If adversarial training or augmentation training is used to improve stability, then temporal consistency is improved, but training time and labeling costs increase significantly
Solution Approach 1:
The system uses the neural network's own predictions and optical flow estimates to generate temporal consistency loss signals, eliminating the need for external adversarial perturbations or manual labeling. The network self-regulates its temporal consistency through the computed loss between consecutive frame predictions
Solution Approach 2:
A feedback loop is established where temporal consistency loss is computed from the difference between current frame predictions and optically-flow-warped previous frame predictions. This feedback signal is then used to update the network weights, creating a continuous self-improving mechanism for temporal stability
2Measurement precision
If label propagation techniques are used to increase training data, then accuracy is improved, but training time increases and temporal stability is not significantly improved
Solution Approach 1:
Instead of discrete batch retraining with propagated labels, the system continuously optimizes temporal consistency during normal operation by computing loss on every frame sequence and updating weights incrementally. This maintains accuracy while avoiding the computational overhead of repeated full retraining cycles
Solution Approach 2:
Optical flow serves as an intermediary that bridges consecutive frame predictions, enabling the computation of temporal consistency loss without requiring manual labels or complex label propagation algorithms. The optical flow field mediates the comparison between frames by providing pixel-level correspondence
3Reliability
If pixel-level temporal loss function is used to enforce temporal consistency, then temporal stability is improved, but computational complexity increases
Solution Approach 1:
The system extracts only the essential temporal consistency signal by computing loss at key prediction points and using optical flow to warp predictions, rather than processing all pixel-level temporal variations. This extracts the core temporal stability requirement while discarding computationally expensive redundant calculations
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present invention is related to a method, a computer program, and an apparatus for training a neural network for semantic image segmentation. The invention is further related to an in-car control unit or a backend system, which make use of such a method or apparatus, and to a vehicle comprising such an in-car control unit. In a first step, image data of a sequence of image frames are received (10). Then a frame-based evaluation of semantic segmentation predictions of one or more objects in individual image frames is performed (11). Furthermore, a sequence-based evaluation of temporal characteristics of semantic segmentation predictions of said one or more objects in at least two image frames is performed (12). The results of the frame-based evaluation and the sequence-based evaluation are combined (13).