Neural Network Refinement With Multiple Maps for Coherent Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Likelihood-based semantic segmentation methods can generate incoherent semantic maps due to the independent consideration of each output pixel, leading to unconfident predictions in regions with label-related inconsistency, especially with noisy real-world datasets.
Innovation Solution
A computer processing system is configured to train a model for semantic image segmentation using a refinement neural network and a discriminator neural network. The refinement neural network generates multiple predicted segmentation maps from a predicted label distribution, and the model is trained using an objective function that includes a term representing the difference between the predicted label distribution and the average of the generated segmentation maps.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If likelihood-based semantic segmentation is used with per-pixel cross entropy loss, then each output pixel can be considered independently for simpler computation, but the generated semantic maps become incoherent and predictions become unconfident in regions with label-related inconsistency
Solution Approach 1:
The patent introduces an intermediary mechanism (consistency enforcement module) that mediates between independent pixel predictions and global semantic coherence. This module enforces constraints to ensure that predicted segmentation maps maintain consistency with the input image and with each other, resolving the contradiction between computational simplicity and prediction coherence.
2Reliability
If GANs are used to enforce inter-pixel consistency, then semantic map coherence can be improved, but the training dynamics become unstable and lead to poor classification outcomes
Solution Approach 1:
The patent extracts the coherence enforcement function from the GAN framework and implements it as a separate, dedicated consistency enforcement module. This separation allows the model to benefit from inter-pixel consistency without the training instability inherent in GANs, as the consistency module can be trained independently and then integrated with the segmentation model.
Solution Approach 2:
The patent segments the training process into distinct phases: first training the segmentation model independently, then training the consistency enforcement module separately, and finally integrating them. This segmented approach avoids the unstable joint training dynamics of GANs while still achieving coherent semantic maps.
3Stability of the object's composition
If adversarial loss term is combined with pixel-level cross-entropy loss term, then training dynamics can be stabilized, but the model still struggles with noisy training data and does not achieve optimal performance
Solution Approach 1:
The patent converts the harmful effect of noisy training data into a benefit by using the consistency enforcement module to identify and correct inconsistencies caused by noise. The module enforces constraints that filter out noise-induced errors, transforming the challenge of noisy data into an opportunity to improve robustness and accuracy.
Data Source
AI summary
A computer processing system is configured to train a model for use in semantic image segmentation. The model comprises a refinement neural network, a discriminator neural network. The refinement neural network is configured to receive a predicted label distribution for an image, obtain one or more random values from a random or pseudo-random noise source, use the one or more random values to generate a plurality of predicted segmentation maps from the received predicted label distribution and output the plurality of predicted segmentation maps to the discriminator neural network. The computer processing system is configured to train the refinement neural network using an objective function that is a function of an output of the discriminator neural network and that further includes a term representative of a difference between the predicted label distribution and an average of the plurality of predicted segmentation maps output by the refinement neural network for the predicted label distribution.