Image Segmentation Training for Unknown Object Pixel Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks for semantic segmentation struggle to accurately detect unknown or anomalous objects in scene-centric images due to the vast number of possible out-of-distribution objects and class imbalances, leading to a bias towards predicting the majority class, which is critical in safety-critical applications like autonomous driving.
Innovation Solution
A method for generating out-of-distribution data by adapting the style of copied pixels using a masked noise encoder and StyleGAN2, combined with a loss function that focuses on logarithms of logits for anomalous pixels, allowing the network to learn to identify anomalies based on semantic content rather than style.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If neural networks are trained on scene-centric images with class imbalances, then the network can process real-world driving scenes, but the network develops bias towards predicting majority classes and fails to detect unknown or anomalous objects
Solution Approach 1:
The patent applies preliminary action by performing style transfer on training images before segmentation training. The style of source images is transferred to target images using neural style transfer, creating pre-processed training data that decouples semantic content from visual style. This preliminary style adaptation prevents the network from developing style-based biases and improves its ability to detect anomalies across different visual styles while maintaining reliability.
2Productivity
If the network relies on pixel style for anomaly detection, then detection may be simpler, but the network cannot generalize to different visual styles and becomes overfitted
Solution Approach 1:
The patent extracts and separates the style component from the semantic content component of images using neural style transfer. By transferring style from source images to target images during training, the method extracts style as an independent variable that can be controlled and varied. This allows the network to learn semantic features independently of style, improving generalization to different visual styles while maintaining detection capability.
3Adaptability or versatility
If more OoD exemplars are used during training, then the network can recognize more anomaly types, but the training data becomes harder to construct and manage
Solution Approach 1:
The patent applies copying by using style transfer to replicate and adapt existing training images into new style variations. Instead of manually constructing diverse OoD exemplars, the system copies semantic content from source images and overlays it with styles from target images, automatically generating diverse training examples. This reduces the complexity of training data construction while increasing the number of detectable anomaly types.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Computer-implemented method for training an image segmentation system (60) comprising the steps of: • Obtaining a training image (xi) and a training label map (ti), wherein the training label map (ti) indicates pixels of the training image (xi) as belonging to an anomaly class; • Providing the training image (xi) as input to the image segmentation system (60), wherein the image segmentation system (60) determines values characterizing logits for different classes for pixels of the training image (xi); • Determining a loss value based on a loss function, wherein the loss function comprises a first term that characterizes a logarithm of a sigmoid of logits determined for pixels in the training image (xi), wherein the logarithm of the sigmoid is determined for pixels, which are labeled as belonging to the anomaly class in the training label map (ti). • Training the image segmentation system (60) by minimizing the loss value.