Segmentation Training Images With Synthetic Anomaly Embedding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing semantic segmentation neural networks struggle to detect unknown or anomalous objects in scene-centric images due to the vast number of possible out-of-distribution objects and class imbalances, leading to a bias towards predicting the majority class, which is critical for safety-critical applications like autonomous driving.
Innovation Solution
A method for generating training data by adapting the style of out-of-distribution data using a masked noise encoder and StyleGAN2 to create realistic embeddings of anomalous objects in scene-centric images, with labeled pixels marked as anomalies, and training the network using a specialized loss function to enhance detection capabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If neural networks are trained on scene-centric images with class imbalances, then the network can handle real-world driving scenes, but the network develops bias towards predicting majority classes and fails to detect unknown or anomalous objects
Solution Approach 1:
The patent applies preliminary action by generating synthetic anomaly examples before training the neural network. Out-of-distribution objects are created and embedded into training scenes in advance, allowing the network to learn anomaly detection patterns during training rather than encountering them for the first time during deployment. This proactive preparation resolves the contradiction by equipping the network with anomaly detection capability while maintaining its ability to handle real-world scenes.
Solution Approach 2:
The patent uses an intermediary approach by introducing synthetic anomaly embeddings as a bridge between normal training data and real-world anomalies. These synthetic intermediaries are generated by embedding out-of-distribution objects into scene-centric images, creating a middle ground that teaches the network to distinguish anomalies without requiring actual unknown objects during training. This intermediary training data resolves the contradiction between handling real scenes and detecting unknown objects.
2Adaptability or versatility
If more diverse out-of-distribution objects are included in training data, then the network's ability to detect unknown objects improves, but the complexity of data generation and training increases
Solution Approach 1:
The patent applies copying by creating synthetic representations of out-of-distribution objects rather than requiring actual diverse anomaly data. The system embeds synthetic anomaly examples into training scenes, copying the essential characteristics of unknown objects in a controlled manner. This copying approach resolves the contradiction by providing diverse anomaly training examples without the complexity of collecting and managing real-world diverse anomaly data.
Solution Approach 2:
The patent uses parameter changes by modifying the embedding parameters of out-of-distribution objects when generating synthetic anomaly examples. By adjusting parameters such as object size, position, and embedding strength, the system can generate diverse anomaly training data from a single out-of-distribution object. This parameter-based approach resolves the contradiction by achieving diversity in training data without requiring diverse source objects, thus reducing data generation complexity.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Computer-implemented method for generating a training image (xi) and a training label map (ti) for training and/or testing an image segmentation system (60), wherein the method comprises the steps of: • Obtaining a first image (i1) and a first label map (l1) corresponding to the first image (l1) and obtaining a second image (i2) and a second label (l2) map corresponding to the second image (i2); • Determining a third image (i3), wherein the third image is determined by providing the second image (i2) as input to a machine learning system (70), wherein the machine learning system (70) is configured for determining images in the style of the first image (i1) based on provided images; • Determining the training image (xi) by replacing pixels from the first image (i1) with pixels from the third image (i3), wherein the pixels from the third image (i3) are determined based on a class from the second label map (l2); • Determining the training label (ti) map by replacing class labels from the first label map (l1) with class labels indicating an anomaly class, wherein the class labels in the first label map (l1) are replace for which corresponding pixels in the first image (i1) are replaced.