Semantic Image Segmentation for New Classes Without Pixel Labels
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Semantic image segmentation models struggle to learn and segment objects from new classes without training data containing pixel-level annotations for those classes, limiting their ability to adapt to novel environments and objects.
Innovation Solution
The system employs a semantic image segmentation module with a learning module that updates parameters of the localizer, encoder, and decoder modules based on image-level class labels and semantic similarity maps, allowing the model to identify new classes without pixel-level annotations by leveraging semantic relationships between classes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the model is trained with pixel-level annotations for new classes, then segmentation accuracy for new classes is improved, but the complexity and cost of data annotation increases significantly
Solution Approach 1:
The patent introduces semantic similarity maps as an intermediary between image-level annotations and pixel-level segmentation. These maps serve as a bridge that translates coarse image-level labels into fine-grained pixel-level predictions by leveraging semantic relationships between classes, eliminating the need for direct pixel-level annotation of new classes
Solution Approach 2:
The system pre-computes semantic similarity maps between different object classes before training. These pre-computed similarity relationships are then used during training to guide the segmentation of new classes, allowing the model to leverage knowledge from previously seen classes without requiring new pixel-level annotations
2Ease of manufacture
If the model is trained only with image-level annotations without pixel-level annotations, then the ease of data collection is improved, but the segmentation precision for new classes deteriorates
Solution Approach 1:
Semantic similarity maps act as an intermediary that enriches image-level annotations with spatial and semantic information. The maps provide pixel-level guidance derived from semantic relationships, enabling the model to achieve segmentation precision comparable to pixel-level annotated training while using only image-level annotations
Solution Approach 2:
The patent changes the parameter representation by introducing semantic similarity scores between classes. Instead of directly using pixel coordinates from annotations, the system transforms the problem into learning from semantic similarity parameters, which can be derived from image-level labels and pre-computed similarity matrices
3Adaptability or versatility
If the model uses semantic similarity maps to learn new classes, then the adaptability to novel classes is improved, but the computational complexity increases
Solution Approach 1:
Semantic similarity maps are pre-computed offline between all object classes before training begins. This preliminary computation transfers the computational burden from the training phase to an offline preprocessing phase, allowing the training process itself to remain computationally efficient while still leveraging rich semantic relationships
Solution Approach 2:
The system applies semantic similarity information locally to each training sample by generating class-specific similarity maps. Rather than computing all possible class relationships globally, the method focuses computational resources on the specific semantic relationships relevant to each training instance, improving efficiency
Data Source
AI summary
A semantic image segmentation (SIS) system includes: a semantic segmentation module trained to segment objects belonging to predetermined classes in input images using training images; and a learning module configured to selectively update at least one parameter of each of a localizer module, an encoder module, and a decoder module of the semantic segmentation module to identify objects having a new class that is not one of the predetermined classes: based on an image level class for a learning image including an object having the new class that is not one of the predetermined classes; and without a pixel-level annotation for the learning image.