Semantic Image Augmentation for Class Imbalance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image augmentation methods for neural networks do not effectively address class imbalance in training datasets, as they fail to incorporate object-specific context, leading to poor performance in detecting underrepresented object classes during inference.
Innovation Solution
The method involves semantic augmentation by adding contextually relevant objects to images using natural language processing techniques, such as word embeddings, without requiring additional neural networks or significant processing power, to generate augmented images that balance class distribution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If existing image augmentation methods (geometric or photometric changes) are used to expand training dataset size, then the number of training samples increases, but object-specific information is not augmented and class imbalance remains unresolved
Solution Approach 1:
The patent applies local quality by performing object-specific augmentation rather than global image modification. It identifies individual objects in images and adds contextually relevant objects near them, ensuring that each object region receives targeted augmentation appropriate to its class, thereby addressing class imbalance while preserving object-specific information.
Solution Approach 2:
The patent uses an intermediary approach by introducing contextually relevant objects as mediators between existing objects and the training process. Instead of directly modifying existing objects or using simple geometric transformations, it inserts intermediate objects that provide contextual information and help balance class representation in the training dataset.
2Reliability
If additional neural networks are used to learn image context for object-aware augmentation, then contextual relevance of augmented objects improves, but processing power and memory resources increase
Solution Approach 1:
The patent uses copying by replicating existing objects from the training dataset and inserting them into new contexts. Instead of training additional neural networks to generate new objects, it copies existing object instances and places them near relevant objects in the image, preserving contextual relevance while avoiding the computational overhead of additional neural networks.
Solution Approach 2:
The patent applies parameter changes by modifying the spatial parameters (position, scale, rotation) of copied objects to ensure they are appropriately placed and sized relative to the target object. This allows contextual relevance to be achieved through parameter adjustment rather than through complex neural network processing.
Data Source
AI summary
Methods and systems for generating a semantically augmented image are disclosed. An embedding is generated for each object label associated with a segmented input image. For each embedding associated with a respective object label, a similarity score is computed between the embedding associated with the object label and an embedding representing an object class in an object bank storing a plurality of object images. At least one object is selected, the selected object being associated with a respective object image in the object bank, the selected at least one object being from an identified object class that is identified as contextually relevant to at least one object label associated with the segmented input image, based at least on the similarity score. The selected object is added into the segmented input image to generate the augmented image.