Scribble-Based Image Segmentation Using ML Loss Functions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing semantic segmentation methods face performance issues due to limited data volume and diversity, inter-annotator agreement problems, and the inability to efficiently annotate images with multiple objects or irregular shapes using traditional user inputs like points or boxes.
Innovation Solution
A processor-implemented method using a machine learning model for generating an optimized image with scribble-based annotation, which segments images into classes, allows users to mark or scribble on unclassified areas, assigns color masks, computes a loss function, and modifies weights to match user annotations, enabling efficient annotation of complex images.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If traditional interactive segmentation methods annotate one object at a time using points or boxes, then the annotation process is simple to implement, but the overall annotation time increases significantly for images with multiple objects
Solution Approach 1:
The patent merges multiple object annotation operations into a single scribble-based operation. Instead of requiring users to annotate each object separately using points or boxes, the system accepts freehand scribbles that can simultaneously define multiple objects and their spatial relationships, thereby reducing the total number of annotation operations while maintaining segmentation accuracy
Solution Approach 2:
The patent segments the annotation task into two phases: an automated segmentation phase that processes the scribble input and generates initial object boundaries, followed by a refinement phase that allows users to correct errors. This segmentation of the annotation process enables faster processing while maintaining accuracy through human-in-the-loop refinement
2Measurement precision
If existing methods use shape-based regularization with anchor points to refine boundaries, then the segmentation is precise for single objects, but the method cannot handle complex images with multiple objects or irregular shapes
Solution Approach 1:
The patent changes the fundamental parameter of user input from structured geometric constraints (points, boxes, polygons) to freehand scribble curves. This parameter change enables the system to handle any object shape or configuration while maintaining boundary accuracy through the loss function that measures the distance between scribble points and segmented boundaries
Solution Approach 2:
The patent introduces a loss function as an intermediary that mediates between the scribble input and the segmentation output. The loss function computes the distance between scribble points and the segmented object boundaries, providing a gradient signal that guides the optimization process to align the segmentation with the user's scribble annotations, thereby achieving both precision and versatility
3Reliability
If semi-automated methods require joint training using images and user inputs, then the system achieves good performance during training, but performance deteriorates when new annotators who did not provide training inputs try to annotate images
Solution Approach 1:
The patent implements a self-service annotation system where the deep learning model automatically performs segmentation without requiring new annotators to provide training inputs. The model uses pre-trained knowledge and the loss function to automatically adjust to new scribble inputs, eliminating the need for retraining when new annotators join the system
4Productivity
If limited datasets are used for training semantic segmentation models, then the training process is faster and requires less data storage, but the model performance deteriorates due to insufficient data volume and diversity
Solution Approach 1:
The patent performs preliminary training on a large, diverse dataset to create a robust base model with general segmentation capabilities. During deployment, the system uses the loss function to efficiently fine-tune the model on new scribble annotations without requiring retraining on the entire large dataset, thereby maintaining high performance while ensuring training efficiency
Data Source
AI summary
A system and method for generating an optimized image with scribble-based interactive image segmentation model using a machine learning are provided. The method includes, (i) segmenting, using a machine learning model, an image to classify into classes each class is represented with a label, (ii) displaying the classified image which specifies the classes on the classified image with outlines, (iii) enabling a user to scribble on the classified image to annotate the classes if an area is not classified, (iv) assigning a color mask for each scribbled area, (v) computing, using the machine learning model, a loss function for a location of pixels based on color mask, (vi) modifying pre-defined weights for each scribbled area to match the annotated image and a determined class on the classified image, and (vii) generating the optimized image if the annotated image is matched with the determined class on the classified image.