Flexible Image Segmentation Using Latent Mask Representations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing image segmentation methods are inflexible and struggle to handle unexpected objects that do not fit into a fixed catalogue of classes, particularly in automated driving applications, necessitating a more adaptable and accurate 'open-vocabulary' segmentation.
Innovation Solution
A method using a vision processing network that outputs pixel masks and a weight matrix, combined with an encoder network, iteratively refines these masks and matrices to adapt to the input image, leveraging a CLIP network for semantic alignment and transformer networks for attention relationships, allowing for flexible and accurate segmentation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a fixed catalogue of classes is used for segmentation, then the segmentation process is simple and fast, but the system cannot handle unexpected objects that do not fit into the predefined classes
Solution Approach 1:
The patent applies dynamics by making the segmentation system adaptive rather than static. The vision processing network dynamically generates segmentations based on input images without being constrained by a fixed class catalogue. The system evolves its understanding of objects through the encoder network that processes image data and generates segmentations on-the-fly, allowing it to adapt to unexpected objects while maintaining operational efficiency.
2Adaptability or versatility
If open-vocabulary segmentation is implemented to handle unexpected objects, then the system becomes more flexible, but the computational complexity and processing time increase
Solution Approach 1:
The patent applies preliminary action through the encoder network that is pre-trained and prepared to process images. The vision processing network is pre-configured with the capability to generate segmentations for any object type. This preliminary preparation allows the system to handle open-vocabulary segmentation requests efficiently without requiring extensive computational resources during actual processing, as the fundamental segmentation capability is already established.
3Measurement precision
If iterative refinement processes are used to improve segmentation accuracy, then the precision increases, but the computational resources and processing time required increase
Solution Approach 1:
The patent applies feedback through the iterative refinement process where the vision processing network generates initial segmentations, the encoder network processes these results, and the segmentations are refined based on the feedback from the encoder. This feedback loop continues until the desired accuracy is achieved or a maximum number of iterations is reached. The feedback mechanism ensures that computational resources are used efficiently by refining only when necessary and stopping when sufficient accuracy is obtained.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method (100) for determining a segmentation (7) of an input image (1), said input image (1) comprising pixels carrying pixel values, said segmentation (7) assigning, to each pixel of the input image (1), a class of an entity that has given rise to the pixel value of this pixel, the method comprising the steps of: • providing (110) the input image (1) to a vision processing network (2a) that outputs masks (3) designating sets of pixels belonging to different object types, as well as an associated weight matrix (4) that is indicative of distinguishing features characterizing entities of different types in the input image (1); • transforming (120), by an encoder network (5), said weight matrix (4) in combination with the input image (1) into at least one mask representation (6) in a latent space that is a notion of assignments of classes to masks; • processing (130) the input image (1), together with said mask representation (6), by a vision processing network (2b, 2c), into a refinement (3*, 3**) for the masks (3) and a refined weight matrix (4*, 4**); • transforming (140), by the encoder network (5), the refined weight matrix (4*, 4**) in combination with the input image (1) into at least one refined mask representation (6*, 6**) in the latent space; and • computing (150) the sought segmentation (7) from the masks (3) and the at least one mask representation (6) in the latent space, or from further refinements (3*, 3**; 6*, 6**) thereof obtained by further passes through a vision processing network (2b, 2c) and the encoder network (5).