Multimodal Learning for Target Position Estimation Across Object Sizes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies struggle to accurately estimate a target position in an image designated with reference to a reference object, particularly when the object's size varies in the input image.
Innovation Solution
A learning apparatus and method that utilizes a multimodal model with a hierarchical structure, including a feature extraction unit, text extraction unit, reference object encoding unit, and target position estimation unit, to accurately estimate the target position by processing input images and texts, using a loss function that considers distance, angle, and penalty areas to refine the estimation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a single-resolution feature representation is used for the reference object, then the model complexity is reduced, but the target position estimation accuracy deteriorates when reference objects have varying sizes
Solution Approach 1:
The patent segments the feature representation of the reference object into multiple resolutions. The encoding unit generates feature amounts at different resolutions (e.g., first resolution, second resolution, third resolution) to capture both fine-grained and coarse-grained spatial information. This segmentation allows the model to accurately estimate target positions for reference objects of varying sizes without requiring an overly complex architecture, as each resolution level contributes specific spatial details.
2Measurement precision
If multiple feature amounts with different resolutions are generated for the reference object, then the target position estimation accuracy is improved, but the computational cost increases
Solution Approach 1:
The patent employs dynamic feature generation where the encoding unit adaptively creates feature amounts at multiple resolutions based on the input image and text. The system dynamically adjusts which resolution levels are generated and how they are combined in the decoding unit, optimizing computational resources while maintaining estimation accuracy. This dynamic approach allows the model to allocate computational effort efficiently rather than processing all possible resolutions uniformly.
Solution Approach 2:
The patent adds the resolution dimension to the feature representation. Instead of using a single feature vector, the system generates feature amounts across multiple resolution dimensions (first resolution, second resolution, third resolution). This dimensional expansion allows the model to capture spatial information at different scales, improving target position estimation accuracy while the decoding unit efficiently integrates these multi-dimensional features to manage computational complexity.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A learning apparatus (100) configured to perform machine learning is provided. The learning apparatus is characterized by comprising: acquisition means configured to acquire teaching data (111) including input data (112) and ground truth data (113), the input data including an input image (202) and an input text (201), the input image including a reference object (203), the input text relatively designating a target position with reference to the reference object; generation means configured to generate output data by inputting the input data to a model (400), the output data being for specifying the target position; and update means configured to update a parameter of the model so as to reduce a loss obtained by inputting the output data and the ground truth data to a loss function (405). The model includes: a first submodel (401) that generates, based on the input image and the input text, a plurality of feature amounts representing the reference object, the plurality of feature amounts having different resolutions from each other; and a second submodel (404) that generates the output data based on the plurality of feature amounts and the input text, and each of the plurality of feature amounts is input to the second submodel.