Language Feature Extraction Model for Image-Text Association
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current AI systems require a large number of text-image pairs for learning, making it difficult to generate a general-purpose language feature extraction model that can accurately convert text into feature vectors for various tasks, such as medical image diagnosis, without requiring extensive correct answer feature data.
Innovation Solution
A method is developed to train a language feature extraction model using a system with multiple processors, where a text related to an image is processed to output a feature amount including the position information of a region of interest, allowing the model to estimate the region of interest without needing extensive correct answer feature data, even with a small amount of learning data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a large number of text-image pairs are used for learning, then the model can be trained more accurately, but the data requirements and system complexity increase significantly
Solution Approach 1:
The patent introduces an image processing model as an intermediary component that extracts visual features from images, which are then combined with text features through a cross-modal fusion mechanism. This intermediary feature extraction approach allows the system to learn effective representations without requiring large amounts of paired text-image data, as the image processing model pre-processes visual information into compact feature vectors that can be more efficiently learned from limited paired data.
2Adaptability or versatility
If a language feature extraction model is designed to be general-purpose, then it can be applied to various tasks, but it requires extensive correct answer feature data for training
Solution Approach 1:
The patent segments the feature extraction process into distinct components: an image processing model that handles visual features, a text processing model that handles linguistic features, and a cross-modal fusion mechanism that integrates them. This segmentation allows each component to be optimized independently and applied to different tasks, providing general-purpose functionality without requiring extensive task-specific correct answer data for the entire system.
Solution Approach 2:
The patent creates a universal language feature extraction model that can serve multiple purposes: extracting features for text classification, text similarity search, and image-text association tasks. The model achieves this universality by incorporating cross-modal fusion capabilities that allow it to process both text and image inputs effectively, enabling a single model to be applied across various NLP and multimodal tasks without requiring separate training data for each application.
3Measurement precision
If the model processes text to output feature amounts including position information, then it can estimate region of interest more accurately, but the processing complexity increases
Solution Approach 1:
The patent applies preliminary action by having the image processing model extract visual features and identify potential region of interest locations before the text processing occurs. The extracted image features include spatial information that pre-marks candidate regions, which then guides the text processing model to focus on relevant portions of the input text. This preliminary feature extraction reduces the complexity of subsequent processing by narrowing down the search space for region of interest estimation.
Data Source
AI summary
A method of generating a language feature extraction model that causes a computer to extract a feature from a text related to an image, includes that a system performs machine learning using training data including a first image, first position information related to a region of interest in the first image, and a first text that describes the region of interest to input the first text into a first model, which is the language feature extraction model, to cause the first model to output a first feature amount, input the first image and the first feature amount into a second model to cause the second model to estimate the region of interest, and train the first model and the second model such that an estimated region of interest output from the second model matches the region of interest of a correct answer indicated by the first position information.