Image-Guided Text Translation With Selective Visual Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing multi-modal translation systems fail to effectively utilize visual information for improving translation quality, as visual information is often spliced with text vectors without considering its positive contribution, leading to translation results inferior to text-only models.
Innovation Solution
A text translation model that incorporates image information to correct initial translations by generating first image description information, using a trained text translation model to obtain target translations and provide target description information explaining the correction rationale.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If visual information is spliced and fused with text vector as global information and input into a model, then multi-modal translation is performed, but the translation result cannot be better than text-only machine translation
Solution Approach 1:
The patent segments the visual information processing into two distinct stages: first extracting local visual features from images, then selectively fusing them with text vectors only when needed. This segmentation allows the system to maintain text-only translation quality as a baseline while adding multi-modal capability only when visual information provides value, thus resolving the contradiction between adaptability and reliability.
Solution Approach 2:
The patent implements dynamic selection of visual information fusion based on the specific translation task and input content. The system dynamically determines whether to incorporate visual features by evaluating the relationship between text and image, allowing the translation model to adapt its behavior flexibly. This dynamic approach enables the system to achieve both text-only accuracy and multi-modal enhancement when appropriate.
2Adaptability or versatility
If visual information is always integrated with text translation, then translation coverage is improved, but translation accuracy for ambiguous words may deteriorate
Solution Approach 1:
The patent performs preliminary extraction of local visual features from images before the main translation process. These pre-extracted features are then selectively integrated based on the specific translation requirements. This preliminary action allows the system to prepare visual information in advance while maintaining the flexibility to use it only when it enhances accuracy, preventing deterioration of translation precision for ambiguous words.
Solution Approach 2:
The patent applies local quality by extracting and utilizing specific local visual features from images rather than using global visual information uniformly. The system identifies and integrates only the relevant local features that contribute to accurate translation of ambiguous terms, while ignoring unrelated visual content. This selective local integration maintains translation accuracy while expanding coverage.
3Adaptability or versatility
If visual information is fused with text vectors, then multi-modal translation is achieved, but the complexity of the translation system increases
Solution Approach 1:
The patent segments the complex visual information processing into independent modules: image input module, local feature extraction module, and selective fusion module. This segmentation allows each component to perform its specific function independently, reducing the overall system complexity while maintaining multi-modal translation capability. The modular architecture makes the system more manageable and easier to implement.
Solution Approach 2:
The patent applies partial action by implementing visual information fusion only when and where it is beneficial, rather than universally integrating it into all translation processes. The system uses a threshold or selection mechanism to determine when visual features should be incorporated, avoiding unnecessary computational complexity in cases where text-only translation is sufficient. This partial integration approach reduces system complexity while preserving multi-modal capability when needed.
Data Source
AI summary
A method of text translating method, a storage medium, an electronic device are provided. The method includes: obtaining a to-be-translated text, image information associated with the to-be-translated text, and an initial translation of the to-be-translated text; and inputting the to-be-translated text, the image information, and the initial translation into a trained text translation model to obtain a target translation and target description information. The text translation model is configured to obtain first image description information corresponding to the image information based on the image information, and correct the initial translation based on the first image description information and the to-be-translated text to obtain the target translation and the target description information.