Multimodal Translation Model with Shared Neural Layers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Multimodal translation methods face challenges such as loss of scene information during text extraction, high memory consumption due to domain-specific translation models, and long translation latency, especially when processing on devices with limited computing power.
Innovation Solution
A multimodal feature fusing method based on location attention is employed to accurately predict domain labels, and a neural network layer selection method is used to share layer parameters across domains, reducing memory consumption and improving translation speed by selecting relevant layers for decoding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If domain-specific translation models are used for each domain, then translation accuracy is improved, but memory consumption increases
Solution Approach 1:
The patent merges multiple domain-specific translation models into a single unified model by sharing encoder and decoder layer parameters across different domains. Only domain-specific embedding layers and classification heads are maintained separately, while the core neural network architecture is shared, reducing memory consumption while preserving translation accuracy for multiple domains
Solution Approach 2:
The unified translation model is designed to serve multiple domains universally by incorporating domain identification mechanisms that dynamically adapt the model's behavior to the specific domain of input text, allowing a single model to perform specialized translation tasks across different domains without requiring separate dedicated models for each
2Measurement precision
If domain-specific translation models are trained for each domain, then translation quality is improved, but translation latency increases
Solution Approach 1:
The patent combines multiple domain-specific models into one unified model with shared parameters, reducing the total number of computations required. The domain identification component quickly determines the input domain and routes it through the shared model architecture, eliminating the need to load and process through multiple separate domain-specific models, thus reducing translation latency
Solution Approach 2:
The model performs preliminary domain identification at the beginning of the translation process by analyzing the input text and determining its domain classification. This preliminary action allows the system to prepare the appropriate processing path in advance, optimizing the translation workflow and reducing overall latency by avoiding unnecessary processing steps
3Adaptability or versatility
If text is extracted from multimodal content for translation, then translation capability is enabled, but scene information is lost
Solution Approach 1:
The patent introduces scene information as an intermediary element that bridges the extracted text and the translation process. Scene information acts as contextual metadata that preserves spatial, temporal, and semantic relationships from the original multimodal content, providing the translation model with additional context that prevents information loss while maintaining translation capability
Solution Approach 2:
The translation input is constructed as a composite structure combining extracted text with scene information metadata. This composite representation integrates multiple types of information (textual content, spatial relationships, temporal context, object relationships) into a unified input format that preserves the richness of the original multimodal content while enabling machine translation processing
Data Source
AI summary
A method for providing multimodal translation of a content in a source language is provided. The method includes receiving a user input with respect to a translation request of text included in the content, in response to receiving the user input, acquiring a multimodal input from the content, the multimodal input including location information related to the content other multimodal inputs, generating scene information representing the multimodal input related to the content by using a fusion layer based on the location information and the other multimodal inputs, identifying a candidate word set in a target language, determining at least one candidate word from the candidate word set based on the scene information, and translating the text included in the content into the target language using a translation model based on the determined at least one candidate word.