Image Text Translation Model Without Separate OCR
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems for translating text in images require multiple models, such as OCR and machine translation, which introduce errors and resource inefficiencies, and fail to maintain contextual accuracy due to line-by-line processing.
Innovation Solution
An end-to-end image translation model using a feature extractor and decoder, trained on image features and text-to-text translation, directly translates text from a source language to a target language, accommodating errors and preserving contextual accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple models (OCR and machine translation) are used for translation, then text extraction and translation can be performed, but resource requirements and system complexity increase
Solution Approach 1:
The patent merges the OCR model and machine translation model into a single unified model that performs both text extraction and translation functions. This unified model takes an input image and directly outputs translated text, eliminating the need for separate models and reducing system complexity while maintaining translation accuracy.
Solution Approach 2:
The unified model serves multiple functions: it performs optical character recognition to extract text from images and simultaneously performs machine translation to translate the extracted text into the target language. This multi-functional approach reduces the number of components needed in the system.
2Reliability
If multiple models are used for translation, then text processing can be performed, but resource consumption increases
Solution Approach 1:
By combining the OCR and translation functionalities into a single model, the system reduces the computational overhead associated with running multiple separate models. The unified model processes the input image in one pass, extracting and translating text simultaneously, thereby reducing energy consumption and computational resource usage.
3Productivity
If line-by-line processing is used, then text can be translated, but contextual accuracy is lost
Solution Approach 1:
The patent moves from one-dimensional line-by-line processing to two-dimensional image-based processing. By treating the entire image as the input unit rather than processing individual lines sequentially, the model preserves spatial relationships and contextual information across the entire text region, thereby maintaining contextual accuracy while still achieving efficient processing.
4Ease of manufacture
If OCR is used as a separate step, then text extraction can be performed, but errors are introduced
Solution Approach 1:
The patent integrates the text extraction function directly into the translation model, eliminating the separate OCR step. The unified model extracts and translates text in a single end-to-end process, preventing error propagation between separate processing stages and improving overall translation accuracy.
Solution Approach 2:
The model performs text extraction as a preliminary step within its internal processing, but integrates it seamlessly with the translation function. This allows the model to extract text features and translate them simultaneously, preventing the introduction of errors that would occur with separate processing steps.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, that translate text depicted in images from a source language into a target language. Methods can include obtaining a first image that depicts first text written in a source language. The first image is input into an image translation model, which includes a feature extractor and a decoder. The feature extractor accepts the first image as input and in response, generates a first set of image features that are a description of a portion of the first image in which the text is depicted is obtained. The first set of image features are input into a decoder. In response to the input first set of image features, the decoder outputs a second text that is a predicted translation of text in the source language that is represented by the first set of image features.