Multimodal Language Model for Accurate Text Recognition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text recognition methods in computer vision fields, such as OCR, separate detection and recognition processes, failing to consider linguistic context and grammar, leading to incomplete or inaccurate text recognition in images.
Innovation Solution
A method using a multimodal language model that detects text positions and characters in images, generates cropped images, and infers original sentences by inputting position and character data, applying different weights to modalities, and displays corrected sentences with grammar and spelling modifications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If OCR method is used to detect and recognize text separately, then text detection and recognition can be performed independently, but linguistic context and grammar are not reflected in the text recognition result
Solution Approach 1:
The patent combines text detection and recognition into a unified end-to-end neural network model that processes the entire image and outputs complete sentences with proper linguistic context and grammar, eliminating the need for separate detection and recognition stages while improving recognition accuracy
2Device complexity
If simple text information is used for recognition, then the process is simpler, but complete sentences with grammar and spelling accuracy cannot be generated
Solution Approach 1:
The patent transforms the output parameters of the recognition system from simple text strings to structured sentence representations that include grammatical structures, spelling corrections, and linguistic context, achieved by training the neural network with appropriate loss functions that enforce these linguistic constraints
Data Source
AI summary
A method and an electronic device for recognizing text are provided. The method includes detecting positions of pieces of text included in the text in the image, generating cropped images by cropping areas corresponding to the pieces of text in the image, recognizing characters of the pieces of text based on the cropped images, generating a sentence by inputting the positions of the pieces of text and the characters of the pieces of text to a multimodal language model, wherein the multimodal language model is an artificial intelligence (AI) model for inferring an original sentence of the text, and displaying the sentence.