Multimodal Character Recognition to Reduce OCR Over-Correction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing OCR technologies suffer from over-correction issues due to reliance on language models that modify correct recognition results to incorrect ones, leading to inaccurate text recognition.
Innovation Solution
A data processing method that combines visual and audio features to enhance character recognition, reintroducing the original visual feature to correct the language model's output, thereby reducing over-correction by leveraging the strengths of both modalities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If language models are used to correct character information identified by visual models, then recognition accuracy is improved, but over-correction occurs where correct results are modified to incorrect ones
Solution Approach 1:
The patent implements a feedback mechanism where the visual model's original recognition results are fed back into the final output alongside the language model's corrections. This allows the system to compare and selectively combine results, preventing the language model from unilaterally over-correcting accurate visual recognitions while still benefiting from its semantic correction capabilities on genuinely erroneous outputs.
Solution Approach 2:
The patent merges the output of the visual model (first modal feature) with the corrected output of the language model (second modal feature) to produce the final recognition result. This combination strategy ensures that both the visual model's direct pattern recognition strength and the language model's semantic understanding strength are utilized, while the visual model's original result serves as a safeguard against over-correction.
2Measurement precision
If only corrected second modal feature is used for recognition, then semantic accuracy is improved, but visual feature information is lost leading to over-correction
Solution Approach 1:
The patent applies local quality by treating different parts of the recognition pipeline differently - the visual model provides global structural recognition while the language model provides local semantic refinement. The final system selectively applies corrections only where needed, preserving visual feature information for cases where the visual model's direct recognition is accurate, while incorporating language model corrections only for cases where semantic context indicates errors.
Data Source
AI summary
A data processing method includes obtaining input data, where the input image is image data or audio data; obtaining a second modal feature based on a first modal feature of the input data, where the first modal feature is a visual feature of the image data or an audio feature of the audio data, and the second modal feature is a character feature; and fusing the first modal feature and the second modal feature to obtain a target feature.