Character Error Correction via Reasonability Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current character error correction methods in natural language processing are limited by the requirement that candidates must be similar in pronunciation or pattern to the original character, which reduces the accuracy of error correction results.
Innovation Solution
A method that involves acquiring a reasonability score for each character in a pre-constructed vocabulary, selecting top K candidates based on these scores, and replacing the character with the optimal candidate if different, using a pre-trained language model for scoring and fine-tuning to improve diversity and accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the limitation that candidates must be similar to the original character in pronunciation or pattern is added, then the error correction process becomes more constrained and controlled, but the quality of candidates and the accuracy of error correction results deteriorate
Solution Approach 1:
The patent changes the selection parameters from traditional phonetic/pattern similarity metrics to a comprehensive reasonability score that evaluates multiple dimensions including semantic compatibility, contextual appropriateness, and linguistic plausibility. This parameter transformation enables the system to break free from the constraints of traditional similarity-based selection while maintaining rigorous error correction standards, thereby improving both candidate diversity and correction accuracy simultaneously
2Adaptability or versatility
If top K characters are selected from the whole vocabulary based on reasonability scores, then candidate diversity improves, but the complexity of the selection process increases
Solution Approach 1:
The patent performs preliminary actions by pre-computing and storing reasonability scores for all character-vocabulary pairs before the actual error correction process. This preprocessing step creates a ready-to-use scoring matrix that can be quickly queried during correction, eliminating the need for complex real-time calculations and reducing the computational burden of selecting top K candidates from the entire vocabulary
3Productivity
If traditional similarity constraints are applied for candidate selection, then the selection process is simpler and faster, but the quality of correction results deteriorates
Solution Approach 1:
The patent applies partial action by selectively applying reasonability score evaluation only to positions in the text where errors are detected or suspected, rather than processing every character uniformly. This approach maintains high processing speed by focusing computational resources on problematic areas while still achieving improved accuracy through the more sophisticated scoring mechanism
Data Source
AI summary
A method and apparatus for correcting character errors, an electronic device and a storage medium are disclosed, which relates to the natural language processing field and the deep learning field. The method may include: for a character to be processed, acquiring the score of each character in a pre-constructed vocabulary, the score being a score of the reasonability of the character in the vocabulary at the position of the character to be processed; selecting top K characters as candidates of the character to be processed, K being a positive integer greater than one; selecting an optimal candidate from the K candidates; and replacing the character to be processed with the optimal candidate if the optimal candidate is different from the character to be processed. With the solution of the present application, the accuracy of an error correction result, or the like, may be improved.