Word Repetition Error Identification Using Contextual Entropy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current Chinese text automatic proofreading methods fail to accurately and efficiently identify word repetition errors, as they rely on binary or ternary information and do not account for repeated words that are common in daily language but not listed in dictionaries.
Innovation Solution
A method involving word segmentation, statistics on two-tuple and three-tuple structures, and contextual information entropy to identify repeated words, using a repeated word library and thresholds to judge correct or incorrect repetitions in text.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If simple binary or ternary information of words is used for judgment, then the judging process is simple and fast, but the accuracy of identifying word repetition errors is low and produces many misjudgments
Solution Approach 1:
The patent transitions from simple binary/ternary word information to multi-dimensional contextual information including left and right contextual adjacent word information entropy, repeated combination degrees, and n-tuple structures. This dimensional expansion enables accurate identification of repeated words while maintaining processing efficiency through structured statistical analysis.
2Ease of manufacture
If repeated words not in dictionaries are excluded from judgment, then dictionary-based checking is simple and fast, but common language expressions with repeated words are incorrectly identified as errors
Solution Approach 1:
The patent pre-calculates and stores statistical information including repeated combination degrees, contextual information entropy, and n-tuple structures from training corpora before actual error checking. This preliminary preparation enables fast and accurate judgment of repeated words during the checking phase without requiring complex real-time calculations, thus maintaining simplicity while improving reliability.
Solution Approach 2:
The patent introduces contextual information entropy and repeated combination degree as intermediary metrics to mediate between dictionary-based checking and repeated word identification. These intermediaries enable the system to distinguish between erroneous repetitions and acceptable repeated expressions in common language use, resolving the conflict between simplicity and accuracy.
3Measurement precision
If comprehensive statistical analysis of training corpus is performed, then the identification accuracy of repeated words is improved, but the processing time and computational complexity increase
Solution Approach 1:
The patent performs comprehensive statistical analysis including n-tuple structure extraction, repeated combination degree calculation, and contextual information entropy computation during the offline training phase. These pre-computed statistical results are stored and reused during online error checking, achieving high identification accuracy without incurring computational overhead during the actual checking process.
Solution Approach 2:
The patent extracts key statistical features (repeated combination degrees, contextual information entropy, n-tuple structures) from the training corpus and separates them from the main checking process. This extraction allows the system to use only essential pre-computed features during error checking, reducing processing time while maintaining high accuracy.
Data Source
AI summary
A method for automatically identifying word repetition errors includes the following steps: after performing word segmentation on a large-scale training corpus, performing statistics to obtain two-tuple and three-tuple structures including repeated words in the training corpus, and repeated combination degrees, left contextual adjacent word information entropy and right contextual adjacent word information entropy in the training corpus; performing statistics and recording words containing repeated characters in a Chinese dictionary and establishing a repeated word library of the Chinese dictionary; judging the repeated words appearing in the text to be subjected to error checking based on the repeated words in the Chinese dictionary; and judging the repeated words appearing in the text to be subjected to error checking based on the repeated combination degrees, left contextual adjacent word information entropy and right contextual adjacent word information entropy obtained by performing statistics.