Document Classification Model Using Character Features for Typo Resilience
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Document classification accuracy decreases when typographical errors or omitted characters are present in input documents, as existing techniques fail to effectively handle spelling inconsistencies and variations, leading to reduced performance in machine learning models.
Innovation Solution
A document classification device and trained model that extracts words and character information from documents as features, using machine learning to generate a classification model that considers character features, thereby improving accuracy even with typographical errors or omitted characters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a dictionary for coping with spelling inconsistency is prepared in advance, then spelling inconsistencies can be handled, but costs are required for preparing the dictionary
Solution Approach 1:
The system performs self-service by automatically learning spelling variations and typos from the training data itself, without requiring external dictionary preparation. The machine learning model inherently captures spelling patterns during the training process, eliminating the need for manual dictionary creation and maintenance.
Solution Approach 2:
The system performs preliminary action by pre-training the machine learning model on diverse spelling variations and typos that are naturally present in the training corpus. This preliminary exposure to various spellings during training enables the model to handle spelling inconsistencies without requiring a pre-prepared dictionary.
2Reliability
If a dictionary for spelling inconsistency is used, then spelling variations can be handled, but time is required for dictionary preparation and maintenance
Solution Approach 1:
The system eliminates time-consuming dictionary preparation by using self-service machine learning that automatically adapts to spelling variations present in the training data. The model learns spelling patterns during training without requiring external intervention for dictionary creation or updates.
Solution Approach 2:
The patent replaces the mechanical system of manual dictionary preparation and maintenance with an automated machine learning system. The ML model automatically learns and adapts to spelling variations through training, substituting the manual process with an automated computational process that requires no ongoing maintenance.
3Ease of operation
If only words are extracted as features without character information, then processing is simpler, but classification accuracy decreases when typographical errors are present
Solution Approach 1:
The system applies segmentation by breaking down words into their constituent characters and using both word-level and character-level features. This segmentation allows the model to capture spelling patterns at the character level while maintaining the overall word structure, improving robustness to typos without significantly increasing complexity.
Solution Approach 2:
The patent uses a composite feature representation that combines word-level features with character-level features. This composite approach integrates multiple levels of information (whole words and their constituent characters) to create a more robust feature set that maintains simplicity while improving accuracy for handling spelling variations.
Data Source
AI summary
A document classification device is a device that generates a document classification model which outputs identification information for identifying a result of classification on the basis of an input document by machine learning, and includes: an acquisition unit configured to acquire learning data including a document and the identification information correlated with the document; a feature extracting unit configured to extract words included in the document and character information which is a character string including one character of characters constituting the words or a plurality of characters consecutive in the words and which is one or more pieces of information capable of being extracted from the words as features; and a model generating unit configured to perform machine learning on the basis of the feature extracted from the document and the identification information correlated with the document and to generate the document classification model.