Automatic Charset Detection Using SVM Feature Vectors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current technologies face challenges in accurately detecting the encoding scheme of received data, leading to improper display and analysis, especially when encoding information is missing, affecting global information exchange and email content filtering.
Innovation Solution
A computer-implemented method using Support Vector Machines (SVM) for automatic charset detection, which involves training with text document samples to generate machine learning models and applying them to target documents to compute similarity indicia, determining the most suitable encoding scheme.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If automatic charset detection is implemented, then encoding scheme identification accuracy is improved, but detection precision deteriorates due to missing or incorrect encoding information
Solution Approach 1:
The system performs preliminary training with multiple text document samples encoded with different encoding schemes to generate machine learning models before actual detection. This preliminary action creates a knowledge base that improves detection reliability when encoding information is missing or incorrect.
Solution Approach 2:
The patent introduces similarity indicia as an intermediary metric to compare target document features with training document features. This intermediary allows the system to indirectly determine encoding schemes when direct encoding information is unavailable, resolving the precision-reliability contradiction.
2Reliability
If machine learning models are trained with multiple text document samples, then detection reliability is improved, but computational complexity increases
Solution Approach 1:
The patent segments the charset detection problem into distinct phases: training phase with multiple text samples to generate machine learning models, and detection phase applying these models to target documents. This segmentation allows reliable detection while managing computational complexity by performing intensive training only once.
Solution Approach 2:
The system creates machine learning models as copies of knowledge extracted from multiple training text documents. These model copies can then be applied repeatedly to detect encoding schemes of target documents without requiring the original training data or repeating the complex training process.
Data Source
AI summary
The invention relates, in an embodiment, to a method for handling a received document. The method includes receiving a plurality of text document samples. The method includes training, using a plurality of text document samples, to obtain a set of machine learning models. Training includes generating fundamental units from the plurality of text document samples for charsets of the plurality of text document samples. Training includes extracting a subset of said fundamental units as feature lists and converting the feature lists into a set of feature vectors. Training further includes generating the set of machine learning models from the set of feature vectors. The method includes applying the set of machine learning models against a set of target document feature vectors converted from the received document. The method includes decoding the received document to obtain decoded content of the received document based on at least the first encoding scheme.