Dynamic Word Segmentation List for Text Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text information classification methods suffer from reduced precision due to the inability to recognize new words, as they rely on fixed dictionaries and fail to incorporate emerging vocabulary.
Innovation Solution
A method and apparatus that perform word segmentation on target text, identify new words by comparing segmentation results with a preset list, add these new words to the list, and recalibrate classification accuracy, thereby enhancing text classification precision by dynamically updating the segmentation list.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a fixed dictionary is used for word segmentation, then the classification process is simple and fast, but new words cannot be recognized leading to reduced classification precision
Solution Approach 1:
The patent implements a dynamic word segmentation list that can be automatically updated with newly discovered words. Instead of using a static fixed dictionary, the system continuously learns from text data, identifies new words through segmentation analysis, and incorporates them into the segmentation list. This dynamic update mechanism ensures the system adapts to emerging vocabulary while maintaining classification precision.
Solution Approach 2:
The system performs self-updating by automatically identifying new words from segmented text data without requiring manual dictionary updates. The classification module itself generates the segmentation list through continuous analysis of text patterns and word frequencies, making the system self-improving and adaptive to new terminology.
2Adaptability or versatility
If the word segmentation list is updated dynamically, then new words can be recognized improving classification precision, but the system complexity and processing time increase
Solution Approach 1:
The patent performs word segmentation and new word identification in advance during the text processing phase. By pre-segmenting text and identifying potential new words before final classification, the system prepares the segmentation list proactively. This preliminary action reduces the time penalty of dynamic updates during actual classification operations.
Solution Approach 2:
The system maintains continuous text processing and segmentation operations, allowing new words to be identified and incorporated into the segmentation list without interrupting the classification workflow. The continuous operation ensures that vocabulary adaptation happens seamlessly in the background, minimizing time loss while maintaining high adaptability.
Data Source
AI summary
Method, apparatus, and storage medium for text information processing are provided. The method includes: performing word segmentation on a target text according to a preset fixed word segmentation policy, and comparing a word segmentation result with a preset word segmentation list, to obtain a new word; adding the new word to the preset word segmentation list, to obtain a test word segmentation list; classifying a test text according to the preset word segmentation list, to obtain a first text, and classifying the test text according to the test word segmentation list, to obtain a second text; comparing classification accuracy of the first text with classification accuracy of the second text, and determining a target new word from the new word according to a comparison result; and adding the target new word to the preset word segmentation list, and classifying the target text.