Keyword Extraction via Multi-Level Segmentation and Co-Occurrence
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing keyword extraction methods face challenges in accurately extracting keywords from text due to low word segmentation accuracy and difficulty in identifying key phrases, especially in business data with many business terms and short key phrases, leading to low extraction accuracy.
Innovation Solution
A keyword extraction method that involves word segmentation, entity recognition, part-of-speech tagging, scoring, filtering, and word co-occurrence statistics to improve the accuracy of keyword extraction by filtering noise words and identifying key phrases.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If word segmentation is performed using a general corpus-based word segmenter, then the segmentation process is simple and fast, but the word segmentation accuracy is low for business language corpus containing many business terms
Solution Approach 1:
The patent changes the training data parameter from general corpus to business language corpus, and adjusts the segmentation granularity parameter to enable both word-level and phrase-level segmentation. This allows the system to achieve high accuracy for business terms while maintaining reasonable system complexity through configurable parameters.
Solution Approach 2:
The patent applies segmentation at multiple levels: first segmenting text into words using a trained word segmenter, then further segmenting into phrases through phrase extraction. This multi-level segmentation approach resolves the contradiction by achieving high word segmentation accuracy while keeping the overall system manageable through staged processing.
2Measurement precision
If keyword extraction algorithms focus on word granularity, then the extraction process is straightforward, but it is difficult to identify key phrases consisting of two or three words
Solution Approach 1:
The patent introduces phrase segmentation as an additional layer after word segmentation. By segmenting the text first into words and then into phrases, the system can identify key phrases consisting of two or three words while maintaining a structured, manageable extraction process that doesn't require complete algorithmic redesign.
Solution Approach 2:
The patent adds the phrase level as another dimension of analysis beyond word granularity. This dimensional expansion allows the system to capture multi-word key phrases while maintaining the simplicity of word-level processing as a foundation, thus improving extraction accuracy without proportionally increasing complexity.
3Measurement precision
If unsupervised training based on statistical methods is used, then the model can be trained without labeled data, but the keyword extraction accuracy is low due to lack of text semantics utilization
Solution Approach 1:
The patent performs preliminary actions of word segmentation and phrase extraction before final keyword selection. By pre-processing the text to identify candidate words and phrases with statistical significance, the system reduces the need for extensive labeled data while improving extraction accuracy through semantic-aware candidate generation.
Solution Approach 2:
The patent introduces phrase extraction as an intermediary step between raw text and final keyword extraction. This intermediary process utilizes statistical methods to identify meaningful phrases, bridging the gap between unsupervised training limitations and the need for semantic-aware keyword extraction, thereby improving accuracy without requiring large labeled datasets.
Data Source
AI summary
The application relates to the technical field of information extraction, in particular to a keyword extraction method. The method comprises: acquiring a text to be processed, and performing word segmentation on the text to be processed to obtain at least one word segmentation result; performing entity recognition on all the word segmentation results through a preset entity recognition model to obtain at least one entity recognition result; performing part-of-speech tagging on all the entity recognition results to obtain part-of-speech tagging results; scoring all the part-of-speech tagging results through a preset scoring metric to obtain score values; filtering all the part-of-speech tagging results based on all the score values to obtain at least one target word; running word co-occurrence statistics on all the target words to obtain word co-occurrence values, and performing keyword extraction on all the target words based on the word co-occurrence values to obtain keyword extraction results.