Keyword Extraction via Association and Divergence Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing keyword extraction methods are inaccurate and computationally expensive due to their reliance on statistical and graph-based approaches that prioritize frequent terms over semantic associations, and require large labeled corpora, limiting their applicability and efficiency.
Innovation Solution
A method and device for keyword extraction that assesses candidate phrases based on their association degree and divergence degree with the original document, selecting key phrases that balance similarity and difference to improve accuracy and diversity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If statistical-based or graph-based keyword extraction methods are used, then frequent terms are preferred as keywords, but the screened keyword may not always be in a semantic association relationship with the text and accuracy is reduced
Solution Approach 1:
The patent transforms the keyword extraction problem from frequency-based selection to similarity-based selection by changing the evaluation parameter from term frequency to cosine similarity between document vectors and keyword vectors. This allows the system to identify keywords that are semantically associated with the text rather than merely frequent, resolving the contradiction between extraction efficiency and keyword accuracy.
Solution Approach 2:
The patent replaces traditional statistical and graph-based mechanical extraction methods with a vector-space model approach. By representing documents and keywords as vectors in a high-dimensional space and using cosine similarity calculations, the system achieves both computational efficiency and semantic accuracy, overcoming the limitations of frequency-based methods.
2Measurement precision
If sequence-labeling-based or translation-alignment-based methods are used, then supervised learning with labeled corpora is applied, but calculation complexity is high and it is only applied to tasks in the field of corpus training
Solution Approach 1:
The patent uses pre-trained word vectors (such as Word2Vec or GloVe) that can be obtained from public resources, eliminating the need for expensive and time-consuming labeled corpus training. This allows the system to achieve high accuracy without the computational burden of supervised learning, making keyword extraction accessible for general applications beyond corpus training.
3Measurement precision
If graph-based keyword extraction method is used, then the method is applied to text analysis, but time complexity is usually greater than O(n2) and computational efficiency is low
Solution Approach 1:
The patent replaces the computationally intensive graph-based method with a vector-space model approach. By representing documents and keywords as vectors and using efficient cosine similarity calculations, the system achieves O(n) time complexity instead of O(n2), maintaining semantic association accuracy while dramatically improving computational efficiency.
Data Source
AI summary
A method and device for keyword extraction and a storage medium. The method includes receiving, at a terminal, an original document, acquiring, at the terminal, a candidate set by extracting at least one candidate phrase from the original document, acquiring, at the terminal, an association degree between the at least one candidate phrase in the candidate set and the original document, acquiring, at the terminal, a divergence degree of the at least one candidate phrase in the candidate set, and updating, at the terminal, a key phrase set of the original document by selecting the at least one candidate phrase from the candidate set as at least one key phrase based on the association degree and the divergence degree.