Unsupervised Keyword Extraction via Scoring
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing keyword extraction methods are inefficient and resource-intensive, requiring supervised learning, corpus-level analysis, and often fail to provide satisfactory results, especially when dealing with large volumes of documents and varying content types.
Innovation Solution
An unsupervised keyword extraction mechanism that operates on a single document, using a keyword extraction module to calculate keyword scores based on word frequencies and spans, allowing for flexible keyword lengths and domain independence, without the need for corpus-level statistics or human annotation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If supervised machine learning through annotating documents is employed to extract keywords, then keyword extraction accuracy is improved, but computational resources and time consumption increase significantly
Solution Approach 1:
The system performs self-service by automatically computing keyword scores using unsupervised statistical methods on document corpora, eliminating the need for manual annotation while maintaining extraction quality through algorithms that identify significant terms based on frequency and distribution patterns
Solution Approach 2:
The patent extracts only the essential statistical features (term frequency, document frequency, inverse document frequency) needed for keyword scoring, separating these critical measurements from the complete document content to achieve efficient processing without sacrificing extraction accuracy
2Measurement precision
If corpus level statistics are utilized to facilitate keyword extraction, then extraction quality is improved, but computational complexity and resource requirements increase
Solution Approach 1:
The system applies local quality by computing statistics at the document level rather than requiring comprehensive corpus-level analysis, allowing keyword extraction to be performed on individual documents or small subsets with reduced computational overhead while maintaining relevant quality metrics
Solution Approach 2:
The patent implements partial action by calculating only the necessary statistical measures (TF-IDF, term frequency) required for effective keyword scoring, omitting more complex corpus-wide analyses that would provide diminishing returns while significantly reducing computational complexity
3Quantity of substance
If existing keyword extraction approaches are applied to large volumes of documents, then comprehensive keyword coverage is achieved, but processing time and resource consumption become prohibitive
Solution Approach 1:
The system segments the keyword extraction process into independent document-level operations that can be parallelized, allowing large volumes of documents to be processed simultaneously through distributed computing frameworks, thereby maintaining comprehensive keyword coverage while dramatically reducing total processing time
Solution Approach 2:
The patent changes parameters by using efficient statistical approximations and adjustable scoring thresholds that allow the system to process documents at higher throughput while maintaining adequate keyword coverage, enabling scalability to large document volumes without linearly increasing processing time
Data Source
AI summary
An unsupervised keyword extraction process is disclosed. A single input document can be analyzed to identify multiple candidate keywords by utilizing splitting terms. A keyword score is calculated for each of the candidate keywords. The keyword score for a particular candidate keyword is determined based on the length of the candidate keywords that contain the candidate keyword and the frequency of the words appearing in the candidate keywords. One or more keywords having the highest keyword scores are selected as the extracted keywords. The extracted keywords can be used in applications, such as refining search results, providing suggested search terms, or improving the match rate of a network page at a search engine.