Keyword Extraction Using Linguistic Feature Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing keyword extraction systems rely heavily on statistical frequency measures like tf-idf, which are not sensitive to phrase structure and linguistic features, leading to suboptimal performance in identifying relevant keywords for contextual advertising.
Innovation Solution
A keyword extraction system that employs natural language processing techniques, including preprocessing, candidate extraction, and classification units, to analyze linguistic features such as part of speech, phrase structure, and named entity information, improving the identification of primary and secondary keywords.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If statistical frequency measures like tf-idf are used for keyword extraction, then the system is simple to implement, but the performance in identifying relevant keywords deteriorates due to insensitivity to phrase structure and linguistic features
Solution Approach 1:
The system segments the keyword extraction process into distinct stages: preprocessing to generate initial candidates, candidate extraction to refine the pool using linguistic features, and classification to rank candidates. This segmentation allows each stage to specialize in specific tasks, improving overall accuracy while maintaining manageable complexity
Solution Approach 2:
The patent introduces an intermediary candidate pool between the initial text processing and final keyword selection. This intermediate stage allows for refinement using linguistic features like part-of-speech tagging and phrase structure analysis, acting as a mediator that bridges simple frequency counting and complex linguistic understanding
2Measurement precision
If natural language processing techniques are employed to analyze linguistic features, then keyword extraction accuracy improves, but system complexity increases
Solution Approach 1:
The system divides the complex NLP task into manageable segments: preprocessing for basic text preparation, candidate extraction for generating potential keywords using linguistic features, and classification for ranking. This segmentation reduces overall system complexity by breaking down the monolithic NLP process into specialized, modular components
Solution Approach 2:
Different linguistic features are applied locally to different aspects of keyword extraction: part-of-speech tagging is applied to identify noun phrases, phrase structure analysis is applied to capture contextual relationships, and named entity recognition is applied to identify proper nouns. Each local application targets specific quality requirements for different types of keywords
3Measurement precision
If linguistic features such as part of speech and phrase structure are analyzed, then recall and precision increase, but processing time and computational resources increase
Solution Approach 1:
The system performs preliminary actions by pre-processing the text to identify and tag linguistic features such as part-of-speech, phrase structure, and named entities before the actual keyword extraction and classification. This preliminary tagging allows the classification stage to efficiently evaluate candidates using pre-computed linguistic features, reducing processing time during the critical keyword selection phase
Solution Approach 2:
The system applies linguistic analysis partially rather than comprehensively to all text. It focuses NLP resources on generating and evaluating keyword candidates rather than analyzing every aspect of the document, applying linguistic features only where they most directly contribute to keyword identification accuracy
Data Source
AI summary
One exemplary aspect comprises a computer system comprising: (a) a preprocessing unit that extracts text from a webpage to produce at least a first set of candidate keywords, applies language processing to produce at least a second set of candidate keywords, and combines said first and second sets of candidate keywords into a first candidate pool; (b) a candidate extraction unit that receives data from said preprocessing unit describing at least said first candidate pool and produces a second candidate pool; (c) a feature extraction unit that receives data describing at least said second candidate pool and analyzes said second candidate pool for general features and linguistic features; and (d) a classification unit that receives said data describing at least said second candidate pool and related data from said feature extraction unit, and determines a likelihood of each candidate in said second candidate pool being a primary or secondary keyword.