Phrase Chunking Using POS Tagging and Grammar Rules
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional Latent Semantic Analysis (LSA) methods for identifying phrases in text are inefficient, producing numerous unwanted phrase candidates and fragments, which slows down processing and prevents recognition of meaningful phrases with more than four words.
Innovation Solution
A method that uses a lightweight parts-of-speech (POS) tagger with a lexicon and heuristics to accurately tag words, followed by fast phrase chunking using a stack of grammar productions, to identify meaningful phrases while reducing unnecessary candidates and fragments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional LSA methods are used to identify phrases in text, then comprehensive phrase coverage is achieved, but processing speed decreases and system efficiency worsens
Solution Approach 1:
The patent segments the phrase identification process into distinct stages: tokenization, POS tagging, and phrase chunking. Each stage processes specific linguistic features independently, allowing parallel processing and optimizing overall system performance while maintaining comprehensive phrase coverage.
Solution Approach 2:
The system performs preliminary POS tagging on all tokens before phrase identification. This preliminary action creates structured information about word categories that accelerates subsequent phrase chunking operations, enabling faster processing without sacrificing recognition accuracy.
2Measurement precision
If traditional n-gram methods are used to generate phrase candidates, then all possible phrases are captured, but the number of unwanted candidates and fragments increases
Solution Approach 1:
The patent applies different processing rules to different linguistic structures. Noun phrases, verb phrases, and adjective phrases are identified using POS tag-specific chunking rules that match their grammatical characteristics. This local quality approach ensures complete phrase capture while eliminating fragments that don't conform to proper grammatical structures.
Solution Approach 2:
The system changes the parameter of phrase identification from fixed n-gram windows to dynamic grammar-based chunking. By using POS tags as parameters, the system adapts phrase boundaries to linguistic structure, reducing unwanted candidates while maintaining completeness of meaningful phrases.
3Measurement precision
If phrase identification processes are slowed down to recognize longer phrases, then phrase recognition accuracy improves, but overall system throughput decreases
Solution Approach 1:
The patent implements continuous processing through a streaming architecture where tokens are tagged and chunked in sequence without interruption. The phrase identification process continues uninterrupted, maintaining high throughput while accurately identifying phrases of any length by processing the entire token stream systematically.
Solution Approach 2:
The system replaces mechanical n-gram window sliding with grammar-based chunking algorithms that directly identify phrase boundaries using POS tag patterns. This substitution eliminates the need to process every possible n-gram combination, reducing processing time while maintaining accurate identification of phrases regardless of length.
Data Source
AI summary
A method includes accessing text that includes a plurality of words, tagging each of the plurality of words with one of a plurality of parts of speech (POS) tags, and creating a plurality of tokens, each token comprising one of the plurality of words and its associated POS tag. The method further includes clustering one or more of the created tokens into a chunk of tokens, the one or more tokens clustered into the chunk of tokens based on the POS tags of the one or more tokens, and forming a phrase based on the chunk of tokens, the phrase comprising the words of the one or more tokens clustered into the chunk of tokens.