Informative Phrase Extraction from Unstructured Text
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Manually tracking topics and discussions from large volumes of unstructured text on the web is laborious and inefficient, as existing methods fail to quickly extract meaningful and informative phrases that characterize a selected subject or document set.
Innovation Solution
A computer-implemented method that builds an index of phrases by parsing parts of speech, identifying and filtering out uninformative phrases, and scoring informative phrases based on their occurrence in a foreground and background set of documents, using a combination of POS tagging, phrase chunking, and probability calculations to extract highly informative phrases.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual tracking of topics and discussions is performed, then accuracy and depth of analysis is improved, but time consumption and labor requirements increase significantly
Solution Approach 1:
The patent replaces manual mechanical reading and analysis with an automated computer system that uses natural language processing techniques. The system automatically extracts phrases, identifies topics, and analyzes discussions from large volumes of unstructured text without human intervention, thereby maintaining accuracy while eliminating time consumption associated with manual processing
Solution Approach 2:
The system performs self-service by automatically processing, analyzing, and extracting informative phrases from documents. The automated phrase extraction and topic identification processes eliminate the need for manual analysis, allowing the system to independently complete the entire workflow of tracking topics and discussions across large document corpora
2Quantity of substance
If all phrases are extracted from documents, then completeness of data is improved, but quality and meaningfulness of extracted phrases deteriorate due to inclusion of uninformative phrases
Solution Approach 1:
The patent applies local quality by differentiating between types of phrases and applying selective extraction criteria. The system extracts only informative phrases (nouns, verbs, adjectives, adverbs) while excluding uninformative phrases (proper nouns, prepositions, conjunctions, articles). This selective approach maintains completeness of relevant data while ensuring high quality by filtering out meaningless phrases
Solution Approach 2:
The system changes the parameter of phrase selection by using part-of-speech tagging and informativeness scoring. Phrases are extracted based on their grammatical category and calculated informativeness score, which compares phrase frequency in the document corpus against expected frequencies. This parameter-based filtering ensures only high-quality informative phrases are extracted while maintaining comprehensive coverage
3Quantity of substance
If phrases from lists, menus, and duplicated text are included, then completeness of phrase collection is improved, but reliability of phrase relevance deteriorates
Solution Approach 1:
The patent applies the extraction principle by removing unwanted elements from the phrase collection. The system specifically extracts and excludes phrases from lists, menus, and duplicated text sections. This selective extraction maintains completeness of relevant phrases while improving reliability by eliminating phrases that are clearly not relevant to the document's main content
Solution Approach 2:
The system uses feedback mechanisms through informativeness scoring and relevance analysis. Each extracted phrase is evaluated against the document context and compared against the overall corpus to determine its relevance. Phrases that fail to meet the relevance threshold are rejected, ensuring high reliability of the final phrase collection while maintaining completeness through continuous evaluation
Data Source
AI summary
Disclosed is a method of extracting informative phrases from a full corpus of documents. An index of phrases contained in the full corpus of documents is built. Then, a user specifies a subset of text to analyze. The subset may be defined as: (1) all paragraphs or sentences containing terms selected as defining a subject; (2) all documents in a category; (3) all documents written within a date range; and/or (3) all documents matching a Boolean query of terms. Once the subset is specified, it is analyzed to extract informative phrases. Specifically, the index is queried to retrieve all phrases within the subset. The number of times each of the phases occurs in the subset and in the corpus is counted. Each phrase contained in the subset is scored according to informativeness based on a comparison of a likelihood that the phrase occurs in the subset and a likelihood that the phrase occurs in the corpus as a whole. Only those phrases having an informativeness score above a predetermined value are considered highly informative and extracted.