Key-term Extraction Using Frequency-Based Scoring
Solution Overview
Problem
Current information retrieval techniques for extracting key-terms and synonyms from documents are computationally intensive, require large training datasets, and struggle with generalizing across various datasets, especially in identifying relevant phrases and their synonyms.
Innovation Solution
A method and system that extracts key-terms and their synonyms using frequency-based techniques independent of supervised training, involving parsing, contraction and acronym resolution, pre-processing, and domain language models for relevancy scoring, without requiring extensive training data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If supervised learning methods with machine learning or deep learning algorithms are used for key-term extraction, then the extraction accuracy can be improved, but the computational complexity and training data requirements increase significantly
Solution Approach 1:
The patent segments the key-term extraction process into multiple independent stages: text parsing, pre-processing (tokenization, stop-word removal, lemmatization), candidate key-term identification, and synonym extraction. Each stage processes specific aspects independently, avoiding the need for complex end-to-end supervised learning while maintaining extraction accuracy through systematic multi-stage processing.
Solution Approach 2:
The patent introduces an intermediary frequency-based scoring mechanism that bridges raw text processing and final key-term selection. This mediator computes relevancy scores based on term frequency and document frequency without requiring trained models, enabling accurate key-term extraction through intermediate computational steps rather than direct supervised learning.
2Measurement precision
If supervised learning methods are used for key-term extraction, then the extraction performance can be improved, but the requirement for huge training data increases
Solution Approach 1:
The patent enables the system to extract key-terms and synonyms autonomously using frequency-based algorithms and linguistic rules without requiring external training data. The system serves itself by computing relevancy scores from the input text statistics and applying pre-defined linguistic patterns for synonym generation, eliminating dependency on large training datasets.
Solution Approach 2:
The patent changes the fundamental parameters of the extraction approach from model-based probabilities to frequency-based statistics. By computing term importance based on occurrence frequency, document frequency, and inverse document frequency metrics, the system achieves extraction performance without requiring training data, fundamentally altering how relevancy is measured.
3Device complexity
If unsupervised learning methods based on statistical or linguistic approaches are used for key-term extraction, then the computational requirements can be reduced, but the generalization capability across different datasets deteriorates
Solution Approach 1:
The patent creates a universal extraction framework that processes diverse text types (scientific papers, news articles, technical documentation) using the same frequency-based algorithms and linguistic rules. The system achieves broad generalization capability by relying on universal linguistic patterns and statistical properties that apply across different domains and text genres without dataset-specific tuning.
Solution Approach 2:
The patent performs preliminary text pre-processing (tokenization, stop-word removal, lemmatization, parsing) before key-term extraction to standardize the input data. This preliminary action ensures that the frequency-based algorithms receive consistent, normalized text regardless of the source dataset, improving generalization capability across different types of input documents.
4Productivity
If traditional key-term extraction techniques are used, then the extraction speed can be maintained, but the ability to identify relevant synonyms deteriorates
Solution Approach 1:
The patent merges key-term extraction with synonym identification into a unified process. After extracting key-terms using frequency-based methods, the system immediately processes each key-term through synonym lookup using language models and domain-specific dictionaries, combining both functions in one efficient pass rather than requiring separate processing steps.
Solution Approach 2:
The patent introduces an intermediary synonym lookup mechanism that operates on extracted key-terms. This mediator queries language models and domain dictionaries to identify synonyms while maintaining the efficiency of the original extraction process, adding synonym capability without significantly increasing computational overhead or extraction time.
Data Source
AI summary
This disclosure relates generally to a method and a system for extraction of key-terms and synonyms for the key-terms. The method discloses to extract the key-terms and further determine a set of relevant synonyms for the extracted key-terms, wherein the key-terms is a collection of small sub-set of words and phrases which are individually meaningful and collectively provide a generic context of a given input text. The method discloses techniques for extraction of key-terms that is independent of any specific supervised training, does not require a huge amount of training data and computationally less intensive. The key-terms are determined in several stages using frequency based techniques based on a relevancy scoring of the key-terms. Further a set of synonyms are determined for the identified key-terms based on a language-based approach or domain specific approach.