Keyword Identification in Textual Data Using Word Combinations
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods fail to efficiently identify and define key words within large groups of documents, especially when new documents contain undefined terms, making it challenging to analyze and manage textual data effectively.
Innovation Solution
A computer-implemented method that identifies a first group of textual data, determines word combinations, and identifies key words by utilizing these combinations, employing techniques like latent Dirichlet allocation (LDA) and Bayesian analysis to weight and rank words, allowing for comparison against defined key words in other documents to identify undefined but important terms.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional keyword identification methods are used on large groups of documents, then the process becomes computationally expensive and time-consuming, but the accuracy of identifying undefined key words remains insufficient
Solution Approach 1:
The patent segments the large group of documents into multiple smaller groups or batches for parallel processing. Each batch is analyzed independently to identify key words, which are then aggregated. This segmentation reduces the computational burden on any single processing unit and enables distributed computing approaches, thereby reducing analysis time while maintaining identification accuracy through comprehensive coverage of all documents.
Solution Approach 2:
The patent performs preliminary actions by pre-processing documents to extract potential key word candidates before the main analysis phase. This includes removing stop words, stemming, and initial frequency counting. By preparing data in advance, the system reduces the complexity of the subsequent key word identification phase, allowing for faster processing while improving accuracy through focused analysis on relevant terms only.
2Reliability
If comprehensive analysis of all documents is performed to identify key words, then identification accuracy improves, but computational resources and processing time increase significantly
Solution Approach 1:
The patent applies local quality by tailoring the analysis depth and methodology to specific document characteristics. Different document types, lengths, or domains receive customized processing parameters. For example, technical documents may undergo more rigorous analysis with domain-specific dictionaries, while general documents use standard processing. This approach ensures high reliability for each document type without uniformly increasing system complexity across all processing paths.
Solution Approach 2:
The patent dynamically adjusts analysis parameters such as minimum word frequency thresholds, n-gram window sizes, and weighting factors based on document properties and analysis stage. These parameter changes optimize the balance between identification reliability and processing efficiency for different scenarios, avoiding the need for a uniformly complex system configuration while maintaining high reliability where needed.
3Measurement precision
If manual review and definition of key words is performed, then accuracy of key word identification improves, but productivity and processing speed decrease
Solution Approach 1:
The patent implements feedback mechanisms where the system's automatic key word identification results are evaluated and used to refine future processing. Statistical patterns from analyzed documents feed back into updating word frequency thresholds, creating domain-specific dictionaries, and adjusting weighting algorithms. This continuous improvement loop enables the system to achieve high precision automatically over time, eliminating the need for manual review while maintaining or improving productivity through automated iterative refinement.
Solution Approach 2:
The patent enables the system to self-service by automatically learning and adapting to document characteristics without human intervention. The system autonomously identifies patterns, defines domain-specific terminology, and refines its analysis methodology based on accumulated data. This self-improving capability allows the system to achieve manual-level precision through automated learning processes, dramatically increasing productivity while maintaining high key word identification accuracy.
Data Source
AI summary
According to one embodiment, a computer program product for identifying key words within a plurality of documents comprises a computer readable storage medium having program instructions embodied therewith, wherein the computer readable storage medium is not a transitory signal per se, and where the program instructions are executable by a processor to cause the processor to perform a method comprising identifying, by the processor, a first group of textual data, determining, by the processor, a plurality of word combinations within the first group of textual data, and determining, by the processor, a first plurality of key words for the first group of textual data, utilizing the plurality of word combinations.