Predictive Word Sequence Filtering for Document Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for accessing documents from large collections, such as aviation safety reports, face challenges with high dimensionality of word sequences, where rare but highly predictive sequences are difficult to identify efficiently, requiring excessive manual effort from subject matter experts.
Innovation Solution
A method and system that generate and expand candidate lists of word sequences by combining elements from a document corpus, determining predictive power using measures like information gain, and pruning sequences with low predictive power to efficiently access relevant documents.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional approaches focus on finding the most frequently occurring word sequences, then the dimensionality issue is addressed, but rare but highly predictive word sequences are missed
Solution Approach 1:
The patent changes the parameter for selecting word sequences from frequency-based to predictive power-based. By using information gain as the selection criterion, the system identifies word sequences that are most predictive of the target concept, regardless of their frequency. This allows rare but highly predictive sequences to be captured while managing the high dimensionality through the information gain threshold filtering.
2Measurement precision
If manual identification of highly predictive word sequences is performed by subject matter experts, then accurate predictive sequences are identified, but excessive manual effort is required
Solution Approach 1:
The patent implements a self-service approach where the system automatically identifies predictive word sequences using information gain calculations. The algorithm autonomously processes the document collection, computes predictive power for candidate sequences, and filters them based on predefined thresholds, eliminating the need for manual expert intervention while maintaining high accuracy in identifying predictive sequences.
Solution Approach 2:
The patent replaces the mechanical manual identification process with an automated computational system. By substituting human expert analysis with algorithmic information gain calculations and automated filtering mechanisms, the system achieves the same accuracy in identifying predictive word sequences without requiring excessive manual time and effort.
3Measurement precision
If the candidate list of word sequences is expanded to include more sequences, then coverage of predictive sequences improves, but the complexity of processing increases
Solution Approach 1:
The patent applies preliminary action by pre-calculating and storing the information gain values for candidate word sequences before the final selection. The system generates a comprehensive candidate list, pre-computes their predictive power metrics, and then filters them based on thresholds. This preliminary processing allows the system to handle a large number of sequences efficiently, maintaining completeness while managing processing complexity through structured computation.
Data Source
AI summary
Methods and systems for accessing documents in document collections using predictive word sequences are disclosed. A method for accessing documents using predictive word sequences include creating a candidate list of word sequences where respective ones of the word sequences comprise one or more elements derived from the document corpus; expanding the candidate list by adding one or more new word sequences, where each new pattern is created by combining one or more elements derived from the document corpus with one of the word sequences currently in the candidate list; determining a predictive power with respect to the subject for respective ones of entries of the candidate list, where the entries include the word sequences and the new word sequences; pruning from the candidate list ones of said entries with the determined predictive power less than a predetermined threshold; and accessing documents from the document corpus based on the pruned candidate list. The expanding of the candidate list can include creating each new pattern as a gapped sequence, where the gapped sequence comprises one of the word sequences and one of said elements separated by zero or more words. Corresponding system and computer readable media embodiments are also disclosed.