Phrase-Based Document Indexing for Versioned Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional information retrieval systems face limitations in indexing and retrieving documents based on concepts rather than individual terms, failing to capture topical relationships and unable to efficiently index and archive large-scale document collections like the Internet, which results in missing a significant portion of available pages and lacking the ability to search for prior document versions.
Innovation Solution
An information retrieval system that indexes and searches documents using phrases, identifies related phrases through co-occurrence analysis, and employs a multiple index structure to handle large volumes of data, allowing for the storage and retrieval of multiple document versions over time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional systems index documents based on individual terms, then indexing is simple and storage requirements are reduced, but the system cannot identify conceptually related documents that do not have exact query terms
Solution Approach 1:
The patent segments the indexing process into two distinct phases: phrase identification (where n-grams are extracted and evaluated for phrasehood) and indexing (where only validated phrases are indexed). This segmentation allows the system to maintain conceptual relationships while managing complexity through structured processing stages.
Solution Approach 2:
The patent introduces an intermediary evaluation stage between raw n-gram extraction and final indexing. This intermediary phase filters and validates potential phrases using statistical criteria, acting as a mediator that transforms raw text segments into meaningful indexed phrases without directly indexing all possible combinations.
2Reliability
If the system indexes all possible phrases (including n-grams of 3-5 words), then phrase-based retrieval accuracy improves, but memory requirements become prohibitively large
Solution Approach 1:
The patent changes the parameter of phrase length consideration by focusing on n-grams of specific sizes (2-4 words) rather than all possible lengths. This parameter change significantly reduces the number of candidate phrases while maintaining retrieval accuracy, as most meaningful phrases fall within this length range.
Solution Approach 2:
The patent creates a simplified representation of phrases through standardized n-gram extraction and validation processes. By copying and validating only the most promising phrase candidates rather than storing all possible phrase combinations, the system reduces memory requirements while preserving essential phrase information for retrieval.
3Measurement precision
If conventional systems use inverted index with positional information for every term, then term location precision is improved, but storage capacity is severely limited
Solution Approach 1:
The patent extracts and stores only the essential phrase information (phrase text and document references) while omitting redundant positional information that can be derived during retrieval. This extraction of only necessary data reduces storage capacity requirements while maintaining the ability to locate phrases within documents.
Solution Approach 2:
The patent creates a universal phrase index structure that serves multiple functions: identifying phrase occurrences, determining phrase relationships, and enabling retrieval operations. This multi-functional index design eliminates the need for separate positional indexing structures, reducing overall storage requirements while maintaining precision.
4Productivity
If the system indexes only current document versions, then indexing complexity is reduced, but the ability to search for prior document versions is lost
Solution Approach 1:
The patent performs preliminary actions by capturing and indexing document versions at specific time points during the indexing process. By preparing and storing historical versions proactively during normal indexing operations, the system enables future retrieval of prior versions without significantly increasing ongoing indexing complexity.
Solution Approach 2:
The patent introduces dynamic version management that allows the index to capture multiple snapshots of document versions over time. This dynamic approach enables the system to adapt between indexing only current versions (for simplicity) and indexing historical versions (for versatility), based on configuration and retrieval needs.
Data Source
AI summary
An information retrieval system uses phrases to index, retrieve, organize and describe documents. Phrases are identified that predict the presence of other phrases in documents. Documents are the indexed according to their included phrases. Index data for multiple versions or instances of documents is also maintained. Each document instance is associated with a date range and relevance data derived from the document for the date range.