Document Similarity Computation Using Inverted Index Grams
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional systems for computing document similarity are inefficient due to the time-consuming process of comparing queries with numerous documents, often resulting in low precision and missing relevant documents.
Innovation Solution
The use of an inverted index that stores grams, document identifiers, and offsets allows for partial document reconstruction based on matched grams, followed by edit distance computation between query and document strings, with additional grams like skipping and mismatch grams enhancing the similarity assessment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional systems retrieve and compare queries with numerous documents from storage, then document similarity can be computed, but the computational overhead and retrieval time increase significantly
Solution Approach 1:
The patent segments documents into grams (character n-grams) and creates an inverted index structure where grams are separated from full documents. This allows the system to work with small gram units for similarity computation without retrieving entire documents, significantly reducing retrieval time while maintaining similarity accuracy through gram-based matching.
Solution Approach 2:
The patent extracts grams from documents and stores them in an inverted index separately from the full document content. This extraction enables the system to perform similarity computations using only the gram data structure, avoiding the need to retrieve and process complete documents from storage, thus reducing computational overhead.
2Productivity
If gram-counting approach is used to improve efficiency, then retrieval time decreases, but precision decreases and relevant documents are missed
Solution Approach 1:
The patent introduces an inverted index as an intermediary data structure between the query grams and the full documents. This inverted index stores grams with their positions and document identifiers, enabling efficient gram-based matching while preserving the ability to accurately assess document relevance through position-aware gram matching, thus maintaining precision while improving efficiency.
3Reliability
If all candidate documents are retrieved for similarity computation, then comprehensive results are obtained, but the number of documents to process increases computational overhead
Solution Approach 1:
The patent performs preliminary action by building an inverted index structure that pre-organizes grams from all documents before query processing. This pre-computation enables the system to efficiently identify candidate documents through gram matching without retrieving all documents, reducing computational complexity while maintaining result completeness through the structured index.
Data Source
AI summary
Systems, methodologies, media, and other embodiments associated with efficiently computing document similarity are described. One exemplary system embodiment includes logic to produce a gram from a string and logic to identify candidate documents based on identifying matches between query grams and document grams stored in an inverted index that relates grams to documents. The example system may also include logic to selectively partially reconstruct a candidate document from entries in the inverted index and logic to compute an edit distance between a string associated with a query and a string associated with the partially reconstructed candidate document. The example system may also include a signal logic configured to provide a signal corresponding to the edit distance.