Document Indexing via Ranked Term Encoding and Metadata Embedding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional search engines using inverted indexes face inefficiencies when processing large volumes of documents, particularly when frequent terms like 'the' and 'in' slow down searches, consuming significant CPU resources and taking milliseconds to generate results, even for infrequently used terms like 'Obama' or 'California', due to the need to scan millions of documents.
Innovation Solution
A method that assigns ranks to terms based on historical occurrence, encodes them with varying bit sizes, and embeds metadata within documents to quickly identify relevant terms, allowing for efficient querying by accessing specific arrays and bit-level counts, thereby bypassing unnecessary document scans.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If an inverted index is used to scan all documents for search terms, then all documents can be searched, but search time increases significantly when frequent terms are involved
Solution Approach 1:
The patent segments the inverted index into multiple data structures including a term index, document index, and posting list. Each structure serves a specific function: the term index stores term frequency information, the document index stores document frequency information, and the posting list stores actual document positions. This segmentation allows the system to quickly filter documents using frequency information before performing full scans, significantly reducing search time while maintaining completeness.
2Reliability
If conventional inverted index scanning is performed for millions of documents, then all documents are processed, but CPU resources are overwhelmed
Solution Approach 1:
The patent performs preliminary actions by pre-computing and storing term frequency and document frequency information in the term index and document index during the indexing phase. When a query arrives, the system uses these pre-computed frequency data to quickly estimate relevance and filter documents before performing detailed scanning. This preliminary preparation eliminates the need to scan all millions of documents for every query, dramatically improving CPU efficiency and query throughput while maintaining search accuracy.
Data Source
AI summary
This application relates to a technique that enables efficient indexing and querying of a set of documents stored at a computing device. The technique assigns a rank to each term of a set of terms based on a historical occurrence of each term to produce a set of ranked terms. The technique also creates subsets that include terms from the set of ranked terms to produce ranked term subsets that are encoded in accordance with an allocated bit size. The technique generates an index for each document of the set of documents through: (1) the identification of the ranked term subsets within the document, (2) the generation of arrays to store each ranked term subset identified within the document, and (3) the embedding of metadata within the document. The metadata includes: (i) one or more arrays and (ii) one or more counts corresponding to each array.