Document Identification Tag Assignment for Search Indexing Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current search engines face challenges in reducing latency between the time a web page is posted or updated and when it is indexed, as they require crawling and indexing the entire document collection, which becomes inefficient with the growing number of documents on the Internet.
Innovation Solution
The method involves subdividing document identification tags into tiers based on query-independent metrics, allowing for the assignment of tags before complete crawling, ensuring that new documents are indexed with relevance information, even when not all documents are available, by selecting the appropriate tier for each new document based on its metric values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the entire collection of documents is crawled and sorted before assigning document identification tags, then the index can return documents in order of decreasing query-independent relevance, but the latency between page posting and indexing increases significantly
Solution Approach 1:
The patent applies preliminary action by assigning document identification tags based on query-independent metrics (such as PageRank) before the complete document collection is available. This allows new documents to be indexed immediately upon crawling rather than waiting for the entire collection to be processed, thereby reducing indexing latency while maintaining relevance ordering accuracy through the use of pre-computed relevance metrics
Solution Approach 2:
The patent segments the document collection processing into two independent phases: (1) computation of query-independent relevance metrics for each document, and (2) assignment of document identification tags based on these metrics. This segmentation allows tag assignment to proceed as soon as individual document metrics are available, without waiting for complete collection processing
2Stability of the object's composition
If document identification tags are assigned sequentially after complete sorting of the document collection, then documents are properly ordered by relevance, but the time required to replace or update indexed pages increases
Solution Approach 1:
The patent computes query-independent relevance metrics for documents in advance and uses these pre-computed values to assign document identification tags immediately when documents are crawled, rather than waiting for complete collection sorting. This preliminary assignment enables rapid index updates while maintaining ordering consistency through the stability of query-independent metrics
Solution Approach 2:
The patent uses query-independent relevance metrics (such as PageRank values) as a copy or proxy for actual document relevance to queries. These metrics serve as a surrogate ordering mechanism that allows documents to be indexed in relevance order without requiring complete collection processing, thereby enabling faster updates while maintaining ordering consistency
3Measurement precision
If computational resources are allocated to sort the entire collection before tag assignment, then accurate relevance ordering is achieved, but the system becomes inefficient for large-scale document collections
Solution Approach 1:
The patent extracts the computation of query-independent relevance metrics from the complete collection sorting process. By separating metric computation from full collection processing, the system can assign document identification tags based on these extracted metrics without requiring computationally expensive complete sorting, thereby reducing device complexity while maintaining relevance metric accuracy
Solution Approach 2:
The patent performs preliminary computation of query-independent relevance metrics for each document independently before tag assignment. This preliminary action eliminates the need for computationally intensive complete collection sorting, as tags can be assigned based on pre-computed metrics alone, significantly reducing computational resource requirements for large-scale collections
Data Source
AI summary
Document identification tags are assigned to documents to be added to a collection of documents. Based on query-independent information about a new document, a document identification tag is assigned to the new document. The document identification tag so assigned is used in the indexing of the new document. When a list of document identification tags are produced by an index in response to a query, the list is approximately ordered with respect to a measure of query-independent relevance. In some embodiments, the measure of query-independent relevance is related to the connectivity matrix of the World Wide Web. In other embodiments, the measure is related to the recency of crawling. In still other embodiments, the measure is a mixture of these two. The provided systems and methods allow for real-time indexing of documents as they are crawled from a collection of documents.