Fast Merging Inverted Chains for Search Engines
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing search engine technologies face inefficiencies in search time due to long inverted chains, where the overall search time is proportional to the number of data records hit by a keyword, often taking hundreds of milliseconds or seconds to complete, especially when keywords like 'news' or 'sport' result in millions of hits.
Innovation Solution
A method for fast merging inverted chains involves pre-setting an inverted index with recorded chain lengths, sorting chains by ascending length, and merging them sequentially from the shortest, using multi-granularity text segmentation and multidimensional document features to rank high-quality documents, and suspending the merging process when a preset number of documents are recalled.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional inverted index merging is used, then all documents are retrieved, but search time increases significantly
Solution Approach 1:
The patent applies partial action by retrieving only the top K high-quality documents instead of all documents in the inverted chains. The merging process is optimized to stop early when sufficient results are obtained, sacrificing complete retrieval for significantly reduced search time while maintaining high result quality through multidimensional scoring.
2Ease of manufacture
If inverted chains are merged sequentially without sorting, then merging is simpler, but the number of comparisons and traverses increases
Solution Approach 1:
The patent applies preliminary action by sorting the inverted chains in ascending order of their lengths before merging. This preprocessing step ensures that shorter chains are merged first, reducing the number of comparisons and traverses required during the merging process, thereby significantly improving merging speed while maintaining process simplicity.
3Reliability
If all inverted chains are processed completely, then all relevant documents are found, but processing time becomes unacceptably long
Solution Approach 1:
The patent applies local quality by differentiating the processing of documents based on their quality scores. High-quality documents (those with higher multidimensional scores) are prioritized and retrieved first, while lower-quality documents are processed less intensively or not at all. This selective processing maintains reliability for high-value results while dramatically improving processing speed.
Data Source
AI summary
In accordance with various embodiments of the disclosed subject matter, a method for fast merging inverted chains, and a related apparatus are provided. In some embodiments, the method comprises: pre-setting an inverted index including a plurality of inverted chains and recording a length of each inverted chain; searching the inverted index and obtaining a subset of the plurality of inverted chains that correspond to at least one keyword; sorting the subset of the plurality of inverted chains in an ascending order of the lengths of the subset of multiple inverted chains; and merging the subset of the plurality of inverted chains sequentially as the ascending order starting from one of the subset of the plurality of inverted chains that has the shortest length.