Parallel Top-K Search Using Multi-Core Posting List Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current search engines face significant latency issues during the top-k content item retrieval stage, particularly with longer queries, leading to a diminished user experience and increased frustration, as they need to analyze vast amounts of data, which impacts business traffic.
Innovation Solution
Implementing a system that utilizes multi-core processing to analyze multiple posting lists in parallel, where each processor handles a separate posting list, and determines relevant content items based on term scores, allowing for faster retrieval of top-k content items by leveraging parallel computing power and efficient data structure management.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If sequential processing is used to analyze posting lists, then processing accuracy is maintained, but search latency increases significantly
Solution Approach 1:
The patent divides the search processing task into multiple independent segments, where each posting list is processed by a separate processor thread. This segmentation allows parallel execution of multiple posting list analyses simultaneously, dramatically reducing the total time required to retrieve top-k search results while maintaining processing accuracy through independent thread execution.
2Productivity
If multiple posting lists are analyzed in parallel, then retrieval speed increases, but system complexity increases
Solution Approach 1:
The patent implements dynamic thread management where processor threads are created and terminated based on the actual number of posting lists that need processing. This dynamic approach allows the system to scale processing capacity according to query complexity without maintaining a fixed complex infrastructure, balancing retrieval speed improvements with manageable system complexity.
3Measurement precision
If comprehensive data analysis is performed on all posting lists, then result accuracy is improved, but processing time increases
Solution Approach 1:
The patent performs preliminary actions by analyzing posting lists in parallel before final result compilation. Multiple processors simultaneously evaluate different posting lists and prepare their results in advance, allowing the system to aggregate accurate results from all sources without sequentially processing each list, thus maintaining result accuracy while reducing overall processing time.
Data Source
AI summary
Methods, systems, and programming for retrieving content items for a search are described herein. In a non-limiting embodiment, a query including a plurality of terms may be received. For each of the plurality of terms, a posting list of one or more content items may be obtained. The posting list may include a ranked list of term scores corresponding to the one or more content items, each of the term scores being indicative of a level of relevance of a corresponding content item to a term associated with the posting list. A list of relevant content items for the query may be determined based on the term scores in each posting list for the one or more content items identified with respect to each term. At least one of the relevant content items may be provided as a response to the query.