Inverted Index Search Using Multithreaded Segment Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current search systems face inefficiencies in processing multiple search queries simultaneously due to static partitioning of data into shards, leading to increased execution time and reduced performance as the number of shards increases, especially when handling queries of varying complexity and load.
Innovation Solution
A computer-implemented method and system that allows for the multithreaded searching of interspaced segments of posting lists corresponding to multiple search terms, using separate threads to yield per-thread search results and aggregate them for faster query execution, while prioritizing complex queries with additional threads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is divided into more shards to handle larger data collections, then the system can manage larger data volumes, but search query execution time increases and performance deteriorates
Solution Approach 1:
The patent divides each shard into multiple segments and processes these segments in parallel using multiple threads. This segmentation allows the system to handle large data volumes while reducing query execution time by processing different segments simultaneously rather than sequentially through the entire shard.
Solution Approach 2:
The patent introduces dynamic thread pooling that adapts the number of processing threads based on query complexity and system load. For simple queries, fewer threads are used; for complex queries involving multiple shards and segments, more threads are allocated. This dynamic adjustment optimizes performance across varying workloads while managing large data collections.
2Productivity
If the number of shards is increased to improve parallelism, then more queries can be processed in parallel, but the complexity of query coordination and result aggregation increases
Solution Approach 1:
The patent merges the processing of multiple segments within each shard into a unified thread pool architecture. Instead of coordinating independently across many shards, the system combines segment processing tasks from multiple shards and executes them through a shared pool of threads, simplifying coordination while maintaining parallelism.
Solution Approach 2:
The thread pool serves multiple functions: it processes segments from different shards, handles different types of search operations, and manages result aggregation. This universal processing mechanism reduces the complexity of query coordination by using a single multi-functional system rather than separate specialized components for each shard.
3Ease of manufacture
If static shard partitioning is used to simplify system architecture, then the system is easier to implement and maintain, but it cannot efficiently handle queries of varying complexity and leads to poor load balancing
Solution Approach 1:
The patent maintains the simplicity of static shard partitioning for system architecture while introducing dynamic thread allocation at the segment processing level. The thread pool dynamically adjusts the number and allocation of threads based on query complexity, allowing simple implementation of the base architecture with enhanced performance through adaptive resource allocation.
4Speed
If more threads are allocated to process complex queries, then query execution speed improves, but system resource consumption increases
Solution Approach 1:
The patent implements dynamic thread pooling that adjusts the number of active threads based on query complexity and current system load. For simple queries, minimal threads are used; for complex queries involving multiple shards and segments, the system allocates additional threads from the pool. This dynamic allocation optimizes execution speed while controlling resource consumption by matching thread usage to actual processing needs.
Data Source
AI summary
Computer-implemented method of and system for searching an inverted index having a plurality of posting lists, comprising: Receiving a search query including a plurality of search terms to be searched. Multithreadedly searching a plurality of complementary sets of corresponding interspaced segments of each of the plurality of posting lists corresponding to the plurality of search terms, each set being searched via a separate thread to yield per-thread search results. Aggregating the per-thread search results to yield aggregated search results. Transmitting at least a portion of the aggregated search results.