Batch Prospective Search Algorithm for Low Latency Document Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Prospective searching in large volumes of electronic documents is resource-intensive and inefficient, particularly when performed in quasi-real time, as existing systems require significant processing resources to match each query with incoming documents, leading to increased latency and resource utilization.
Innovation Solution
A batch prospective search algorithm that constructs an index from tokens in batches of documents, allowing for efficient matching against a large set of queries without requiring significant increases in processing resources, utilizing multi-core processors for parallel processing to handle high-bandwidth searches with low latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If prospective searching is performed in quasi-real time on large volumes of electronic documents, then search results can be provided timely, but processing resources are significantly consumed and latency increases
Solution Approach 1:
The system performs preliminary actions by constructing an index from tokens in batches of documents before queries are processed. This batch indexing allows the system to pre-organize document content into searchable structures, so that when queries arrive in quasi-real time, matching can be performed efficiently without consuming excessive processing resources. The index construction is done in advance on batches of documents, enabling fast query response while avoiding the resource intensity of real-time indexing.
2Measurement precision
If traditional prospective search methods are used to match each query with incoming documents, then accurate search results can be provided, but resource utilization increases and latency is introduced
Solution Approach 1:
The system performs preliminary batch indexing of document tokens before query processing, creating pre-organized index structures that enable fast query matching. This eliminates the need to perform resource-intensive matching operations in real-time, thereby reducing latency while preserving search accuracy through the use of pre-computed index data.
Solution Approach 2:
The system segments the search process into distinct phases: batch document indexing is performed separately from query processing. The index construction phase processes documents in batches to build token-based index structures, while the query phase uses these pre-built indexes for efficient matching. This segmentation allows accurate search results to be provided without the latency and resource consumption of traditional real-time matching.
3Productivity
If batch prospective search is implemented with multi-core parallel processing, then high-bandwidth searches can be handled with low latency, but system complexity increases
Solution Approach 1:
The system segments the batch indexing task into independent units that can be processed in parallel by multiple cores. Each core can process a subset of documents or tokens independently, constructing portions of the index simultaneously. This segmentation enables high throughput and low latency by utilizing parallel processing capabilities, while the modular nature of the segmented tasks keeps system complexity manageable through clear division of work.
Data Source
Figure 1~2
Figure 3A
Figure 3B
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for prospective searching a batch of electronic documents. In one aspect, a method includes identifying a plurality of electronic documents for a prospective search against a plurality of predefined queries, generating an index for the plurality of electronic documents, wherein generating an index includes identifying a plurality of tokens associated with content of electronic documents in the plurality of electronic documents, the index including one or more index records each corresponding to a token in the plurality of tokens and identifying one or more electronic documents, in the plurality of electronic documents, having content associated with the token, and using the index to determine, for each electronic document in the plurality of electronic documents, whether one or more queries in the plurality of queries match the electronic document.