Bit Vector Search Indexing with Accumulation Buffers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current search engine technologies face inefficiencies in updating and indexing large volumes of documents due to the complexities of posting lists, leading to slow processing speeds and inability to provide real-time search results for new information.
Innovation Solution
The implementation of a bit vector search index using accumulation buffers to efficiently index documents, allowing for instant or near-instant updates and real-time addition of new/changed documents by avoiding the need to sort and reorganize data, and using probabilistic approaches to allow for false positives during the matching process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional posting lists are used to index documents, then comprehensive search coverage is achieved, but processing speed and update efficiency deteriorate
Solution Approach 1:
The patent segments the search index into multiple independent bit vector files, each representing a different term or term group. This segmentation allows parallel processing of multiple terms simultaneously, dramatically improving processing speed while maintaining comprehensive search coverage through the union of all bit vectors.
Solution Approach 2:
The patent replaces the traditional mechanical posting list structure with a bit vector-based system. Instead of storing and processing lengthy posting lists that require sorting and merging, the system uses compact bit vectors that enable fast bitwise operations, achieving both speed improvement and comprehensive coverage.
2Reliability
If posting lists are updated with new documents, then index completeness is maintained, but update time and computational cost increase
Solution Approach 1:
The patent performs preliminary actions by pre-processing documents into term-frequency vectors before indexing. When new documents arrive, they are converted to bit vectors and merged with existing index files using efficient bitwise operations, avoiding the need for complete re-indexing and maintaining index completeness with minimal update time.
Solution Approach 2:
The patent changes the parameter representation from traditional posting lists to bit vectors. This parameter change enables incremental updates through bitwise OR operations, allowing new documents to be integrated efficiently while maintaining complete index coverage.
3Measurement precision
If all documents are ranked for every search query, then search accuracy is maximized, but processing time becomes unacceptable
Solution Approach 1:
The patent applies partial action by using bit vectors to quickly identify candidate documents that contain query terms, then applying full ranking only to these candidates. This approach maintains search accuracy for relevant documents while dramatically reducing processing time by avoiding unnecessary ranking of non-matching documents.
Solution Approach 2:
The patent segments the search process into two stages: first using bit vectors to filter candidate documents, then applying detailed ranking only to candidates. This segmentation enables the system to maintain high search accuracy while achieving acceptable processing speeds.
Data Source
AI summary
The technology described herein provides for indexing information in a bit vector search index. The bit vector search index comprises a data structure for indexing data about terms from a corpus of documents. The data structure includes a number of bit vectors. Each bit vector comprises an array of bits and corresponds to a different set of terms. Bits in the bit vector are used to represent whether at least one document corresponding to the bit includes at least one term from the set of terms corresponding to the bit vector. The bit vector search index is stored by first indexing information about documents using bit vectors on a first accumulation buffer storage device. When a threshold is satisfied, the information is transferred to bit vectors on a subsequent storage device.