Mask-Augmented Inverted Index for Phrase Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing search engines either cannot perform phrase queries or require significant storage, computing, and network resources to support them, due to limitations in current inverted index technologies such as document-ID and positional indices.
Innovation Solution
A mask-augmented inverted index that uses probabilistic representations of integers to map tokens and their positional relationships, allowing for efficient phrase query searching while minimizing processing and storage resources, implemented through Bloom filters and rotation of masks to identify relevant documents.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a positional index is used to support phrase queries, then phrase query capability is improved, but storage cost increases significantly
Solution Approach 1:
The patent segments the positional information into two separate components: a Bloom filter that stores the presence/absence of tokens at specific positions, and a separate posting list that stores actual position details. This segmentation allows the system to support phrase queries using only the compact Bloom filter portion, avoiding the need to store complete positional information for all tokens, thereby reducing storage costs while maintaining phrase query capability.
Solution Approach 2:
The patent employs Bloom filters, which are probabilistic data structures that provide space-efficient membership testing. The Bloom filter acts as a 'porous' structure that allows quick filtering of candidate documents for phrase queries while accepting a small probability of false positives. This enables the system to support phrase queries with minimal storage overhead, as the Bloom filter occupies significantly less space than traditional positional indices.
2Quantity of substance
If a document-ID index is used to reduce storage cost, then storage efficiency is improved, but phrase query performance deteriorates
Solution Approach 1:
The patent enhances the document-ID index by segmenting it into two functional parts: the Bloom filter component that enables phrase query filtering, and the traditional posting list component that provides document identification. This segmentation allows the index to maintain the storage efficiency of document-ID indices while adding phrase query capability through the Bloom filter, which can quickly determine whether candidate documents contain the required token sequences.
Solution Approach 2:
The Bloom filter serves as an intermediary structure between the document-ID index and phrase query processing. It acts as a filtering layer that quickly identifies candidate documents containing the required token sequences before the system needs to load and scan full documents. This intermediary structure enables phrase queries to be performed efficiently without requiring complete positional information to be stored in the index.
3Measurement precision
If candidate documents are loaded and scanned to filter correct token sequences, then phrase query accuracy is improved, but computational and network resources increase
Solution Approach 1:
The patent performs preliminary filtering using the Bloom filter before documents are loaded and scanned. The Bloom filter pre-identifies candidate documents that are likely to contain the required token sequences based on their compact probabilistic representation. This preliminary action reduces the number of documents that need to be loaded from storage and scanned for verification, thereby reducing computational and network resources while maintaining phrase query accuracy through subsequent verification of the filtered candidates.
Data Source
AI summary
The embodiments disclosed herein are related to a computing system for generating a mask-augmented inverted index. The mask-augmented inverted index is structured to allow phrase query searching while minimizing the amount of computing system processing and memory resources needed to generate the mask-augmented inverted index. In one embodiment, a first token is mapped to a first listing of documents that include the first token. A first mask is included that comprises a probabilistic representation of a set of integers corresponding to one or more locations of the first token in each of the individual documents of the first listing. A second mask is included that comprises a probabilistic representation of a set of integers that indicate a positional relationship between the first token and one or more other tokens in each of the individual documents of the first listing.