Hybrid Positional Posting Lists for Smaller, Faster Inverted Indexes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Inverted indexing systems face challenges in efficiently managing large document collections due to the tradeoff between index size and search retrieval time, with existing solutions consuming significant storage and computational resources, especially when dealing with redundant documents and multiple domains.
Innovation Solution
The implementation of a hybrid positional posting list that uses a single numeric value to represent both a document identifier and the local positions of frequently occurring terms, employing a global positions block and bitrank vector to minimize storage overhead and enable efficient cross-domain searches.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the index size is increased to improve search retrieval speed, then search performance may improve, but storage requirements increase significantly
Solution Approach 1:
The patent segments the traditional posting list into two distinct structures: a compressed posting list for documents with many occurrences (using single numeric values) and a traditional posting list for documents with few occurrences (storing document IDs and positions). This segmentation allows the system to optimize storage for high-frequency terms while maintaining search efficiency for both cases.
Solution Approach 2:
The patent changes the parameter representation for postings by using a single numeric value to encode both document ID and position information when the number of postings exceeds a threshold. This parameter transformation reduces the storage space required per posting from multiple values (document ID + position) to a single compressed value, directly addressing the storage vs. speed contradiction.
2Measurement precision
If the index includes detailed positional information for all terms, then search precision improves, but computational resources increase
Solution Approach 1:
The patent applies partial action by selectively storing detailed positional information only when necessary (when posting count is below the threshold). For high-frequency terms with many postings, the system uses compressed representation that provides sufficient precision for most search operations without the overhead of storing every individual position, thus reducing computational resources while maintaining adequate search precision.
3Reliability
If the system stores separate document IDs and positions for each posting, then search accuracy is maintained, but storage overhead increases
Solution Approach 1:
The patent merges the document ID and position information into a single numeric value for postings that exceed the threshold. This merging combines multiple data elements (document identifier and position) into one compressed representation, reducing storage overhead while maintaining the ability to retrieve accurate search results through the hybrid structure that preserves detailed information for low-frequency terms.
Data Source
AI summary
An information retrieval system uses an inverted indexing system composed of hybrid positional posting lists. A hybrid positional posting list includes a number of postings, where each posting contains a single numeric value to represent both the document containing a term and the location of the term in the document. When the number of postings for a document exceeds a threshold, a single-valued posting represents all the locations of a term in a document and the locations within the document. When the number of postings for a document is less than a threshold, each location of a term and the associated document is represented by a single-valued posting. In this manner, the storage requirements for the inverted indexing system are reduced and the processing speed increased.