Normalized Inverted List Structure for Efficient Data Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Inverted lists and indices face challenges with large information bases, including expensive insertions and deletions, significant space overhead, and inefficient list operations, especially in applications with non-uniform key distributions.
Innovation Solution
A normalized inverted list structure is introduced, using pairs to represent the list, with prefix compression to reduce space overhead and enable sublinear list operations, and extended to efficiently represent contiguous and sparse intervals using bitmaps, allowing for efficient insertion, deletion, and list operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If traditional inverted lists are used to store mappings from keys to pointers, then quick access to inverted lists is achieved, but insertions and deletions become expensive when lists span multiple pages
Solution Approach 1:
The patent segments the inverted list into multiple pages, with each page containing a portion of the list. This segmentation allows the system to maintain quick access within pages while managing insertions and deletions at the page level, reducing the complexity of operations when lists span multiple pages.
Solution Approach 2:
The patent introduces a hierarchical structure with pages as intermediate units between the inverted list and the underlying storage. This adds a dimensional layer that enables efficient access within pages while simplifying insertion and deletion operations by confining them to page-level modifications rather than entire list modifications.
2Productivity
If inverted lists maintain ordered pointers to enable linear time list operations, then list operations such as intersection and union are efficient, but space overhead increases significantly
Solution Approach 1:
The patent applies different storage strategies to different portions of the inverted list based on local characteristics. Pages containing fewer pointers use compact representations, while pages with more pointers utilize more space-efficient encoding schemes. This local quality approach optimizes space usage while maintaining the ordered structure needed for efficient list operations.
Solution Approach 2:
The patent changes the representation parameters of pointers within pages, using variable-length encodings where the number of bits required to represent a pointer depends on the page context. This parameter change reduces space overhead while preserving the ordered nature of pointers, enabling efficient list operations.
3Ease of operation
If inverted lists store all pointers in sorted order, then linear time list operations are achieved, but the structure becomes complex and expensive to maintain
Solution Approach 1:
The patent segments the inverted list into pages, each maintaining sorted order independently. This segmentation reduces the complexity of maintaining global sorted order by confining sorting operations to smaller page-level units, making the overall structure easier to manage while preserving the linear time list operation capability.
Solution Approach 2:
The patent pre-sorts pointers within each page during the initial construction phase, establishing the ordered structure before operations are performed. This preliminary action eliminates the need for complex runtime sorting operations, simplifying the maintenance of sorted order while enabling efficient list operations.
Data Source
AI summary
A process is disclosed for the computer management of inverted lists and inverted indices, in which the standard representation and processing of inverted lists is changed in order to achieve a simpler, more compact and more efficient architecture.