Normalized Inverted List Structure for Efficient Data Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Inverted lists and indices face challenges with large information bases, including expensive insertions and deletions, significant space overhead, and costly list operations, especially in applications like text retrieval where lists can be very large and non-uniformly distributed.
Innovation Solution
A normalized inverted list structure is introduced, using prefix compression to reduce storage overhead and allowing for sublinear list operations by representing lists as pairs and extending to efficiently represent contiguous intervals of pointer values, which simplifies insertions, deletions, and reduces space requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional inverted lists are used to store mappings from keys to pointers, then the structure can support information retrieval operations, but insertions and deletions become expensive when lists span multiple pages and contain large numbers of pointers
Solution Approach 1:
The patent divides the inverted list into multiple segments or pages, where each page contains a portion of the pointer list. This segmentation allows insertions and deletions to be performed locally within a page rather than requiring reorganization of the entire list, reducing the computational cost of these operations while maintaining the ability to retrieve pointers across all pages.
Solution Approach 2:
The patent introduces an additional dimension by organizing pointers not just in a single linear list but across multiple pages with hierarchical or multi-level indexing. This dimensional expansion allows the system to maintain ordered access capabilities while distributing the data structure across multiple accessible units, reducing the complexity of individual operations.
2Productivity
If inverted lists maintain ordered pointers to enable linear-time list operations, then intersection, union, and subtraction operations can be performed efficiently, but the space required increases significantly
Solution Approach 1:
By segmenting the ordered pointer list into multiple pages, the patent reduces the space required in any single location while maintaining the overall ordered structure. Each page stores only a portion of the pointers, distributing the storage burden and enabling efficient operations on smaller, more manageable data units that can be loaded into memory as needed.
Solution Approach 2:
The patent loads only the necessary portions of the ordered pointer list into memory during operations rather than requiring the entire list to be resident. This partial action approach maintains the efficiency of ordered list operations for the active data while reducing overall space requirements by keeping inactive portions on secondary storage.
3Quantity of substance
If inverted lists are stored in fixed-size pages as the minimum access unit, then storage is optimized, but insertions and deletions require maintaining order across page boundaries which is difficult and expensive
Solution Approach 1:
The patent embraces the fixed-page structure by segmenting the inverted list into multiple pages, each serving as an independent storage unit. Insertions and deletions are performed within the context of these pages, with the system managing page boundaries and overflow/underflow conditions. This segmentation transforms the problem from maintaining order across unbounded lists to managing ordered sequences within fixed boundaries, which is more tractable.
Solution Approach 2:
The patent introduces dynamic page management where pages can be allocated, deallocated, and reorganized based on the needs of insertions and deletions. When a page becomes full or empty, the system can dynamically adjust the page structure, merge or split pages, and redistribute pointers to maintain optimal storage utilization and operation efficiency.
Data Source
AI summary
A process is disclosed for the computer management of inverted lists and inverted indices, in which the standard representation and processing of inverted lists is changed in order to achieve a simpler, more compact and more efficient architecture.