Inverted Index Compression for In-Memory Search Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing text database management systems face inefficiencies in in-memory searching and retrieval due to the high cost and limited capacity of RAM compared to disk storage, necessitating improved data compression and indexing techniques for faster access and processing.
Innovation Solution
Implementing a document-grained inverted index with compressed data structures and positional information stored in main memory, using techniques like Golomb encoding and binary search, along with a 'bag of words' data structure to facilitate quick reconstruction and retrieval of documents without disk access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is stored in RAM for faster access, then search speed is improved, but memory cost and capacity requirements increase
Solution Approach 1:
The patent applies parameter changes by transforming the representation of inverted index data from traditional formats to variable-length encodings (Golomb, Elias, Fibonacci). This changes the fundamental parameter of data representation to achieve higher compression ratios, allowing more index data to fit in RAM while maintaining fast access speeds.
Solution Approach 2:
The patent implements local quality by using different compression schemes for different parts of the inverted index based on local characteristics. Specifically, it uses Golomb coding for terms with geometric distribution of document frequencies, Elias coding for power-law distributions, and Fibonacci coding for uniform distributions. This localized adaptation optimizes memory usage for each term's specific access pattern.
2Ease of operation
If traditional inverted index structures are used, then search functionality is provided, but memory efficiency is poor
Solution Approach 1:
The patent fundamentally changes the parameter of data encoding by replacing fixed-width integer representations with variable-length prefix codes. This allows the same search functionality to be maintained while using significantly fewer bits per term-document pair, thereby improving memory efficiency without sacrificing operational capability.
Solution Approach 2:
The patent uses copying by creating compressed representations of the inverted index that can be loaded into memory. The compressed index structure copies only the essential information needed for search operations, eliminating redundant data while preserving full search functionality through efficient decoding algorithms.
3Quantity of substance
If disk storage is used for text databases, then storage capacity is adequate, but access speed is limited
Solution Approach 1:
The patent applies parameter changes by compressing the inverted index data structure to reduce its size by orders of magnitude. This transformation enables the entire index to fit in RAM, changing the access medium from slow disk I/O to fast memory access, while the compressed format preserves all necessary information for complete search functionality.
Solution Approach 2:
The patent segments the inverted index into term-specific compressed blocks, each optimized with appropriate encoding schemes. This segmentation allows selective loading of only relevant term data into memory during search operations, reducing overall memory requirements while maintaining fast access to actively searched terms.
Data Source
AI summary
A system may include a provider database, a reader database, and a database management system. The provider database may include a provider data area having a plurality of provider block addresses, and the reader database may include a reader data area having a plurality of reader block addresses, and a mapping of provider-specific identifiers to block addresses of the plurality of provider data pages and of reader-specific identifiers to block addresses of the plurality of reader data pages. The database management system may modify a database object of the reader database, the object being is associated with a provider-specific identifier; and modify the mapping to map the provider-specific identifier to a first block address of one of the plurality of reader data pages.