Compressed Inverted Index Layout for Lower Memory Search
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Inverted indexes consume significant memory resources as data grows, limiting available memory for other data or processes, despite providing performance benefits by being stored in primary memory.
Innovation Solution
Implement memory optimization techniques such as compressing and paginating inverted indexes, using Golomb encoding, and dividing the index into smaller portions to reduce memory footprint without hindering query performance, including the use of skip pointers for efficient data retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If inverted indexes are stored in primary memory to improve query performance, then search speed is improved, but memory consumption increases
Solution Approach 1:
The inverted index is divided into multiple segments or partitions, where each segment contains a subset of the index data. This allows the system to load only necessary segments into primary memory during queries, reducing overall memory consumption while maintaining query performance for specific search terms.
Solution Approach 2:
The patent introduces a hierarchical memory structure with multiple levels (e.g., L1 cache, L2 cache, and disk storage). By organizing the inverted index across different memory dimensions, frequently accessed portions are kept in fast primary memory while less frequently accessed data is stored in slower but larger-capacity storage, balancing speed and memory usage.
2Reliability
If the index size grows with data growth to maintain comprehensive coverage, then search completeness is improved, but available memory for other uses decreases
Solution Approach 1:
The patent extracts only the essential or frequently accessed portions of the inverted index into primary memory, while the complete index remains stored on disk or in secondary storage. This extraction approach ensures that search completeness is maintained by having access to the full index when needed, while primary memory is freed up for other uses.
Solution Approach 2:
The system pre-loads or caches frequently accessed index segments into primary memory before they are actually needed during queries. This preliminary action ensures that when searches are performed, the necessary data is already in fast memory, maintaining search completeness and performance while optimizing memory utilization patterns.
Data Source
AI summary
Disclosed herein are system, method, and computer program product embodiments for providing a memory optimization system for inverted indexes. An embodiment operates by determining a value identifier corresponding to a value to be searched for in a database based on a query. From a lookup table, a prefix of an offset corresponding to a location in a compressed posting list that identifies row locations where the value is found in the database is retrieved. From a directory page, a suffix of the offset corresponding to the location is retrieved. The offset is generated based on the prefix and the suffix. From the posting list, row identifiers corresponding to the generated offset are retrieved. The retrieved row identifiers are returned responsive to the query.