Paged Inverted Index Layout for Lower Database Memory Use
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional database systems face memory constraints due to large index sizes, which limit memory availability for other database operations, as they load entire indexes into memory for faster data access.
Innovation Solution
Implementing a paged inverted index that stores only the necessary pages in memory, using a chain of pages with a directory and posting list, allowing for efficient retrieval of row positions without requiring the entire index to be in main memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the entire index is loaded into database memory, then data access speed is improved, but memory footprint increases
Solution Approach 1:
The index is divided into multiple pages, where each page contains a subset of the index data. Only the necessary pages are loaded into memory at any given time, rather than loading the entire index. This segmentation allows the system to maintain fast access speeds for active data while significantly reducing the overall memory footprint.
Solution Approach 2:
The patent introduces a hierarchical memory structure with index pages stored in database memory and corresponding data rows stored in table space. This multi-dimensional storage approach allows the system to manage large indexes by separating the index metadata (in memory) from the actual data rows (in table space), enabling efficient access without requiring the entire index to reside in memory simultaneously.
2Productivity
If the entire index is loaded into database memory, then data retrieval efficiency is improved, but memory availability for other operations decreases
Solution Approach 1:
By segmenting the index into multiple pages and loading only the necessary pages into memory based on query requirements, the system maintains high data retrieval efficiency for active operations while freeing up memory resources for other database operations. The directory structure enables selective page loading, ensuring that memory is allocated dynamically based on actual needs.
Solution Approach 2:
The index structure employs dynamic page loading and unloading based on query patterns and memory availability. The system can adaptively load relevant index pages into memory when needed and unload them when no longer required, allowing memory resources to be dynamically allocated between index operations and other database functions, thereby maintaining productivity while optimizing memory availability.
Data Source
Figure 1
Figure 2A~2C
Figure 3
AI summary
Disclosed herein are system and method embodiments for generating a paged inverted index. An embodiment is generated by storing a first data structure and the second data structure in a plurality of pages, where the plurality of pages are stored in the one or more memories. The first data structure is stored in the plurality of pages and includes a plurality of value identifiers, where a value identifier corresponds to an offset. The second data structure stored in the plurality of pages includes a plurality of row positions, wherein a row position is at a location that corresponds to the offset in the first data structure and identifies a position of row in a table that stores data associated with the value ID.