Paged Inverted Index for Low-Memory Database Retrieval

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional database indexing methods require loading the entire index into memory, leading to a large memory footprint that occupies space needed for other database operations, limiting memory usage efficiency.

Innovation Solution

Implementing a paged inverted index that stores data in a chain of pages, allowing only necessary pages to be loaded into memory, with a directory and posting list stored in compressed format to reduce memory usage and enable efficient data retrieval without requiring the entire index to be in main memory.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Speed

If the entire index is loaded into database memory, then data access speed is improved, but memory footprint increases

Engineering Contradiction:

Improvedata access speedVSAvoidmemory footprint

Core Design Contradiction:

SpeedVSQuantity of substance

Solution Approach 1:

The index is divided into multiple pages stored in database storage, with only the currently needed page loaded into memory at any given time. This segmentation allows the system to maintain fast access speeds for active data while dramatically reducing the overall memory footprint by eliminating the need to hold the entire index in memory simultaneously.

Inventive Principle:

Principle #1Segmentation

2Productivity

If the entire index is loaded into database memory, then data retrieval efficiency is improved, but available memory for other operations decreases

Engineering Contradiction:

Improvedata retrieval efficiencyVSAvoidavailable memory

Core Design Contradiction:

ProductivityVSQuantity of substance

Solution Approach 1:

The index pages are extracted from the main database storage and loaded into memory on-demand only when needed for query processing. This extraction approach maintains efficient data retrieval by keeping necessary index pages in fast memory, while freeing up substantial memory resources for other database operations by not pre-loading the entire index.

Inventive Principle:

Principle #2Taking out (Extraction)

3Quantity of substance

If the index is stored in compressed format, then memory usage is reduced, but access complexity increases

Engineering Contradiction:

Improvememory usageVSAvoidaccess complexity

Core Design Contradiction:

Quantity of substanceVSDevice complexity

Solution Approach 1:

The index pages are stored in a compressed format in database storage, reducing the space required to hold the entire index. When pages are loaded into memory, they are decompressed as needed. This parameter change in storage format achieves significant memory usage reduction while the decompression process is handled transparently by the system, managing the added complexity through automated page management.

Inventive Principle:

Principle #35Parameter changes

Data Source

PatentUS10140326B2Paged inverted index

Publication Date: 2018.11.27 SAP SE

AI summary

Disclosed herein are system and method embodiments for generating a paged inverted index. An embodiment is generated by storing a first data structure and the second data structure in a plurality of pages, where the plurality of pages are stored in the one or more memories. The first data structure is stored in the plurality of pages and includes a plurality of value identifiers, where a value identifier corresponds to an offset. The second data structure stored in the plurality of pages includes a plurality of row positions, wherein a row position is at a location that corresponds to the offset in the first data structure and identifies a position of row in a table that stores data associated with the value ID.