In-Memory Array to On-Disk Page Materialization

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Database systems face challenges in managing increasing data capacities and balancing the performance of in-memory and on-disk storage approaches, particularly in supporting large-scale databases with terabytes or petabytes of data, where existing methods struggle with efficient data access and storage.

Innovation Solution

The implementation of a dual storage method using an in-memory array and an on-disk page structure, where data is materialized from the in-memory array to pages based on marking and using protocols like lazy or batch materialization, allowing for efficient data persistence and random access without requiring translation, thereby integrating the advantages of both in-memory and on-disk systems.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Quantity of substance

If data is stored only in on-disk storage, then storage capacity is sufficient for large databases, but data access speed and system performance deteriorate

Engineering Contradiction:

Improvestorage capacityVSAvoiddata access speed

Core Design Contradiction:

Quantity of substanceVSSpeed

Solution Approach 1:

The database system segments data storage into two distinct parts: hot data stored in memory arrays for fast access, and cold data stored in on-disk page structures for capacity. This segmentation allows the system to optimize for both speed and storage capacity simultaneously by placing frequently accessed data in memory and less frequently accessed data on disk.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent introduces a temporal dimension to data storage by categorizing data into hot and cold based on access patterns. This dimensional approach allows the system to manage data across different storage media (memory and disk) based on their access frequency, thereby resolving the contradiction between access speed and storage capacity.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

2Speed

If data is stored only in in-memory storage, then data access speed improves, but storage capacity becomes insufficient for large databases

Engineering Contradiction:

Improvedata access speedVSAvoidstorage capacity

Core Design Contradiction:

SpeedVSQuantity of substance

Solution Approach 1:

Inventive Principle:

Principle #1Segmentation

3Reliability

If data is materialized from in-memory array to on-disk pages continuously, then data persistence is improved, but system overhead and complexity increase

Engineering Contradiction:

Improvedata persistenceVSAvoidsystem overhead

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

Instead of continuous materialization, the system employs periodic action by using trigger-based materialization. Data is materialized from memory arrays to on-disk pages only when specific conditions are met, such as when a threshold is reached or a particular event occurs. This reduces system overhead while maintaining data persistence reliability.

Inventive Principle:

Principle #19Periodic action

Solution Approach 2:

The system changes the parameter of materialization frequency from continuous to conditional/periodic. By introducing parameters such as materialization thresholds and trigger conditions, the system optimizes the balance between data persistence and system overhead, materializing data only when necessary rather than continuously.

Inventive Principle:

Principle #35Parameter changes

4Productivity

If hybrid in-memory and on-disk approach is used, then both performance and storage capacity are improved, but system complexity increases

Engineering Contradiction:

Improvesystem throughputVSAvoidsystem complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent merges in-memory and on-disk storage systems into a unified hybrid architecture. The memory arrays and on-disk page structures work together as an integrated system, with clear protocols for data materialization and access. This merging allows the system to achieve both high performance and large storage capacity while managing complexity through unified access protocols and coordinated operation between the two storage media.

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS9830109B2Materializing data from an in-memory array to an on-disk page structure

Publication Date: 2017.11.28 SAP SE

AI summary

The subject matter disclosed herein provides methods for materializing data from an in-memory array to one or more pages. An in-memory array holding a column of data can be maintained. One or more pages can be maintained. Each of the one or more pages can have one or more rows for storing the column of data. At least one of the one or more pages can be marked for materialization. The column of data can be materialized by copying the data from the in-memory array to the one or more rows of the one or more pages. The materializing can be based on the marking. Related apparatus, systems, techniques, and articles are also described.