Asynchronous Database Persistence via In-Memory Buffering

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Combined transactional and analytical database management systems face significant overhead due to persistent recovery logs, slowing down processing and requiring optimized usage of changes from the source database in the target database, while also needing to address the recovery problem elegantly.

Innovation Solution

A method for crash recovery in linked databases that synchronizes selected table content between a source and target database, applies changes to an in-memory target database using a recovery log, and stores changes asynchronously to persistent storage, allowing the target database to restore from the latest snapshot and replay missing changes upon a crash, decoupling logging from processing and enabling incremental recovery.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If changes are written to persistent storage synchronously during transaction processing, then data durability is guaranteed, but processing speed deteriorates due to I/O wait time

Engineering Contradiction:
Improvedata durabilityVSAvoidprocessing speed
Core Design Contradiction:
ReliabilityVSProductivity

Solution Approach 1:

The patent segments the persistence operation from the transaction processing flow. Changes are first applied to the in-memory target database portion, allowing immediate processing continuation. The persistent storage write is separated as an asynchronous background operation, eliminating the I/O wait from the critical transaction processing path while still ensuring durability through eventual persistence.

Inventive Principle:
Principle #1Segmentation

Solution Approach 2:

The patent performs preliminary action by pre-writing changes to the in-memory target database portion before persistent storage. This allows the system to have data ready for immediate use while the slower persistent write occurs in the background. The in-memory portion serves as a preliminary buffer that enables fast processing while persistence is being prepared.

Inventive Principle:
Principle #10Preliminary action

2Reliability

If synchronous persistence is implemented, then data consistency is maintained, but system overhead increases due to logging requirements

Engineering Contradiction:
Improvedata consistencyVSAvoidsystem overhead
Core Design Contradiction:
ReliabilityVSDevice complexity

Solution Approach 1:

The patent extracts the persistence overhead from the main transaction processing system. By using an asynchronous write mechanism for the target database changes, the logging and persistence operations are removed from the critical path. This reduces system overhead while maintaining consistency through the use of the in-memory portion as a buffer and recovery logs for the source database.

Inventive Principle:
Principle #2Taking out (Extraction)

Solution Approach 2:

The in-memory target database portion acts as an intermediary between the source database changes and persistent storage. It buffers changes temporarily, allowing the system to decouple the timing of processing from the timing of persistence. This intermediary layer reduces overhead by batch processing and asynchronous writes rather than requiring synchronous persistence for each transaction.

Inventive Principle:
Principle #24Intermediary (Mediator)

3Productivity

If the target database mirrors the source database snapshot, then data availability for analysis is improved, but recovery complexity increases after crashes

Engineering Contradiction:
Improvedata availabilityVSAvoidrecovery complexity
Core Design Contradiction:
ProductivityVSDevice complexity

Solution Approach 1:

The patent performs preliminary action by continuously maintaining an in-memory target database portion with recent changes from the source database. This pre-prepared in-memory buffer enables fast recovery after crashes because the system already has recent data ready in memory, requiring only the application of recovered log entries rather than full reconstruction. The in-memory portion serves as a preliminary recovery artifact that simplifies the recovery process.

Inventive Principle:
Principle #10Preliminary action

Solution Approach 2:

The patent changes the state parameter of the target database by introducing an in-memory portion that exists in a different state than traditional persistent storage. This in-memory state allows for faster recovery operations because memory operations are significantly faster than disk I/O. The system leverages the parameter change from persistent to volatile storage to reduce recovery time and complexity while maintaining data availability through the duality of in-memory and persistent portions.

Inventive Principle:
Principle #35Parameter changes

Data Source

PatentUS11797570B2Asynchronous persistency of replicated data changes in a database accelerator
Publication Date: 2023.10.24 INTERNATIONAL BUSINESS MACHINE CORPORATION
  • US11797570B2 patent drawing
  • US11797570B2 patent drawing
  • US11797570B2 patent drawing

AI summary

A computer-implemented method for a crash recovery for linked databases may be provided. The linked databases comprise a source and related target database. Selected queries of the source database are transferred to the target database. The method comprises synchronizing selected portions of the source database with tables of an in-memory portion of target database and, storing persistently applied changes to the in-memory target database portion asynchronously and persistently. Upon a database crash of the target database system, the method also comprises restoring, the in-memory target database portion with the latest snapshot available, and applying, changes from the source database recovery log file that have a later timestamp than the latest snapshot available in the persistent target database storage of the in-memory target database portion.