Live Database Export Using Change Logs for Transactional Consistency

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Directly exporting a live database to another system can result in an exported database that lacks transactional consistency due to partially applied transactions, increasing latency and processing requirements.

Innovation Solution

Utilize change records to identify and apply unapplied and partially applied transactions, ensuring transactional consistency by generating a transactionally consistent copy of the live database without creating an internal copy first.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If a transactionally consistent copy of the live database is generated before export, then transactional consistency is maintained, but export latency and processing resource utilization increase

Engineering Contradiction:

Improvetransactional consistencyVSAvoidexport latency

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The system performs preliminary actions by recording change records (such as transaction logs or write-ahead logs) during database operations before export is needed. These change records capture all modifications to the live database in advance, allowing the export process to simply copy and apply these pre-recorded changes rather than generating a consistent copy in real-time, thereby reducing export latency while maintaining transactional consistency

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

Instead of creating a full transactionally consistent copy of the entire live database before export, the system copies only the essential change records (logs) that represent database modifications. This selective copying approach significantly reduces the data volume and processing resources required, while still enabling the generation of a consistent exported database by applying these change records to a snapshot of the live database

Inventive Principle:

Principle #26Copying

2Reliability

If a transactionally consistent copy of the live database is generated before export, then transactional consistency is maintained, but processing resource utilization increases

Engineering Contradiction:

Improvetransactional consistencyVSAvoidprocessing resource utilization

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The system extracts and separates the change recording function from the main database export process. By using independent change records (such as write-ahead logs or transaction logs) that are maintained separately from the live database, the system avoids the need to process and consistent-ify the entire live database before export. This extraction allows export operations to work with lightweight change records rather than heavy database copies, reducing processing resource utilization while maintaining transactional consistency

Inventive Principle:

Principle #2Taking out (Extraction)

Solution Approach 2:

The system copies only the minimal necessary data (change records/logs) rather than creating a complete consistent copy of the live database. These change records are typically stored in a compact, sequential format that requires far less processing power to copy and apply compared to full database consistency checks, thereby reducing processing resource utilization during export operations

Inventive Principle:

Principle #26Copying

3Ease of operation

If disk-based data storage is used on cloud instances, then data can be stored and accessed, but durability, reliability, and scalability issues arise

Engineering Contradiction:

Improvedata storage and accessVSAvoiddurability and reliability

Core Design Contradiction:

Ease of operationVSReliability

Solution Approach 1:

The system introduces change records (such as write-ahead logs or transaction logs) as an intermediary layer between the live database and the exported database. These change records serve as a mediator that captures all data modifications before they are applied to the main database, ensuring that even if disk-based storage fails or becomes corrupted, the change records preserve the transaction history needed to reconstruct a consistent database state, thereby improving durability and reliability

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system performs preliminary recording of all database changes in change records before these changes are committed to the main database storage. This preliminary action ensures that even if the disk-based storage fails, the pre-recorded changes in the change logs can be used to restore or reconstruct the database, enhancing durability and reliability of disk-based cloud storage

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS12481639B2Transactionally consistent database exports

Publication Date: 2025.11.25 RUBRIK INC

AI summary

Techniques for data management are described. A file may be saved that includes a representation of an exported database, where the representation may be based on a live database being exported from a computing system via an export operation that spans a time period. A determination that one or more transactions were committed to the live database during the time period may be made. Based on the determination, a log indicating the one or more transactions may be obtained from the computing system. Based on the log, a determination of whether the exported database captured the one or more transactions may be made. Based on the determination, the file may be updated to represent an updated database that reflects an updated version of the live database, the updated version of the live database reflection application of the one or more transactions to the exported database.