Continuous Data Protection via Transaction Log Application
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database backup solutions are inefficient, taking too much time, interrupting regular workflows, and providing limited options for restoring data to a specific point-in-time, while also being costly and lacking a managed experience.
Innovation Solution
A continuous data protection system that captures all changes on a data store and builds system snapshots by applying transaction logs to previous snapshots, allowing for point-in-time logical backups without interrupting production workloads.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If scan-based backup solutions are used to create backups of large databases, then backup completeness is improved, but backup time increases significantly and production workflow is interrupted
Solution Approach 1:
The system performs preliminary actions by continuously capturing and logging data changes in real-time through change data capture (CDC) mechanisms. Transaction logs are continuously written and maintained, so when a backup is needed, the system can immediately restore from these pre-captured logs without performing a time-consuming scan of the entire database at backup time.
Solution Approach 2:
Instead of copying entire databases through scanning, the system creates compact representations by capturing only the essential change data through transaction logs. These logs serve as efficient copies that contain all necessary information for restoration, dramatically reducing backup time and resource consumption while maintaining completeness.
2Reliability
If periodic backups are performed by scanning the data store, then data recovery capability is provided, but the time between backups prevents restoration to specific point-in-time
Solution Approach 1:
The system implements continuous data capture through transaction logging, maintaining an unbroken chain of change records from the initial database state. This continuous logging ensures that every data modification is captured in chronological order, enabling restoration to any specific point-in-time within the retention period, not just at periodic intervals.
Solution Approach 2:
The system performs preliminary continuous logging of all data changes, maintaining a complete transaction history before any restoration operation is needed. This pre-captured transaction log enables flexible point-in-time restoration by allowing the system to replay transactions up to any desired moment, providing both reliability and temporal flexibility.
3Measurement precision
If scan-based backup solutions are used to ensure data protection, then backup accuracy is improved, but compute costs and storage costs increase significantly
Solution Approach 1:
The system creates efficient change data captures by logging only the essential modifications rather than scanning and copying entire datasets. Transaction logs record precise change information (what changed, when, and to what state), providing accurate backup data with minimal storage requirements and compute resources compared to full database scans.
Solution Approach 2:
The system performs preliminary continuous logging of data changes at the transaction level, capturing precise change information as it occurs. This approach ensures backup accuracy by recording exact state transitions while minimizing compute and storage costs, as the logs contain only the necessary change data rather than requiring expensive full-database scans.
4Reliability
If full database scans are performed for backups, then complete data coverage is achieved, but production workload resources are tied up
Solution Approach 1:
The system extracts and captures data changes directly from the transaction log, separating the backup function from production database operations. This extraction approach ensures complete data coverage by capturing all changes at the source while allowing production workloads to continue uninterrupted, as the logging mechanism operates independently without blocking or scanning production data.
Solution Approach 2:
The system performs preliminary continuous capture of data changes through transaction logging, ensuring complete data coverage is achieved in real-time as changes occur. This preliminary action allows production workloads to proceed at full capacity without being interrupted by backup scans, as the change capture happens concurrently and non-invasively.
Data Source
AI summary
Changes made to a database table are accumulated, in durable storage, and snapshots of partitions of the table are obtained. For successive snapshots of a partition, the system accesses a previous snapshot, applies changes from the accumulated changes, and stores the updated snapshot to a durable data store. The accumulated changes and the successive partition snapshots are made available to restore the database to any point in time across a continuum between successive snapshots. Although each partition of the table may have a backup snapshot that was generated at a time different from when other partition snapshots were generated, changes from respective change logs may be selectively log-applied to distinct partitions of a table to generate an on-demand backup of the entire table at common point-in-time across partitions. Point-in-time restores of a table may rely upon a similar process to coalesce partition snapshots that are not aligned in time.