Database Block Restoration via Remote Key-Value Backup
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current database systems require restoring the entire data set before resuming operations after a failure, leading to significant downtime and costs, especially in large data warehouse systems that perform incremental backups or rely on transaction logs for reconstruction.
Innovation Solution
Implementing a distributed data warehouse system that uses a remote key-value durable storage system for incremental backups, allowing for selective restoration of data blocks on demand, enabling the system to continue accepting queries without waiting for the entire data set to be restored, by streaming in backup copies of lost or corrupted data blocks from the remote storage system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the entire data set is restored from backup before resuming operations, then data integrity is ensured, but system downtime increases significantly
Solution Approach 1:
The patent divides the data restoration process into individual block-level operations rather than restoring the entire data set at once. Each corrupted block is identified and restored independently from backup storage, allowing the system to resume operations with intact blocks while recovering damaged ones progressively.
Solution Approach 2:
The system performs partial restoration by recovering only the specific corrupted blocks that are identified through checksum verification, rather than restoring the complete data set. This selective approach minimizes restoration time while ensuring data integrity for the blocks that actually need recovery.
2Productivity
If incremental backups are used to reduce backup time, then backup efficiency improves, but restoration complexity increases due to multiple restore operations
Solution Approach 1:
The system pre-calculates and stores checksums for all data blocks during the backup process. This preliminary action enables quick identification of corrupted blocks during restoration without requiring complex analysis of incremental backup chains, simplifying the restore operation while maintaining backup efficiency.
Solution Approach 2:
The system uses checksum verification as feedback to identify which specific blocks are corrupted and need restoration. This feedback mechanism allows the system to selectively restore only the necessary blocks from incremental backups, reducing restoration complexity compared to restoring entire incremental chains.
3Reliability
If transaction logs are traced to reconstruct system state, then data accuracy is maintained, but restoration time increases significantly
Solution Approach 1:
The patent extracts and verifies data block integrity using pre-stored checksums independent of transaction log analysis. By separating the integrity verification process from transaction log tracing, the system can quickly identify corrupted blocks without performing time-consuming log replay operations to reconstruct system state.
Data Source
AI summary
A distributed data warehouse system maintains data blocks on behalf of clients, and stores primary and secondary copies of data blocks on different disks or nodes in a cluster. The data warehouse system may back up data blocks in a key-value backup storage system. In response to a query targeting a data block previously stored in the cluster, the data warehouse system may determine whether a consistent, uncorrupted copy of the data block is available in the cluster (e.g., by applying a consistency check). If not (e.g., if a disk or node failed), the data warehouse system may automatically initiate an operation to restore the data block from the backup storage system, using a unique identifier of the data block to access a backup copy. The target data may be returned in a query response prior to restoring primary and secondary copies of the data block in the cluster.