Continuous Verified Database Backups via Checksum Verification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for backing up large database systems with data distributed across multiple machines face challenges in efficiently catching bit-rot, stale data, and corruption, especially when data is stored in a native format that may have been corrupted over generations of software stacks, and require processing of petabytes of data upfront, which does not scale well and can impact real-time customer traffic.
Innovation Solution
The system creates verified snapshots and change stream archives by processing data and corresponding change logs separately with forward and reverse transformations, comparing results to the original data to catch corruptions, and uses error detection and handling mechanisms to ensure data integrity during format conversion and backup creation, allowing for continuous verified backups without impacting real-time operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is processed upfront to verify backups, then data integrity is improved, but processing time and system performance deteriorate
Solution Approach 1:
The system performs preliminary actions by creating checksums of source data before backup operations and storing these checksums with the backup metadata. This allows verification to be prepared in advance without blocking the backup process, resolving the contradiction between ensuring data integrity and minimizing processing time.
Solution Approach 2:
The verification process operates continuously in the background by comparing checksums of backup data against stored source checksums without interrupting the backup workflow. This continuous verification approach maintains data integrity checks while allowing the backup process to proceed without time loss.
2Manufacturing precision
If all backup data is verified upfront, then data accuracy is improved, but system productivity deteriorates
Solution Approach 1:
The verification system segments the backup verification process by organizing data into partitions and processing checksums at the partition level rather than requiring verification of the entire backup dataset. This segmentation allows parallel verification operations that maintain data accuracy while preserving system productivity through efficient resource utilization.
Solution Approach 2:
The system creates and stores checksums as separate verification artifacts that can be independently processed and compared. This copying approach allows verification operations to work with lightweight checksum data rather than the full backup data, ensuring data accuracy while minimizing the computational resources required and maintaining system productivity.
3Adaptability or versatility
If data is converted to non-native format for backup, then portability is improved, but risk of data transformation loss increases
Solution Approach 1:
The system performs preliminary checksum creation on the original native-format data before any format conversion occurs. These checksums serve as reference values for subsequent verification of the converted backup data, enabling detection of any transformation losses while maintaining the portability benefits of non-native format storage.
Solution Approach 2:
The verification mechanism provides feedback by comparing checksums of the converted backup data against the pre-computed checksums of the original data. This feedback loop detects any data transformation losses that may occur during format conversion, allowing the system to maintain both portability through format conversion and reliability through loss detection.
Data Source
AI summary
A system and technique for creating, in a non-native format, verified snapshots and change log archives for data in a database (e.g., tables, partitions, etc.). To verify accuracy of a conversion of the data and corresponding change log data from a native format to a non-native format, both data from the database and the corresponding change logs are processed separately with a forward transformation process, and then a reverse transformation process. The results of the reverse transformations are then compared to the original data to catch data corruptions or errors when performing the format conversion and creating the snapshot or change log archive so that the corruption or error is not propagated to the snapshot/archive. Various forms of error detection (e.g., byte-level, raw data comparisons, checksums, etc.) and error handling are disclosed. The verified snapshots and change log archives may be used to restore the database, for example.