Database Synchronization Using Compound Key Verification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Databases face challenges in maintaining data integrity during backup and recovery processes, leading to discrepancies and incorrect results in reporting and restoration, especially when using NoSQL databases that do not adhere to traditional transaction rules.
Innovation Solution
A method for verifying and correcting database backups by generating compound keys from source and target database records, comparing them to identify discrepancies, and updating the target database with only necessary records, either in batch or real-time, while handling NoSQL databases' unique characteristics.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all database records are transferred during backup or update processes, then data integrity is ensured, but bandwidth usage and resource consumption increase significantly
Solution Approach 1:
The patent extracts only the changed records from the source database using change data capture mechanisms, rather than transferring all records. This selective extraction maintains data integrity while significantly reducing bandwidth consumption during backup and synchronization operations.
Solution Approach 2:
The system performs preliminary identification of changed records using timestamps and change tracking before the actual transfer process. This preliminary action allows the system to prepare a minimal dataset for transfer, ensuring completeness while optimizing resource usage.
2Measurement precision
If the entire database is reloaded during verification, then data accuracy is confirmed, but system productivity and time consumption worsen
Solution Approach 1:
The verification process is segmented to operate on individual changed records rather than the entire database. The system divides the verification task into discrete units corresponding to specific changed records, allowing parallel processing and significantly improving productivity while maintaining verification accuracy.
Solution Approach 2:
The system performs partial verification by checking only the changed records that require validation, rather than verifying the entire database. This partial action approach maintains data accuracy for critical records while avoiding the excessive time and resource consumption of full database verification.
3Loss of information
If duplicate records are not filtered during transfer, then all changes are captured, but data quality deteriorates with duplicate entries
Solution Approach 1:
The system implements feedback mechanisms using unique identifiers and timestamps to detect and filter duplicate records during the transfer process. Changed records are tracked and compared against existing records in the target database, with feedback loops that prevent duplicate insertions while ensuring all legitimate changes are captured.
Solution Approach 2:
The system uses parameter changes in record identifiers (such as timestamp modifications and version numbers) to distinguish between updated records and duplicate entries. By tracking these parameter changes, the system can identify genuine updates versus duplicates, maintaining data quality while capturing all necessary changes.
Data Source
AI summary
A system and method to ensure the consistency of a data warehouse or backup database with a source database are described. The method alleviates issues of comparing two sets of the same data on disparate network systems and eliminates having to reload the entire target database or compare every field to ensure reasonable consistency of the contents. The process involves loading a unique record identifier, an optional record change timestamp, and an optional record archive field of a source database into a work file or temporary database table. Source work file records or temporary database tables records that do not exist in the target database or have timestamp mismatches are retrieved from the source database and added to or updated in the target database. Target database records that are archived or missing in the work file or temporary database table are archived or deleted from the target database.