Target Database Partition Loading With Adaptive Replication Modes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Controlling the time required for data loading in database systems is challenging, particularly in synchronizing data between source and target database systems, which affects overall performance.
Innovation Solution
A method involving multiple synchronization programs, including a load program, replication program, and update program, with a mechanism to switch between default and secondary modes of operation for the replication program based on the completion status of the update program, ensuring data consistency and reduced latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a single synchronization program is used for data loading, then the system structure is simple, but the data consistency and replication latency cannot be optimized
Solution Approach 1:
The patent divides the data synchronization process into three distinct programs: a load program for bulk data loading, a replication program for real-time replication, and an update program for maintaining data consistency. This segmentation allows each program to specialize in specific tasks, improving data consistency and replication latency while maintaining manageable system complexity through clear separation of concerns.
Solution Approach 2:
The patent implements dynamic switching between default and secondary modes of operation for the replication program based on the completion status of the update program. This dynamic adaptation allows the system to optimize replication latency by selecting the appropriate mode (synchronous or asynchronous) according to real-time conditions, thereby improving reliability without requiring a fixed complex structure.
2Productivity
If data loading is performed quickly, then productivity is improved, but controlling the time required and ensuring data consistency becomes challenging
Solution Approach 1:
The patent uses a query view to pre-identify and mark the last loaded partition before initiating new data loading operations. This preliminary action allows the system to track data loading progress and manage partition states efficiently, enabling fast data loading while maintaining precise time control and ensuring that already loaded partitions remain visible and accessible.
Solution Approach 2:
The patent implements a feedback mechanism where the query view continuously monitors the completion status of the update program and the current state of target partitions. This feedback enables the replication program to dynamically adjust its operation mode based on real-time conditions, optimizing data loading speed while maintaining data consistency and controlling loading time.
3Reliability
If the replication program operates in synchronous mode, then data consistency is maintained, but replication latency increases
Solution Approach 1:
The patent enables the replication program to dynamically switch between synchronous and asynchronous modes based on the completion status of the update program. When the update program is complete, the system can operate in asynchronous mode to reduce replication latency; when the update program is still running, it operates in synchronous mode to maintain data consistency. This dynamic approach resolves the contradiction between consistency and latency.
Solution Approach 2:
The patent changes the operational parameters of the replication program (synchronous vs. asynchronous mode) based on the state of the update program. This parameter change allows the system to adapt to different conditions, maintaining data consistency when necessary while minimizing replication latency when possible, thereby resolving the contradiction between reliability and time loss.
4Speed
If partitions are made visible immediately after loading, then access speed is improved, but query accuracy may be compromised
Solution Approach 1:
The patent introduces a query view as an intermediary layer between the raw data partitions and the query processing system. This query view selectively exposes only the partitions that have been successfully loaded and validated, making them immediately accessible for fast data access while ensuring that queries operate on complete and accurate data sets. The query view acts as a filter that maintains query accuracy while enabling rapid access to available data.
Data Source
AI summary
A computer implemented method includes identifying a query view of a target database system enabling access to a first source table stored in a first target table, executing a load program for a current version of a first source partition of the first source table, wherein executing the load program includes loading the first source partition in a second load target partition of the first target table, changing the query view to select records of the first target table, executing an update program for assigning records of the first replication target partition to the first load target partition, selecting a mode of operation for a replication program based on determining whether the execution of the update program is finished, executing the replication program in accordance with the selected mode of operation, and changing the query view to further select the records of the first replication target partition.