Parallel Access Plan Delimitation for Database Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large databases face challenges in replicating tables efficiently due to extended transfer times when dealing with large datasets, as existing methods require sequential processing of partitions, leading to prolonged processing demands and potential failures.
Innovation Solution
Implementing parallel processing for delimitation calculations across partitions, allowing for simultaneous determination of access plan boundaries and initiation of data transfer as soon as each partition is processed, using a delimitation processor and access plan processor to coordinate the transfer of data from a source system to a target system.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If sequential processing of table partitions is used for initial load, then processing demands during transfer are reduced, but transfer time is extended
Solution Approach 1:
The patent performs delimitation calculation for each partition in advance, before the actual data transfer begins. By pre-calculating the delimitation information (such as row counts and boundary definitions) for all partitions, the system prepares transfer metadata ahead of time, enabling parallel transfer execution without requiring sequential processing. This preliminary action resolves the contradiction by allowing parallel transfers (reducing time) while maintaining controlled processing demands through pre-computed partition metadata.
Solution Approach 2:
The patent divides the initial load process into independent partition segments that can be processed and transferred in parallel. Each partition is treated as an independent unit with its own delimitation calculation and transfer operation. This segmentation enables multiple partitions to be transferred simultaneously rather than sequentially, significantly reducing total transfer time while maintaining manageable processing demands through division of the large table into smaller, independent partition units.
2Device complexity
If partitioning is implemented as a single process, then processing demands are controlled, but transfer cannot occur until partitioning is complete
Solution Approach 1:
The system performs delimitation calculation as a preliminary step that is independent of the actual data transfer operation. By calculating delimitation information for each partition beforehand and storing it as metadata, the system decouples the partitioning preparation from the transfer execution. This allows transfer to begin immediately using the pre-calculated delimitation data, eliminating the waiting time while keeping the delimitation calculation as a separate, controlled process.
Solution Approach 2:
The patent separates the delimitation calculation function from the data transfer function, creating independent processing segments. The delimitation calculation is performed once per partition to generate metadata, while the actual transfer operates independently using this metadata. This functional segmentation allows both operations to proceed in parallel without one blocking the other, reducing total time while maintaining controlled complexity through clear separation of concerns.
3Loss of time
If parallel processing of delimitation calculations is implemented across partitions, then data transfer time is reduced, but processing complexity increases
Solution Approach 1:
The patent divides the delimitation calculation task into independent segments, one for each partition, that can be executed in parallel. Each partition's delimitation calculation is self-contained and does not depend on other partitions, enabling simultaneous processing. The system manages this parallel processing by treating each partition as an independent unit with its own calculation thread or process, reducing overall time while controlling complexity through modular, independent task design.
Solution Approach 2:
The system introduces an intermediary layer of partition metadata that mediates between the parallel delimitation calculations and the final transfer operation. Each partition's delimitation calculation produces metadata that is stored and then used by the transfer mechanism. This intermediary metadata layer allows multiple parallel calculations to converge without direct interaction, simplifying the management of parallel processing while enabling simultaneous execution across partitions.
Data Source
AI summary
Disclosed herein are system, method, and computer program product embodiments for replicating data from table in a source database to a target database. In some embodiments, data replication includes access plan delimitation and access plan calculation steps and is performed on a table having multiple partitions. A table may be divided into one or more partitions and each partition may be further divided into one or more access plans. Access plan delimitation may involve calculating, in parallel, boundaries of access plans within partitions of the table. Access plan calculation may be initiated on the first partition that has completed the access plan delimitation steps, and may involve transferring data from each delimited partition from the table in the source database to the target database.