Partial Data Reload via Reference Partition Attachment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current partial reloading methods in data synchronization can impact query performance and require significant time, especially as data volume grows, and may necessitate stopping queries to delete outdated data, causing disruptions and performance issues.
Innovation Solution
The method involves loading a source partition from a source database into a target database as a separate reference partition, which is then attached to the target data, allowing for concurrent query execution without commit impacts and minimizing reloading time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If partial reloading is performed by deleting outdated data and loading new data into the target database, then data synchronization is achieved, but query performance is impacted and downtime occurs
Solution Approach 1:
The patent segments the target database into multiple partitions, allowing selective reloading of individual partitions without affecting the entire database. This enables parallel query execution on unaffected partitions while reloading is in progress, thus maintaining query performance during data synchronization operations.
Solution Approach 2:
The patent extracts the reloading operation from the main target database by using temporary staging tables. Data is first loaded into these separate staging tables, then swapped with the original partitions using table exchange operations. This isolation prevents reloading from impacting ongoing queries on the main database.
2Reliability
If partial reloading is performed by deleting outdated data, then data synchronization is achieved, but significant time is required and disruptions occur
Solution Approach 1:
The patent performs preliminary data validation and preparation in staging tables before the actual reload operation. This includes checking data integrity, transforming data formats, and pre-processing validations, so that when the final swap occurs, it can be executed quickly with minimal disruption to production systems.
Solution Approach 2:
The patent uses table exchange operations that atomically swap the staging table with the target partition in a single operation. This bypasses the time-consuming process of incremental data insertion and allows the reload to complete rapidly, minimizing the time the system is in a transitional state.
3Reliability
If traditional partial reloading methods are used, then data updates are achieved, but commit impacts affect concurrent query execution
Solution Approach 1:
The patent introduces temporary staging tables as intermediary structures between the source and target databases. These staging tables serve as a buffer zone where data can be prepared and validated without affecting the main target database. The final table exchange operation then atomically replaces the partition, ensuring data updates are achieved without committing partial changes that would block queries.
Data Source
AI summary
Partial reloading of source data in data synchronization. Determining whether a source partition of source data in a source database is to be reloaded into target data in a target database, the source partition corresponding to a target partition of the target data. In accordance with a determination that the source partition is to be reloaded, one or more processors determine to load the source partition from the source database into the target database as a reference partition, the reference partition being separated from the target data in the target database and being assigned with a reference partition identity. In accordance with a determination that the loading of the source partition is completed, one or more processors determine to attach the reference partition to the target data.