Column Database Synchronization via Consolidated Batch Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The inefficiency in updating column-oriented databases from row-oriented sources due to the need for breaking insert batches into multiple batches separated by delete operations, leading to increased overhead and processing time.
Innovation Solution
The method involves eliminating the need to intersperse update and delete activities with insert operations by organizing changes into consolidated batches, allowing all inserts to be moved in one batch and updates/deletes in separate batches, and pre-converting changes to a column-oriented format for efficient merging with the database.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If insert operations are interspersed with update and delete operations in batch processing, then the column-oriented database can be kept synchronized with the row-oriented source database, but the number of batch moves increases and processing overhead increases
Solution Approach 1:
The patent segments the batch processing into distinct phases: first processing all insert operations in a consolidated batch, then processing update and delete operations in separate batches. This segmentation eliminates the need to intersperse different operation types, reducing the total number of batch moves and processing overhead while maintaining synchronization consistency.
Solution Approach 2:
The patent applies preliminary action by pre-converting row-oriented changes to column-oriented format before batch processing. This pre-processing step organizes the data in the target format beforehand, so that when batches are moved and applied to the column-oriented database, the operations are more efficient and require fewer intermediate steps.
2Productivity
If all insert operations are processed in a single consolidated batch, then processing overhead is reduced and efficiency is improved, but the complexity of managing batch operations increases
Solution Approach 1:
The patent segments batch management into distinct phases: first handling all inserts in a consolidated batch, then handling updates and deletes in separate batches. This segmentation simplifies the management complexity by creating clear, sequential batch processing stages rather than requiring complex interleaved management of multiple operation types simultaneously.
Solution Approach 2:
The patent applies preliminary action by pre-organizing insert operations into a consolidated batch structure before processing. This pre-batching of insert operations simplifies the subsequent processing step, as the system only needs to manage a single consolidated insert batch rather than multiple individual insert operations, reducing the overall management complexity.
Data Source
AI summary
An automated method and computer program product are provided for synchronizing a column-oriented target database with a row-oriented source database. Change data are replicated from a change log of the row-oriented source database via a staging database to the column-oriented target database. The change data including inserts and deletes. Change data of the change log is read into the staging database and is consolidated and grouped into a consolidated grouping of inserts, and a consolidated grouping of deletes. The consolidated grouping of inserts from the staging database are applied to the target database in a batched manner, and the consolidated grouping of deletes from the staging database are applied to the target database in a batched manner.