Partitioned Data Synchronization for Concurrent Store and Metadata Updates
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data synchronization methods in data analysis systems involving data stores and metadata stores are inefficient, leading to high resource usage and suboptimal performance, particularly in cloud-based environments, and lack scalability and flexibility in handling large-scale data ingestion.
Innovation Solution
A method for data synchronization that involves splitting transactions into subsets of change records, executing these subsets concurrently to adapt both the data store and metadata store, allowing for parallel processing and efficient resource utilization, with mechanisms for versioning and query handling to ensure data integrity during synchronization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional sequential data synchronization methods are used, then data integrity is maintained, but data ingestion throughput is low and resource usage is high
Solution Approach 1:
The patent divides a main transaction into multiple sub-transactions, where each sub-transaction processes a subset of change records for a specific partition. This segmentation allows parallel execution of sub-transactions across different partitions, significantly improving data ingestion throughput while maintaining data integrity through isolated processing units.
Solution Approach 2:
The patent introduces a new dimension of parallel processing by executing multiple sub-transactions concurrently across different partitions simultaneously. This transforms the traditional sequential single-threaded data synchronization into a multi-dimensional parallel processing model, enhancing throughput without proportionally increasing resource consumption.
2Productivity
If data synchronization is performed without partitioning, then implementation is simpler, but scalability is limited and performance is suboptimal for large-scale data
Solution Approach 1:
The patent segments the data synchronization process into partition-specific sub-transactions, where each partition can be processed independently. This segmentation enables the system to scale horizontally by adding more partitions and processing units, improving both throughput and scalability while managing complexity through modular design.
Solution Approach 2:
The patent implements dynamic transaction processing where the main transaction automatically decomposes into multiple sub-transactions based on partition identification. This dynamic approach allows the system to adapt to varying data volumes and partition configurations, enhancing scalability without requiring complex static configuration.
3Productivity
If concurrent processing is implemented without proper versioning, then throughput improves, but data integrity and query accuracy deteriorate
Solution Approach 1:
The patent performs preliminary actions by creating versioned snapshots of the data store and metadata store before executing sub-transactions. This preliminary versioning ensures that concurrent processing does not compromise data integrity, as each sub-transaction operates on a consistent baseline and changes are tracked through version control mechanisms.
Solution Approach 2:
The patent implements feedback mechanisms where the system monitors the execution status of sub-transactions and manages versioning accordingly. This feedback loop ensures that data integrity is maintained during concurrent processing by coordinating updates across partitions and resolving any conflicts through version management protocols.
Data Source
AI summary
The present disclosure relates to a method. A main transaction may be received. The main transaction comprises a set of change records of the at least one source table. Each change record represents at least one data record of a source table of the at least one source table. A least one subset of change records of the set of change records may be determined. Each subset of change records is determined such that it represents a distinct source partition of the at least one source table. A sub-transaction may be determined per subset of change records of the at least one subset of change records. The main transaction may be executed by at least concurrently executing the set of sub-transactions, wherein the execution of each sub-transaction of the sub-transactions comprises concurrently adapting a data store and adapting a metadata store in accordance with the sub-transaction.