Transaction Consistency in Replicated Data via Change Data Table
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing solutions for parallel replication of transactions in database systems face challenges in maintaining transaction consistency, leading to inconsistent data in target data stores, especially under heavy workloads, which is unacceptable for business-critical applications.
Innovation Solution
A method that uses a change data table to store information from a recovery log, where each entry identifies transactional changes and completion indicators, allowing for the retrieval of records with transaction consistency by filtering out records with sequence identifiers larger than a maximum commit sequence identifier, ensuring that only committed transactions are returned.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If transactions are replicated in parallel through multiple channels, then throughput and performance are improved, but transaction consistency cannot be maintained at target data stores
Solution Approach 1:
A mediator component is introduced between the source and target data stores that receives parallel replicated transactions, determines their dependency relationships, and serializes them in the correct order before applying to the target. This mediator preserves transaction consistency while allowing parallel replication channels to operate independently, thus maintaining high throughput without sacrificing reliability.
Solution Approach 2:
The system performs preliminary analysis of transaction dependency relationships before applying transactions to the target data store. By pre-determining the correct serialization order of parallel transactions based on their dependency graphs, the system ensures that consistency is maintained from the outset rather than being violated and then corrected, thereby preserving both throughput and consistency.
2Reliability
If transactions are serialized to maintain original order, then transaction consistency is preserved, but parallelism and performance are lost
Solution Approach 1:
The system segments transactions into independent groups based on their dependency relationships. Transactions that do not depend on each other are placed in different segments and can be applied in parallel to the target data store. Only transactions within the same dependency chain are serialized. This segmentation allows the system to maintain consistency for dependent transactions while maximizing parallelism for independent ones, thus preserving both reliability and productivity.
3Productivity
If workload is broken up by tables or finer granularity for performance, then parallelism is improved, but transaction consistency is violated
Solution Approach 1:
The system introduces a new dimension of organization by creating a global transaction dependency graph that spans across all tables and fine-grained operations. This global view allows the system to determine which fine-grained operations from different tables can be executed in parallel while maintaining overall transaction consistency. By adding this global coordination layer, the system can safely break down workloads by table while preserving consistency through the dependency graph-based serialization mechanism.
Data Source
AI summary
Transaction consistency query support is available for replicated data from recovery log to external data stores. An external data store is populated with records using entries of a change data table. The change data table has entries for each transaction that has committed and is to be replicated, and each of the entries stores information for each log entry in a recovery log from a database management system. Each log entry identifies a transactional change of data and a transaction completion indicator of one of commit and abort. In response to receiving a query about a transaction of the transactions, a set of records are retrieved from the external data store for the transaction. From the set of records, records whose sequence identifier values are larger than a maximum transaction commit sequence identifier are removed. From the set of records, remaining records having transaction consistency are returned.