DBMS Transaction Replication via Open-Format Lakehouse Sync
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data lakehouse systems struggle to efficiently replicate real-time database management system transactions while preserving transaction consistency and durability, especially for mission-critical OLTP services like banking applications, due to high overhead and latency issues.
Innovation Solution
A computing system that utilizes open data formats and asynchronous replication to synchronize transactions from a DBMS to a data lakehouse, employing a Capture module, Apply module, and Open Format Builder to generate and manage open format files with timestamps, ensuring real-time replication and transaction integrity.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If real-time replication of DBMS transactions is implemented in data lakehouse, then transaction consistency and durability are improved, but system overhead and latency increase
Solution Approach 1:
The patent introduces a replication service as an intermediary component between the DBMS and data lakehouse. This service captures transactions from the DBMS, transforms them into open format files, and applies them to the data lakehouse tables. The intermediary architecture allows for efficient transaction replication while maintaining transaction consistency and durability without excessive overhead, as the replication service optimizes the data flow and transformation processes.
2Adaptability or versatility
If open data formats are used for transaction replication, then data flexibility and interoperability are improved, but data processing complexity increases
Solution Approach 1:
The patent transforms DBMS transactions into open data formats (such as Avro, Parquet, or ORC) by changing the data representation parameters. The replication service converts internal DBMS formats into standardized open formats that can be read by various analytics tools. This parameter transformation enables data format flexibility and interoperability while the systematic approach to conversion manages processing complexity through defined transformation rules and schemas.
3Productivity
If asynchronous replication is implemented, then replication throughput is improved, but transaction timing precision deteriorates
Solution Approach 1:
The patent implements preliminary actions by capturing transactions from the DBMS before they are fully processed, and by pre-transforming transactions into open format files before applying them to the data lakehouse. The replication service maintains a log of captured transactions and processes them in batches, allowing for asynchronous replication that improves throughput. Transaction timing precision is preserved through timestamp tracking and ordering mechanisms that ensure transactions are applied in the correct sequence despite asynchronous processing.
Data Source
AI summary
A computing system performs real-time replication of database management system transactions. The computing system includes a source DBMS, a replication service and a data lakehouse. The source DBMS stores at least one source transaction table recording at least one source transaction and generates at least one recovery log indicating at least one modification in the at least one source transaction table. The replication service replicates the at least one source transaction recorded in the at least one source transaction table by generating at least one data file having a first data format. The data lakehouse stores at least one lakehouse table corresponding to the at least one source transaction table and having a second data format different from the first data format, and to modify the at least one lakehouse table based on the at least one data file.