Database Data Replication via State Caching and Buffering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data replication methods in database technologies, particularly in OLAP systems and big data analysis, face inefficiencies due to the complexity and time-consuming nature of parsing and replaying redo logs, which affects the speed and efficiency of data replication.
Innovation Solution
A data replication method that involves caching historical state data in a data queue and transmission buffer, and replicating it to a cluster device when specific conditions are met, eliminating the need for format conversion and log replay, thereby streamlining the replication process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data replication uses redo log parsing and replaying to ensure consistency, then data consistency between primary and secondary devices is improved, but replication time and processing complexity increase significantly
Solution Approach 1:
The patent applies preliminary action by pre-processing and filtering data before replication. The system identifies and replicates only the changed data portions (insertions, deletions, modifications) rather than entire tables, and prepares this data in advance to minimize processing time during actual replication operations.
Solution Approach 2:
The patent extracts only the essential changed data from the complete database state. By using incremental replication mechanisms and change tracking, the system separates and replicates only the necessary data portions, eliminating the need to process entire redo logs and reducing both time and complexity.
2Reliability
If data replication processes entire data files periodically, then data synchronization is ensured, but replication efficiency decreases due to processing large volumes of data
Solution Approach 1:
The patent segments data replication into incremental units rather than processing entire files. By dividing data into changed portions and replicating only these segments, the system maintains synchronization while significantly improving processing efficiency and reducing the impact on business operations.
Solution Approach 2:
The system performs preliminary identification and categorization of changed data before replication begins. This pre-processing allows the system to prepare replication tasks efficiently and execute them with minimal overhead, maintaining both synchronization and high productivity.
3Reliability
If redo log parsing is performed to prevent data inconsistency, then data integrity is maintained, but the complexity of the replication process increases
Solution Approach 1:
The patent extracts and processes only the critical change information from redo logs rather than parsing entire log structures. By focusing on extracted data changes (insertions, deletions, modifications) rather than complete log replay, the system maintains data integrity while substantially reducing processing complexity.
Solution Approach 2:
The system creates simplified representations of data changes rather than copying entire redo log structures. This approach preserves the essential integrity information while using simpler data structures and processing mechanisms, reducing overall system complexity.
Data Source
AI summary
This application discloses a data replication method and apparatus, a computer device, and a storage medium, and belongs to the field of database technologies. The method includes: adding, when a commitment operation of a transaction is detected, historical state data of the transaction to a data queue; adding at least one piece of historical state data in the data queue to a transmission buffer; and replicating, when a first preset condition is met, the at least one piece of historical state data in the transmission buffer to a cluster device. In this application, a node device does not need to convert an original historical state data format into a log format, and the cluster device does not need to parse a log into an original data format for storage, thereby avoiding a cumbersome log replay procedure, and improving the efficiency of the data replication process.