Asynchronous Replication Index for In-Scope Table Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current asynchronous replication systems face latency issues due to inefficient data scraping methods, where all redo records are scanned regardless of whether they contain changes for in-scope or out-of-scope tables, wasting time and computing resources.
Innovation Solution
Incorporating a replication index in the transaction log file header to track redo records associated with in-scope tables, allowing only those records to be scraped, and updating metadata or flags within redo records to indicate their relevance, thereby reducing unnecessary data scraping.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all redo records are scanned to identify changes for replication, then data consistency is ensured, but latency increases and computing resources are wasted
Solution Approach 1:
The patent applies preliminary action by pre-populating a transaction log index with metadata about redo records during transaction commit. This index is built in advance before replication needs to occur, allowing the system to quickly identify which redo records contain changes to in-scope tables without scanning all records during replication, thus reducing latency while maintaining data consistency
Solution Approach 2:
The patent extracts only the necessary information (metadata indicating changes to in-scope tables) from the transaction log and stores it separately in an index structure. This allows the replication process to selectively access only relevant redo records by querying the index, rather than scanning the entire transaction log, thereby reducing time loss while ensuring all necessary changes are captured for data consistency
2Reliability
If all redo records are scanned regardless of scope, then no changes are missed, but computing resource usage increases
Solution Approach 1:
The system performs preliminary action by pre-processing transaction logs to create an index that marks which redo records contain changes to in-scope tables. This upfront work allows subsequent replication operations to efficiently query the index and process only relevant records, reducing computing resource usage while maintaining complete capture of all necessary changes through the index's comprehensive tracking
Solution Approach 2:
The patent extracts metadata from redo records that indicates whether they contain changes to in-scope tables, and stores this information separately in a transaction log index. This extraction allows the system to identify and process only the necessary redo records during replication, reducing computing resource consumption while ensuring no relevant changes are missed through the index's complete coverage
3Volume of moving object
If transaction log files are archived in storage, then space is freed, but access time increases
Solution Approach 1:
The patent introduces a transaction log index as an intermediary between the archived transaction log files and the replication process. This index remains in memory and contains metadata that allows quick identification of relevant redo records in archived files, enabling the system to access archived logs efficiently without requiring the entire log file to remain in memory, thus freeing space while minimizing access time
Solution Approach 2:
The system performs preliminary action by maintaining a transaction log index in memory that pre-processes and organizes information from archived transaction logs. This index allows the replication process to quickly determine which archived log files and specific redo records need to be accessed, reducing access time while allowing the majority of log data to be archived and stored offline, freeing up memory space
Data Source
AI summary
A computer-implemented method includes writing a plurality of change records to a first set of redo records in a first transaction log file. The computer-implemented method further includes determining, while writing a change record to a redo record, that the change record includes changes made to data associated with an in-scope table. The computer-implemented method further includes updating the first transaction log file with information indicating that the redo record includes changes made to data associated with the in-scope table. The computer-implemented method further includes accessing the first transaction log file from storage in response to receiving a request for replicating changes made to data associated with in-scope tables. The computer-implemented method further includes scraping a first redo record in the first transaction log file in response to determining that the first redo record includes at least one change record associated with the in-scope table.