In-stream Data Load via Extract Utility Injection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data replication methods face challenges such as exclusive access requirements to the source data, inefficient data access, and verification difficulties, particularly in ensuring data consistency and handling large data sources, which can impact availability and performance.
Innovation Solution
A computer-implemented method that uses an extract utility separate from the replication engine to read data from a source data store and inject records into the replication log, allowing for in-stream data load and refresh, enabling parallel processing, active load, and efficient data validation while avoiding the need for exclusive access and optimizing data access methods.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If an internal refresh is used where the replication product accesses source data and sends it to the target, then convenience for users is improved, but exclusive access requirements to the source data worsen availability
Solution Approach 1:
The patent introduces a replication log as an intermediary mechanism. Instead of the replication product directly accessing source data (internal refresh) or requiring external tools to access source data (external refresh), the solution uses a log that records changes to source data. The replication product reads from this log without needing exclusive access to the source data store, thus maintaining user convenience while eliminating availability issues.
2Adaptability or versatility
If an external refresh is used where the user controls data transfer and formatting, then flexibility is improved, but verification difficulty to ensure data consistency worsens
Solution Approach 1:
The patent implements a feedback mechanism where the replication log provides a standardized interface between the source data store and the replication product. The log records changes in a format that allows the replication product to verify data consistency automatically. This feedback loop ensures that the replication product can detect and measure data consistency without requiring complex user-controlled verification processes.
3Reliability
If traditional replication methods are used that require exclusive access to source data, then data consistency is improved, but processing efficiency during refresh worsens
Solution Approach 1:
The patent applies preliminary action by pre-recording changes to source data in a replication log before the replication process begins. Instead of accessing source data during replication (which requires exclusive access), the changes are already captured in the log in advance. This allows the replication product to process data efficiently without blocking source data access, maintaining both consistency and productivity.
4Ease of operation
If internal refresh is used where the replication product formats source data for the target, then ease of operation is improved, but device complexity of the replication product worsens
Solution Approach 1:
The patent extracts the data formatting and access logic from the replication product into a separate replication log. The replication product no longer needs to directly access source data or perform complex formatting operations, as these functions are separated into the log mechanism. This reduces the complexity of the replication product while maintaining ease of operation, as the log handles the complex interactions automatically.
Data Source
AI summary
In-stream data load in a replication environment includes extracting source data from a source data store that is in a replication relationship with a target data store. A replication facility replicates to a target system changes made to the source data store. The replication facility includes a replication engine that reads a replication log into which the replication facility records the changes and sends the changes to the target system for application to the target data store. The extracting is performed by an extract utility separate from the replication engine. The in-stream data load also includes injecting, by the extract utility, the extracted source data into the replication log in-line with the changes recorded in the replication log by the replication facility and according to a format such that the replication engine can read and send the source data to the target system.