Heterogeneous Data Sharing via Capture-Apply Streaming
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for sharing information between heterogeneous data systems are inefficient, as they often require complex queue management, latch contentions, and are limited to systems that understand SQL statements, making it difficult to achieve robust and efficient integration across diverse hardware and operating systems.
Innovation Solution
A direct communication model between a capture process and an apply process, where the capture process mines log information at a source entity and transfers it to the apply process at a sink entity, allowing for direct exchange of data change information and overall state, eliminating the need for volatile queues and enabling efficient transaction log record streaming and recovery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a transparent gateway is used to pass SQL statements between heterogeneous data systems, then compatibility between different systems is improved, but processing efficiency deteriorates due to reconstruction overhead and mandatory roundtrips
Solution Approach 1:
The patent introduces a gateway as an intermediary component that translates between different data system protocols. The gateway receives data change information from the source system, converts it to the target system's format, and forwards it to the sink system. This mediator approach enables compatibility between heterogeneous systems while maintaining efficient direct streaming without requiring full SQL statement reconstruction and roundtrip communications.
2Adaptability or versatility
If a general purpose buffered queue is used to stage data in heterogeneous data systems, then adaptability between different systems is improved, but system complexity deteriorates due to latch contentions and complicated shared memory management
Solution Approach 1:
The patent extracts the queue management complexity from the data sharing mechanism by using separate, dedicated queues for each data stream direction. Instead of using a single shared buffered queue that requires complex latch management and memory allocation, the system employs independent queues that eliminate contention. This extraction of the queuing function into dedicated structures simplifies the overall system while maintaining adaptability between heterogeneous data systems.
Data Source
AI summary
In accordance with an embodiment of the present invention, a heterogeneous data sharing mechanism comprising a capture process and an apply process may be provided in a database system. The capture process mines or receives, at the source entity, a plurality of records that belong to a plurality of transactions. Here, first records from a first transaction in the plurality of transactions are interleaved with one or more records from one or more different transactions in the plurality of transactions. The capture process orders the plurality of records into a stream of ordered records such that all records that belong to a transaction are stored contiguously in a single block of records in the stream. Subsequently, the capture process sends the stream of ordered records from the source entity to the sink entity. Alternatively, the apply process retrieves the stream of ordered records from the capture process.