Streaming Change Data Capture via Hash Comparison
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The increasing volume of data in enterprises and private environments, coupled with the need for efficient data transmission and replication, poses challenges in transmitting data changes in real-time or periodically across different systems, especially when data is generated outside cloud computing systems or on separate networked resources, requiring efficient methods to manage data consistency without direct communication between producers and consumers.
Innovation Solution
A computer-implemented method that retrieves hash values from a streaming channel, compares them with newly generated hash values based on data portions, and transmits changes to a data consumer via a separate streaming channel, allowing for decoupled data replication without additional protocols or complex transaction logs, using messaging brokers like Kafka or IBM MQ to manage data transmission and storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data is transmitted in bulk to interested parties, then data availability is improved, but transmission efficiency and real-time updates deteriorate due to increasing data volumes
Solution Approach 1:
The patent extracts only the changed data portions from the bulk data and transmits them separately to consumers. Instead of transmitting entire data sets, the system identifies and sends only the delta (changes), reducing transmission volume while maintaining real-time efficiency.
Solution Approach 2:
The system performs preliminary actions by continuously monitoring data changes and pre-computing hash values to identify modifications. This allows the system to be ready to transmit changes immediately when they occur, improving transmission efficiency without waiting for bulk data requests.
2Device complexity
If direct communication between data producers and consumers is used, then data transmission is simple, but system complexity increases when data volumes and real-time requirements grow
Solution Approach 1:
The patent introduces a messaging broker as an intermediary component that sits between data producers and consumers. The broker handles change detection, hash value computation, and selective data transmission, simplifying the architecture for real-time updates while maintaining communication simplicity through standardized messaging protocols.
3Productivity
If change data capture is implemented to transfer changes, then real-time data updates are improved, but system overhead and complexity increase
Solution Approach 1:
The system uses parameter changes in hash values to detect data modifications. By computing hash values of data and comparing them, the system can efficiently identify changes without complex monitoring mechanisms. This parameter-based approach reduces system overhead while enabling real-time detection of data updates.
Data Source
AI summary
A computer-implemented method for transferring changes made to data from a data source to a data destination may be provided. The method involves using a message broker with streaming channels for transferring data changes from a data source to a data consumer. To determine if data has changed, a portion of data labeled with a key is sent through a hash function. A hash value with the corresponding key that has already been sent to the data consumer is retrieved from a streaming channel and compared to the newly created hash value. If the newly created hash value is different the data is sent to data consumers by the message broker via one streaming channel, while the new hash value is stored in the other streaming channel.