Delta Replication for Data Processing Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data processing systems burden resources and performance by retrieving all data, including unchanged data, when only data changes are required, leading to inefficient processing and network usage.
Innovation Solution
A computer-implemented method for replicating data changes by identifying relevant attributes in a data query, retrieving data at two points in time, calculating delta differences, and selectively replicating only the changes, using segment and field attributes to control which data is included or excluded from the replication process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If all data is retrieved to ensure complete information availability, then data completeness is improved, but processing resources and network bandwidth are consumed excessively
Solution Approach 1:
The patent extracts only the changed data (delta) from the complete dataset by comparing current data with previously replicated data. This extraction principle allows the system to transmit only the necessary changes rather than all data, reducing network bandwidth consumption and processing resources while maintaining data completeness for the target system.
Solution Approach 2:
The patent applies partial action by retrieving and transmitting only a portion of the total data - specifically, only the changed portions. By using delta queries that identify and extract only modified records, the system avoids the excessive action of transmitting complete datasets, thereby reducing resource consumption while ensuring all necessary changes are captured.
2Loss of information
If all data is retrieved to ensure complete information availability, then data completeness is improved, but network bandwidth is consumed excessively
Solution Approach 1:
The patent extracts only the changed data (delta) from the complete dataset by comparing current data with previously replicated data. This extraction principle allows the system to transmit only the necessary changes rather than all data, reducing network bandwidth consumption and processing resources while maintaining data completeness for the target system.
Solution Approach 2:
The patent applies partial action by retrieving and transmitting only a portion of the total data - specifically, only the changed portions. By using delta queries that identify and extract only modified records, the system avoids the excessive action of transmitting complete datasets, thereby reducing resource consumption while ensuring all necessary changes are captured.
3Productivity
If delta replication is implemented to transmit only changed data, then processing efficiency is improved, but system complexity increases
Solution Approach 1:
The patent segments the data replication process into distinct components: delta identification through query parameters, delta calculation through data comparison, and selective transmission. This segmentation allows each component to be optimized independently and simplifies the overall system architecture by breaking down the complex replication task into manageable, modular operations.
Solution Approach 2:
The patent introduces an intermediary mechanism - the delta query parameter system - that mediates between the source and target systems. This intermediary layer handles the complexity of delta identification and calculation, shielding the underlying systems from complexity while enabling efficient partial data transmission and improving overall processing efficiency.
Data Source
AI summary
A system, a method, and a computer program product for replication of data changes are disclosed. Based on a received data query, at least one attribute of stored data is identified for replication. The attribute corresponds to at least one parameter of the received query. Data stored at a first time and data stored at a second time are retrieved. The second time is subsequent to the first time. Data stored at the first time is compared to the data stored at the second time to determine a delta difference between the data stored at the first and second times. The computed delta difference is selected for the identified attribute. The selected computed delta difference is replicated in response to the received query.