Data-Conveyor System for Distributed Data Change Propagation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed computing systems, propagating the latest data changes across components is challenging due to high memory usage and the risk of lost updates, especially when systems experience downtime or other disruptions.
Innovation Solution
A data-conveyor system acts as a broker between source and target components, using change-logging with unique identifiers and timestamps to propagate changes efficiently, avoiding the need to pass full data copies and minimizing storage and processing time, while also enabling centralized monitoring and tracking.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full data copies are passed around among components, then data consistency is maintained, but memory usage increases and update speed decreases
Solution Approach 1:
The patent extracts only the essential change information (identifiers and timestamps) from full data copies and propagates these minimal updates among components. This extraction approach maintains data consistency while dramatically reducing memory usage and improving update speed compared to transmitting complete data copies.
Solution Approach 2:
The patent segments the data update process into identification and propagation phases. Change identifiers and timestamps are separated from the actual data content, allowing the system to track and propagate only the necessary update metadata, thereby improving efficiency while maintaining consistency.
2Reliability
If full data copies are passed around among components, then data consistency is maintained, but bandwidth consumption increases
Solution Approach 1:
The patent extracts only the essential change information (identifiers and timestamps) from full data copies and propagates these minimal updates among components. This extraction approach maintains data consistency while dramatically reducing memory usage and improving update speed compared to transmitting complete data copies.
Solution Approach 2:
The patent segments the data update process into identification and propagation phases. Change identifiers and timestamps are separated from the actual data content, allowing the system to track and propagate only the necessary update metadata, thereby improving efficiency while maintaining consistency.
3Loss of information
If updates are passed in their entirety, then data completeness is ensured, but lost update problems increase
Solution Approach 1:
The patent implements a feedback mechanism using unique change identifiers and timestamps that allow components to track and detect lost updates. By monitoring these identifiers, the system can identify when updates are lost and implement recovery procedures, thereby reducing update loss risk while maintaining data completeness.
Solution Approach 2:
The patent assigns unique identifiers and timestamps to changes before propagation. This preliminary action creates a tracking framework that enables the system to detect and recover from lost updates, reducing reliability issues while ensuring data completeness can be maintained.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Disclosed are systems, apparatus, and methods for propagating data changes in a distributed computing system from source components to target components. In accordance with various embodiments, one or more producer components of a data-conveyor system may detect changes to data records in one or more source components, and store backlog entries responsive to detecting the changes, wherein these backlog entries do not include contents of the data record. One or more consumer components of the data-conveyor system may retrieve updated data of changed data records based on the backlog entries and provide the updated data to one or more target component(s).