Epoch Commit Protocol for Consistent Graph Snapshots
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems struggle to efficiently process and analyze real-time data streams from online services, such as social media platforms, to capture dynamic relationships and generate timely insights, while maintaining consistency and handling high update rates.
Innovation Solution
A distributed system architecture that decouples graph mining from graph updates, using an epoch commit protocol to create global consistent snapshots, allowing for incremental graph computations and fault tolerance, and supporting incremental expansion to cope with increasing update rates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If a distributed system processes real-time data streams continuously, then the timeliness of insights is improved, but maintaining graph consistency becomes difficult
Solution Approach 1:
The system implements periodic snapshot creation at epoch boundaries, where graph computations are performed at regular intervals rather than continuously. This periodic action allows the system to maintain continuous data streaming while periodically freezing the graph state for consistent computations, resolving the conflict between timeliness and consistency.
Solution Approach 2:
The patent segments the continuous data stream processing into discrete epochs, where each epoch represents a consistent snapshot of the graph state. By dividing the continuous processing into discrete time segments, the system can guarantee consistency within each epoch while maintaining overall continuous operation across multiple epochs.
2Productivity
If the system handles high update rates from data streams, then the productivity is improved, but the complexity of maintaining consistency increases
Solution Approach 1:
The system performs preliminary actions by pre-defining epoch boundaries and preparing snapshot states before high-rate updates occur. This allows the system to handle bursts of updates efficiently by having pre-prepared consistent states to fall back on, reducing the real-time complexity of maintaining consistency during high-update periods.
3Device complexity
If graph mining is coupled with graph updates, then the system simplicity is maintained, but the measurement precision of graph insights deteriorates
Solution Approach 1:
The patent extracts graph mining operations from the continuous graph update process by performing mining only on frozen snapshot states. This separation allows graph mining to operate on stable, consistent data without being affected by ongoing updates, thereby improving the precision of insights while maintaining relatively simple architecture through the use of periodic snapshots.
4Reliability
If the system creates global consistent snapshots frequently, then the reliability is improved, but the loss of time for computation increases
Solution Approach 1:
The system performs partial graph computations by only processing updates that occur within each epoch rather than re-computing everything. This partial action approach maintains reliability through frequent snapshots while reducing computation time by leveraging the incremental nature of updates between epochs.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A continuous stream data (e.g., messages, tweets) is received by ingest nodes of a platform. The ingest nodes may analyze the data to create a transaction of graph updates, assign a sequence number to the transaction, and distribute the graph updates with the sequence number to graph nodes of the platform. The graph nodes may store graph updates from ingest nodes, and then the ingest nodes may report graph update progresses in a progress table. A snapshot may be taken based on the progress table, and then graph-mining computation may be implemented. Tolerate failures and decaying may be supported and incremental expansion may be allowed to cope with increasing update rates and computation needs.