Streaming Graph Processing via Superstep Scheduling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed processing systems face challenges in efficiently processing streaming graph updates in real-time, as they often require recreating graphs or waiting for computations to complete before introducing new input, which slows down the computation process and lacks integrity in updating graph states.
Innovation Solution
The method involves scheduling graph updates in supersteps, using vertex cursors to track progress, and maintaining vertex histories to ensure timely and accurate processing of streaming input, allowing continuous updates without interrupting the computation, and enabling real-time reflection of changes in graph outputs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If graph computations wait for completion before processing new input, then computation integrity is maintained, but processing speed and real-time responsiveness deteriorate
Solution Approach 1:
The graph computation is divided into discrete supersteps, where each superstep processes a specific batch of graph updates independently. This segmentation allows the system to maintain computation integrity within each superstep while enabling continuous processing of new input in subsequent supersteps, thus improving real-time responsiveness without sacrificing reliability.
Solution Approach 2:
Graph updates are scheduled and staged in advance for specific supersteps before processing begins. This preliminary organization of data ensures that when each superstep executes, all necessary updates are already prepared and ordered, maintaining computation integrity while allowing the system to continuously accept and queue new input for future processing.
2Measurement precision
If graph computations are recreated for each computation cycle, then accurate processing of updates is ensured, but computation time and resource overhead increase
Solution Approach 1:
The graph structure transitions from a static, fully recreated model to a dynamic, incrementally updated structure. Vertices and edges are maintained in memory across supersteps and updated dynamically as new graph updates arrive, rather than recreating the entire graph each cycle. This dynamic approach preserves processing accuracy through version tracking while dramatically reducing computation time.
Solution Approach 2:
The graph computation maintains continuous operation across multiple supersteps without interruption or full recreation. The graph structure persists in memory, and updates are applied continuously as they arrive, eliminating the stop-start nature of full graph recreation. This continuity preserves accuracy through consistent state management while reducing time loss by avoiding repeated full-graph construction.
3Speed
If streaming graph updates are processed immediately, then real-time responsiveness is improved, but computation integrity and order of updates may be compromised
Solution Approach 1:
Incoming graph updates are scheduled and staged in advance for specific supersteps before processing occurs. This preliminary scheduling ensures that updates are processed in the correct order while maintaining real-time responsiveness, as the system continuously accepts and queues updates for future supersteps rather than processing them haphazardly.
Solution Approach 2:
A scheduling mechanism acts as an intermediary between incoming graph updates and the actual processing in supersteps. This intermediary queues updates, assigns them to appropriate supersteps, and ensures proper ordering, thus maintaining computation integrity while enabling continuous real-time input processing without direct immediate execution.
4Productivity
If multiple graph updates are processed in parallel, then processing throughput is increased, but maintaining update order and vertex state consistency becomes more complex
Solution Approach 1:
The parallel processing of graph updates is organized into discrete supersteps, where each superstep processes a specific batch of updates. This segmentation allows parallel execution within each superstep while maintaining overall order, as updates are grouped by superstep and processed independently, reducing state management complexity compared to unstructured parallel processing.
Solution Approach 2:
Updates are pre-scheduled and organized into supersteps before parallel processing begins. This preliminary organization ensures that all updates in a superstep are ready for parallel execution and that dependencies are resolved in advance, enabling high throughput parallel processing while keeping state management complexity manageable through structured grouping.
Data Source
AI summary
Methods, systems, and apparatus are described for performing streaming graph computations in a distributed processing system. In one aspect, a method includes receiving, during a first superstep, streaming graph input that comprises continuous input of first graph updates for updating the state of an existing graph, the existing graph comprising a plurality of vertices and edges, and each first graph update specifying a vertex and a message; scheduling, during the first superstep, the first graph updates for processing during a second superstep; providing, during the first superstep, the first graph updates as input to a graph computation for the existing graph; determining that the graph computation has completed processing for the first superstep, and in response: initiating the second superstep; and processing, during the second superstep, each first graph update; and providing, during the second superstep, graph output based at least in part on the first graph updates.