Distributed Graph Mutations via Shared Arrays and Delta Logs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing graph processing systems face challenges in supporting mutable graphs in distributed environments while maintaining analytical performance and balancing load and memory across nodes, especially as the number of graph mutations increases.
Innovation Solution
The solution involves using shared arrays for original graph information and delta logs for local mutations, with iterators to access the graph as a unified array, and a dictionary to map vertex keys to nodes, allowing for efficient distribution and reconstruction of the graph across nodes, and periodic consolidation of delta logs to minimize overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If a distributed graph stores all mutations in memory to support fast mutations, then mutation speed improves, but memory consumption increases and load balancing deteriorates
Solution Approach 1:
The patent segments the graph storage into shared arrays (original graph information) and delta logs (mutations). Each node stores only its local delta logs in memory while shared arrays can be stored on disk or distributed storage. This segmentation allows fast mutation processing for local changes while avoiding the need to load the entire graph into memory at each node.
Solution Approach 2:
The patent introduces a temporal dimension by maintaining delta logs that record mutations over time. Instead of storing only the current state of the graph in memory, the system stores the sequence of changes (delta logs) which can be replayed or consolidated as needed. This allows the system to support fast mutations by storing only changes rather than the complete graph state in memory.
2Reliability
If delta logs are replicated across all nodes for consistency, then snapshot isolation improves, but communication overhead and memory usage increase
Solution Approach 1:
The patent implements local quality by allowing each node to maintain its own local copy of delta logs relevant to its responsibilities. Instead of uniformly replicating all delta logs to all nodes, each node stores and processes only the delta logs necessary for its local operations. This reduces communication overhead while maintaining snapshot isolation guarantees for local graph operations.
3Reliability
If the graph is frequently reconstructed to maintain consistency after mutations, then data consistency improves, but processing time increases
Solution Approach 1:
The patent applies preliminary action by pre-computing and storing delta logs that capture graph mutations. Instead of reconstructing the entire graph after each mutation, the system pre-processes mutations into delta log format and stores them for later consolidation or replay. This preliminary processing reduces the time required for graph reconstruction while maintaining data consistency.
Solution Approach 2:
The patent implements periodic action by consolidating delta logs at scheduled intervals or when certain thresholds are reached. Instead of immediately reconstructing the graph after every mutation, the system accumulates delta logs and performs periodic consolidation to update the shared arrays. This periodic approach reduces processing time by batching reconstruction operations while maintaining data consistency through the snapshot isolation model.
Data Source
AI summary
Data structures and methods are described for applying mutations on a distributed graph in a fast and memory-efficient manner. Nodes in a distributed graph processing system may store graph information such as vertices, edges, properties, vertex keys, vertex degree counts, and other information in graph arrays, which are divided into shared arrays and delta logs. The shared arrays on a local node remain immutable and are the starting point of a graph, on top of which mutations build new snapshots. Mutations may be supported at both the entity and table levels. Periodic delta log consolidation may occur at multiple levels to prevent excessive delta log buildup. Consolidation at the table level may also trigger rebalancing of vertices across the nodes.