Distributed Graph Sampling Across Storage Nodes for Large Graphs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Graphs consume large amounts of memory, making them impractical to process in a single node due to memory constraints, and distributing them across multiple storage nodes introduces challenges for operations such as graph sampling.
Innovation Solution
A distributed graph sampling system utilizing storage nodes, network switches with computational capabilities, and compute nodes to efficiently sample and process graph data, optimizing the graph sample by removing duplicates and leveraging shared memory architectures for direct data access.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a graph is stored in a single node, then data access is simple and fast, but memory constraints prevent processing of large-scale graphs
Solution Approach 1:
The patent divides a large graph into multiple subgraphs and distributes them across different storage nodes. Each storage node stores a portion of the graph, enabling the system to handle graphs that exceed the memory capacity of a single node. This segmentation allows large-scale graph processing while maintaining manageable memory requirements at each node.
Solution Approach 2:
The patent transitions from a single-node architecture to a distributed multi-node architecture, adding the dimension of spatial distribution. By organizing graph data across multiple storage nodes connected via a network fabric, the system overcomes the memory limitations of a single node while introducing a distributed system architecture.
2Quantity of substance
If a graph is distributed across multiple storage nodes, then memory constraints are relieved, but graph sampling operations become complex and inefficient
Solution Approach 1:
The patent introduces a network switch as an intermediary component that coordinates graph sampling operations across distributed storage nodes. The network switch receives sampling requests, determines which storage nodes hold relevant subgraphs, and aggregates samples from multiple nodes. This intermediary simplifies the sampling process by abstracting the distributed nature of the graph, allowing efficient sampling without requiring complex client-side coordination logic.
Solution Approach 2:
The patent combines samples from multiple storage nodes at the network switch to form a complete graph sample. By merging the distributed subgraph samples into a unified result, the system achieves efficient graph sampling across the distributed architecture, maintaining productivity despite the distributed nature of the data.
3Measurement precision
If graph samples are aggregated from multiple storage nodes, then comprehensive sampling is achieved, but duplicate edges increase data volume and processing overhead
Solution Approach 1:
The patent extracts and removes duplicate graph edges from the aggregated sample at the network switch. By identifying and eliminating redundant edges that appear in multiple subgraph samples, the system reduces the data volume and processing overhead while preserving the completeness and accuracy of the graph sample. This extraction of duplicates ensures that the final sample contains only unique edges.
4Speed
If direct data access is implemented at storage nodes, then data retrieval is fast, but network traffic increases when aggregating samples
Solution Approach 1:
The patent performs preliminary sampling operations at each storage node before aggregation, generating compact subgraph samples that contain only the necessary data. By pre-processing and filtering data at the source, the system reduces the volume of data that needs to be transmitted over the network, thereby minimizing network traffic while maintaining fast data access through direct storage node operations.
Data Source
AI summary
A distributed graph sampling system includes storage nodes that store a graph, a network switch that orchestrates accelerated sampling of the graph, and a compute node that processes a sample of the graph with a GNN. The graph sampling operations are distributed across the storage nodes. Each storage node may store a subgraph (e.g., a portion of the graph) and generate a sample of the subgraph.