Distributed Adjacency-List Traversal for Billion-Node Graphs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional techniques for traversing large connected components with billions of nodes are infeasible due to memory limitations on a single processing node.
Innovation Solution
Distribute the adjacency list of the large connected component across multiple processing nodes, performing iterations and data consolidation using a map-reduce framework to traverse the graph efficiently.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If conventional techniques are used to traverse large connected components, then the approach is simple and easy to implement, but it fails due to memory limitations on a single processing node
Solution Approach 1:
The adjacency list is divided and distributed across multiple processing nodes in a distributed system. Each processing node stores and processes only a portion of the adjacency list, enabling the system to handle large connected components that exceed the memory capacity of a single node while maintaining the simplicity of conventional traversal techniques through parallel execution.
2Device complexity
If a single processing node is used for graph traversal, then the system complexity is low, but it cannot handle connected components with billions of nodes
Solution Approach 1:
The problem is solved by adding a dimensional aspect - distributing the data across multiple processing nodes rather than relying on a single node's memory. This dimensional shift from one node to multiple nodes enables the system to handle connected components with billions of nodes by leveraging the collective memory capacity of the distributed system.
3Quantity of substance
If the adjacency list is distributed across multiple processing nodes, then memory constraints are overcome, but the system complexity increases
Solution Approach 1:
A coordination mechanism acts as an intermediary to manage the distributed traversal process. This intermediary coordinates the traversal operations across multiple processing nodes, manages the distribution of adjacency list segments, and orchestrates the collective processing to overcome memory constraints while controlling system complexity through structured coordination.
Data Source
AI summary
A distributed system including multiple processors associated with non-transitory computer-readable media storing computing instructions. The computing instructions, when collectively executed on the multiple processors, cause the multiple processors collectively to perform certain acts. The acts can include executing multiple iterations until a stopping condition is satisfied, by, for each of the multiple iterations: (i) processing a set of input nodes at the multiple processors using a set of criteria to generate first data at the multiple processors, wherein the set of input nodes is different at each of the multiple iterations; (ii) determining a list of output nodes using adjacency rows of an adjacency list at different ones of the multiple processors, such that each output node of the list of output nodes is one hop from a respective input node of the set of the input nodes; and (iii) updating the set of the input nodes for a subsequent iteration of the multiple iterations based on the list of output nodes when the stopping condition is not satisfied. The acts also can include outputting second data based at least in part on the first data. Other embodiments are disclosed.