Distributed Graph Processing Vertex Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed graph processing systems face challenges in minimizing communication and load balancing, particularly in assigning and managing internal identifiers for vertices across multiple computers, leading to issues like network latency, memory exhaustion, and workload imbalance.
Innovation Solution
The system decouples vertex loading from tracking by having each computer load a subset of edges and using a hashing mechanism to determine which computer hosts a vertex, with a tracking computer managing the assignment to minimize redundant loading and optimize vertex distribution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a global hash function is applied to vertices to decide which machine should load and exclusively own a vertex, then vertex distribution is determined, but network latency and network congestion occur due to irregular communication and random access
Solution Approach 1:
The patent introduces a tracking computer as an intermediary between providing computers and the graph data. The tracking computer receives requests from providing computers, determines which computer should own each vertex using a hash function, and returns the ownership information. This intermediary approach transforms irregular random access patterns into more predictable sequential access patterns, reducing network latency and congestion while maintaining the benefits of hash-based vertex distribution.
2Productivity
If each computer only reads its own portion of the graph data determined by hashing, then data ownership is clear, but workload imbalance occurs because the hash function arbitrarily determines vertex distribution without considering analysis requirements
Solution Approach 1:
The patent implements dynamic vertex distribution by allowing the system to adjust which computer owns which vertex based on runtime conditions and analysis requirements. The tracking computer can reassign vertex ownership dynamically, and providing computers can request vertices they need for their analysis tasks. This dynamic approach replaces the static, arbitrary hash-based distribution with a flexible system that optimizes workload balance while maintaining clear data ownership through the tracking computer.
3Reliability
If all available computers load the whole graph to solve identifier agreement problems, then identifier coherence is achieved, but memory exhaustion and loading latency occur
Solution Approach 1:
The patent segments the graph data into portions distributed across multiple providing computers, with each computer loading only its assigned subset of vertices and edges. The tracking computer maintains a global view and mapping of which vertices are located on which providing computers. This segmentation approach allows identifier coherence to be maintained through the tracking computer's global knowledge while each individual computer uses minimal memory, avoiding both memory exhaustion and excessive loading latency.
4Productivity
If multiple computers simultaneously load graph data to enable parallel processing, then processing speed increases, but identifier agreement becomes difficult and communication overhead increases
Solution Approach 1:
The tracking computer serves as a central intermediary that coordinates identifier assignment and resolution across all providing computers. When multiple computers simultaneously load graph data, they all communicate with the tracking computer to resolve vertex identifiers and determine ownership. This centralized coordination eliminates identifier conflicts by ensuring all computers agree on which computer owns which vertex, while the hash-based determination minimizes the communication required compared to distributed consensus algorithms.
Data Source
AI summary
Techniques minimize communication while loading a graph. In a distributed embodiment, each computer loads some edges of the graph. Each edge connects a source vertex (SV) to a destination vertex. For each SV of the edges, the computer hashes the SV to detect a tracking computer (TrC) that tracks on which computer does the SV reside. Each computer informs the TrC that the SV originates an edge that resides on that computer. For each SV, the TrC detects that the SV originates edges that reside on multiple providing computers (PCs). The TrC selects a target computer (TaC) from the multiple PCs to host the SV. The TrC instructs each PC, excluding the TaC, to transfer the SV and related edges that are connected to the SV to the TaC. A vertex's internal identifier indicates which computer hosts the vertex. The TrC maintains a mapping between external and internal identifiers.