Distributed Graph Loading With Lazy Materialization for Balanced Partitions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed graph loading in large datasets is inefficient due to high computational cost and data imbalance, particularly in distributed graph processing systems, where conventional methods require extensive network communication and complex computation for balancing.
Innovation Solution
A method involving minimal communication between computers to repartition a distributed graph by using intermediate representations of vertices and edges, with mutable data structures and segmented arrays, allowing for efficient reassignment and rebalancing of data partitions through tracking computers and ghost vertices to achieve a balanced workload.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional distributed graph loading methods are used, then data can be loaded into the distributed system, but the process requires extensive network communication and complex computation for balancing, resulting in high computational overhead and long loading time
Solution Approach 1:
The patent applies preliminary action by performing hash computations and partition assignments locally on each computing node before actual data loading occurs. Each node computes hash values for its local data files and determines partition assignments in advance, avoiding the need for extensive network communication during the actual loading phase. This preliminary local computation significantly reduces the time required for distributed graph loading while maintaining balanced data distribution across the cluster.
2Ease of operation
If hash-based partitioning is used to allocate vertices to computers, then partitioning can be achieved, but much communication is required during loading and data imbalance occurs when vertex degree distribution is skewed
Solution Approach 1:
The patent applies local quality by allowing each computing node to independently compute hash values and determine partition assignments based on its local data characteristics. Each node adapts its partitioning strategy to its specific workload, using local hash computations to determine which partitions it will own. This localized approach maintains simplicity while achieving better data balance compared to centralized hash-based partitioning, as each node can optimize for its local vertex degree distribution patterns.
3Reliability
If eager materialization of the initially loaded graph is performed, then the graph is fully materialized in memory, but the computational cost of balancing (repartitioning) increases significantly
Solution Approach 1:
The patent applies partial action by materializing only the necessary portions of the graph data in memory rather than performing eager full materialization. Each computing node materializes graph partitions that it has been assigned to own, based on local hash computations. This partial materialization approach maintains data availability for processing while significantly reducing the computational cost of balancing, as nodes only need to manage and transfer the partitions they actually need rather than materializing and rebalancing the entire graph.
Data Source
AI summary
Techniques herein minimally communicate between computers to repartition a graph. In embodiments, each computer receives a partition of edges and vertices of the graph. For each of its edges or vertices, each computer stores an intermediate representation into an edge table (ET) or vertex table. Different edges of a vertex may be loaded by different computers, which may cause a conflict. Each computer announces that a vertex resides on the computer to a respective tracking computer. Each tracking computer makes assignments of vertices to computers and publicizes those assignments. Each computer that loaded conflicted vertices transfers those vertices to computers of the respective assignments. Each computer stores a materialized representation of a partition based on: the ET and vertex table of the computer, and the vertices and edges that were transferred to the computer. Edges stored in the materialized representation are stored differently than edges stored in the ET.