Heterogeneous Graph Cluster Load Balancing via Synthetic Proxy Profiling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current graph processing systems struggle to accurately model the graph processing capability of heterogeneous machines in clusters, leading to suboptimal load balancing and partitioning, which results in performance degradation due to uneven data distribution across machines with varying computational abilities.
Innovation Solution
The method involves generating synthetic proxy graphs to characterize machines' graph processing speeds, using a Computation Capability Ratio (CCR) metric to represent application-specific processing speeds, and employing heterogeneity-aware partitioning algorithms to distribute graph data based on these metrics, ensuring balanced workload distribution across machines.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If uniform graph/data partitioning is used in heterogeneous clusters, then implementation simplicity is maintained, but load balance deteriorates causing performance degradation
Solution Approach 1:
The patent applies local quality by assigning different partitioning strategies to different machine types in the heterogeneous cluster. Fast machines receive larger graph partitions while slow machines receive smaller partitions, tailored to their specific computational capabilities. This resolves the contradiction by maintaining operational simplicity through automated detection while achieving balanced load distribution across heterogeneous nodes.
Solution Approach 2:
The system dynamically changes partitioning parameters based on detected machine characteristics. By measuring actual graph processing speeds and adjusting partition sizes accordingly, the system transforms the fixed uniform partitioning approach into an adaptive heterogeneous partitioning strategy, resolving the performance degradation issue while keeping the implementation transparent.
2Difficulty of detecting and measuring
If hardware configuration-based speed estimation is used, then measurement simplicity is maintained, but measurement precision deteriorates due to inability to capture actual graph processing capability
Solution Approach 1:
The patent implements feedback by measuring actual graph processing speeds of machines and using this information to adjust partitioning decisions. The system continuously monitors performance metrics and feeds this information back into the partitioning algorithm, enabling accurate speed estimation that reflects real graph processing capabilities rather than relying solely on hardware specifications.
Solution Approach 2:
The system enables machines to self-report their actual graph processing speeds through profiling mechanisms. Each machine executes representative graph processing tasks and reports its performance characteristics, allowing the system to build accurate speed models without complex external measurement infrastructure.
3Measurement precision
If profiling is performed on natural graphs to capture machine capabilities, then measurement accuracy improves, but complexity increases due to difficulty in subsampling representative characteristics
Solution Approach 1:
The patent creates simplified proxy graphs that replicate the essential structural characteristics of natural graphs. Instead of profiling entire large-scale natural graphs, the system generates smaller synthetic graphs that preserve key properties like degree distribution and connectivity patterns, enabling accurate machine capability measurement with reduced complexity.
Solution Approach 2:
The profiling process is segmented into multiple stages: generating proxy graphs with specific properties, executing profiling workloads, collecting performance data, and synthesizing capability models. This segmentation breaks down the complex profiling task into manageable components, reducing overall system complexity while maintaining measurement accuracy.
Data Source
AI summary
A method, system and computer program product for load balancing of graph processing workloads. Synthetic proxy graphs are generated to characterize machines' graph processing speeds in a cluster. Each of the graph applications executing in the cluster is profiled using the synthetic graphs to form profiling sets. These formed profiling sets are run among the machines in the cluster to capture the machines' graph processing speeds. A metric for each of the graph applications is computed from a relative speedup among the machines in the cluster and/or the graph processing speeds. A graph file of a natural graph and a graph application are loaded. A metric out of the computed metrics is selected based on the graph application. The natural graph is then partitioned into multiple chunks which is distributed onto two or more machines in the cluster based on the selected metric and a user selected partitioning algorithm.