Post-Exascale Graph Computing With Topology-Aware Asynchronous Partitioning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing graph computing systems face inefficiencies due to complex data dependency, unbalanced loads, poor data locality, and irregular data accesses, leading to resource underutilization, prolonged convergence times, and excessive communication overheads, which limit the performance and scalability of supercomputers in large-scale graph computing.
Innovation Solution
A post-exascale graph computing method involving distributed, asynchronous processing with topology-aware mechanisms and hierarchical, very-large-scale communication, where graph blocks containing active vertices are prioritized for processing, and data is partitioned in a community-structure-aware manner to reduce redundant communication and improve data locality and parallelism.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If traditional supercomputers process large-scale graph data using basic parallel data processing units, then computing capacity is increased, but complex data dependency and irregular data accesses cause poor data locality and resource underutilization
Solution Approach 1:
The patent segments graph data into super-vertices (groups of vertices) and processes them as unified units. This segmentation transforms fine-grained vertex-level parallelism into coarse-grained super-vertex-level parallelism, reducing the impact of complex data dependencies while maintaining high computing capacity utilization.
Solution Approach 2:
The patent merges multiple vertices into super-vertices and combines their computation and communication operations. This merging reduces the number of independent processing units, decreases communication overhead, and improves data locality by keeping related vertices together in the same memory region.
2Reliability
If graph vertices sequentially propagate states along dependency chains, then correctness is maintained, but parallel computing resources are underutilized and convergence time is prolonged
Solution Approach 1:
The patent introduces dynamic dependency tracking at the super-vertex level, where the system dynamically determines when super-vertices are ready for processing based on their dependency satisfaction. This dynamic approach enables more aggressive parallelization while maintaining correctness, as the system adapts to the actual data dependency patterns rather than following fixed sequential propagation.
Solution Approach 2:
The patent ensures continuous useful action by keeping multiple super-vertices in flight simultaneously, rather than waiting for one to complete before starting the next. The pipeline architecture maintains continuous processing by overlapping computation and communication phases across different super-vertices, maximizing resource utilization and reducing convergence time.
3Loss of information
If frequent small-sized messages are communicated among computing nodes, then state propagation is achieved, but network communication overheads increase significantly
Solution Approach 1:
The patent merges communication operations at the super-vertex level, combining multiple small messages into fewer larger messages. This merging reduces the total number of communication events and improves network utilization by better utilizing bandwidth, thereby reducing communication overhead while maintaining complete state propagation.
Solution Approach 2:
The patent changes the dimension of communication from vertex-level to super-vertex-level aggregation. By batching communications and using collective operations, the system transforms numerous individual point-to-point messages into fewer aggregated communication events, reducing protocol overhead and improving network efficiency.
4Reliability
If all graph vertices are processed uniformly, then completeness is ensured, but load balancing deteriorates due to varying vertex degrees and computation requirements
Solution Approach 1:
The patent applies local quality by creating super-vertices with heterogeneous compositions tailored to local graph structures. Each super-vertex is formed by grouping vertices with similar characteristics (e.g., similar degree, similar computation requirements), ensuring that each processing unit has balanced workloads while maintaining processing completeness through systematic coverage of all vertices.
Solution Approach 2:
The patent segments the graph into super-vertices that are specifically designed to balance workloads. This segmentation strategy groups vertices to create relatively uniform computation patterns across super-vertices, improving load balancing while ensuring all vertices are eventually processed through the segmented structure.
5Ease of operation
If traditional memory access patterns are used for graph data, then data access is simple, but data locality is poor and memory bandwidth is underutilized
Solution Approach 1:
The patent merges multiple vertex data structures into super-vertex structures that are stored contiguously in memory. This merging improves data locality by placing related data close together in memory, enabling more efficient memory access patterns that better utilize memory bandwidth while maintaining simple access through the unified super-vertex interface.
Data Source
AI summary
The present invention relates to a post-exascale graph computing method, and corresponding system, storage medium and electronic device. The invention solves the problems of low computing performance, poor scalability and high communication overhead in the large-scale distributed environment, and improves the performance of the supercomputer when supporting large-scale graph computing.