Graph Partitioning and Update Buffering for Data Propagation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Graph processing systems face inefficiencies due to high data movement between storage and memory, leading to increased processing time, power consumption, and I/O operations, particularly when dealing with large-scale graph processing where vertices and edges exceed available memory limits.
Innovation Solution
The method involves partitioning graph vertices based on their incoming and outgoing edges, placing those with fewer edges in earlier partitions, and using a pipeline architecture with compute units and writers to generate and propagate updates efficiently, reducing synchronization overhead and I/O accesses by buffering and distributing updates across multiple threads and memory banks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If graph processing systems propagate values calculated during one iteration to a later iteration to reduce I/O accesses, then I/O operations are reduced, but memory usage increases and processing complexity increases
Solution Approach 1:
The patent divides the graph into multiple partitions based on vertex degree (number of edges), creating a hierarchical structure where high-degree vertices are separated from low-degree vertices. This segmentation allows the system to process different partitions with different memory requirements, enabling propagation of calculated values for high-degree vertices without requiring all graph data to remain in memory simultaneously.
Solution Approach 2:
The patent applies different processing strategies to different parts of the graph based on local characteristics. High-degree vertices (hubs) receive special treatment with full propagation of calculated values, while low-degree vertices use simpler processing. This local quality approach optimizes memory usage by focusing propagation resources on vertices that provide the most benefit.
2Quantity of substance
If the graph is partitioned into more partitions to fit memory constraints, then memory usage is reduced, but the number of I/O operations increases
Solution Approach 1:
The patent performs preliminary partitioning of the graph based on vertex degree before processing begins. By pre-identifying high-degree vertices and creating partitions that group them together, the system prepares the data structure to maximize propagation opportunities in advance, reducing the need for repeated I/O operations during subsequent processing iterations.
3Productivity
If vertices with more edges are processed earlier to maximize propagation, then processing efficiency improves, but memory requirements increase
Solution Approach 1:
The patent segments the graph processing into distinct phases based on vertex degree. High-degree vertices are processed in dedicated partitions that can be loaded into memory, processed with full propagation, and then unloaded. This segmentation allows efficient processing of memory-intensive operations on only the necessary subset of vertices at any given time.
Data Source
AI summary
A method of partitioning a graph for processing may include sorting two or more vertices of the graph based on incoming edges and outgoing edges, placing a first one of the vertices with fewer incoming edges in a first partition, and placing a second one of the vertices with fewer outgoing edges in a second partition. The first one of the vertices may have a lowest number of incoming edges, and the first one of the vertices may be placed in a first available partition. The second one of the vertices may have a lowest number of outgoing edges, and the second one of the vertices may be placed in a second available partition. A method for updating vertices of a graph may include storing a first update in a first buffer, storing a second update in a second buffer, and transferring the first and second updates to a memory using different threads.