Distributed Graph Data Sharding for I/O Overhead Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale graph data processing systems face challenges due to memory limitations and poor data locality, leading to high I/O overheads and inefficient processing, as the data volume exceeds the system's memory capacity and adjacency relationships are disrupted when only partial data is loaded for processing.
Innovation Solution
A distributed system architecture is employed, where a master node divides graph data into shards and schedules edge sets across multiple worker nodes, utilizing massive memory resources for efficient processing, reducing I/O overheads, and improving scalability and throughput by leveraging both disk and memory resources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If graph data is divided into shards and processed in parts, then memory usage is reduced and processing becomes feasible, but data locality is destroyed and I/O overhead increases
Solution Approach 1:
The patent divides graph data into multiple shards distributed across different storage nodes, allowing the system to process large-scale graph data that exceeds single-machine memory capacity. Each shard contains a subset of vertices and edges, enabling parallel processing while managing memory constraints.
Solution Approach 2:
The system pre-computes and stores adjacency information in an optimized format during data loading, organizing edges by destination vertex and pre-determining the shard locations of source vertices. This preliminary organization reduces the need for frequent disk I/O during the actual computation phase, as the system can efficiently retrieve required data without repeated scanning.
2Productivity
If all graph data is loaded into memory, then processing speed increases, but the system cannot handle graph data exceeding memory capacity
Solution Approach 1:
The patent transitions from a single-machine memory-constrained model to a distributed multi-machine system. By adding the dimension of distribution across multiple nodes, the system can handle graph data scales that exceed individual machine memory capacity while maintaining parallel processing capabilities for improved throughput.
Solution Approach 2:
The system introduces a distributed file system as an intermediary layer between storage and computation. This intermediary provides efficient data retrieval mechanisms, allowing the computation engine to access graph data from disk storage without loading entire datasets into memory, thus bridging the gap between storage capacity and memory constraints.
3Adaptability or versatility
If graph data is processed in a distributed system, then scalability improves, but communication overhead between nodes increases
Solution Approach 1:
The patent assigns specific shards to specific worker nodes, creating a localized processing architecture. Each worker node processes its assigned shard independently, minimizing the need for inter-node communication. The system leverages the locality of data access patterns by keeping related data together in the same shard and processing it on the same node.
Solution Approach 2:
The system extracts and pre-computes the shard assignment information and adjacency metadata during the data loading phase. This extracted information is stored in memory-efficient formats that allow workers to determine data locations and relationships without requiring continuous communication with the master node or other workers during computation.
Data Source
Figure 1~2
Figure 3A
Figure 3B~3C
AI summary
The present invention discloses a graph data processing method applied to a distributed system. The distributed system includes a master node and a plurality of worker nodes. The master nodemethod includes: obtaining, by the master node, graph data; dividing the graph data, to obtain P shards, where P is a positive integer, and the P shards include a first shard and a second shard; master nodedetermining at least two edge sets from each shard; scheduling, to at least two worker nodes for processing, at least two edge sets included in the first shard in the P shards; and scheduling, to the at least two worker nodes for processing, an associate edge set included in the second shard, where the associate edge set is an edge set that includes an outgoing edge of a target vertex corresponding to the first shard. According to the graph data processing method provided in embodiments of the present invention, massive storage resources in the distributed system can be used for graph data processing, so as to improve overall throughput of the system and reduce IO overheads.