Distributed Graph Edge List Indexing for Data Transfer Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed graph processing systems face inefficiencies in data transfer between computers, as existing methods often require repeated transmission of data, which is unnecessary and inefficient, especially when dealing with large datasets that exceed a single computer's memory capacity.
Innovation Solution
A system of computers divides a directed graph into logical partitions and builds edge lists for each computer, allowing for the compact description and exchange of external edges between neighbors, with dynamic values calculated and transmitted in bulk, and using inclusion bitmaps to exclude irrelevant edges, thereby optimizing data transfer during iterations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is transferred repeatedly between computers in distributed graph processing, then data availability is improved, but data transfer overhead and processing time increase
Solution Approach 1:
The system performs preliminary actions by sending data in bulk before it is actually needed for computation. The initiating computer transmits data to the target computer in advance, allowing the target computer to have data ready before computation begins, thereby eliminating repeated transfers and reducing data transfer overhead during the computation process.
2Productivity
If bulk data transfer is used to avoid repeated transmissions, then data transfer efficiency is improved, but data access efficiency deteriorates
Solution Approach 1:
The system segments the bulk transferred data into structured formats with metadata that organizes data by computational operations. The target computer receives structured data packets containing metadata that describes how to access and process specific portions of the data, enabling efficient data access without requiring full bulk data retrieval for each computation.
Solution Approach 2:
The system introduces an intermediary layer of metadata that acts as a mediator between the bulk transferred data and the computation process. This metadata contains information about data location, format, and access methods, allowing the target computer to efficiently locate and access specific data portions without processing the entire bulk transfer for each computation.
3Measurement precision
If data is sent with detailed information about graph parts and operations, then computation accuracy is improved, but communication overhead increases
Solution Approach 1:
The system extracts only the essential metadata information needed for accurate computation from the full data context. Instead of transmitting complete graph structures and operation details, the system sends compact metadata that captures the necessary information about data location, format, and computation requirements, reducing communication overhead while maintaining computation accuracy.
Data Source
AI summary
Techniques herein index data transferred during distributed graph processing. In an embodiment, a system of computers divides a directed graph into partitions. The system creates one partition per computer and distributes each partition to a computer. Each computer builds four edge lists that enumerate edges that connect the partition of the computer with a partition of a neighbor computer. Each of the four edge lists has edges of a direction, which may be inbound or outbound from the partition. Edge lists are sorted by identifier of the vertex that terminates or originates each edge. Each iteration of distributed graph analysis involves each computer processing its partition and exchanging edge data or vertex data with neighbor computers. Each computer uses an edge list to build a compactly described range of edges that connect to another partition. The computers exchange described ranges with their neighbors during each iteration.