Distributed Graph Sharding for Scalable Shortest Path Computation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems for computing shortest paths in large weighted digraphs are not scalable and cannot efficiently handle machine failures, especially when dealing with graphs having trillions of edges and nodes, as they rely on shared memory models and are limited to in-memory graph data.
Innovation Solution
A distributed system that computes single source and multiple source shortest paths across thousands of computers, using a distributed environment where each server processes a shard of the graph data stored on disk, with mechanisms for efficient recovery from machine failures and propagation of distance updates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If shared memory models and in-memory graph data are used, then computation speed is improved, but scalability and fault tolerance deteriorate
Solution Approach 1:
The patent divides the graph data into shards distributed across multiple machines, with each machine processing a portion of the graph independently. This segmentation enables the system to scale to trillions of edges by distributing the computational load across thousands of computers while maintaining computation speed through parallel processing.
Solution Approach 2:
The patent transitions from a single-machine in-memory computation model to a distributed multi-machine architecture, adding the dimension of network distribution. This allows the system to handle graphs too large for single-machine memory while maintaining computational efficiency through coordinated distributed processing.
2Speed
If shared memory models are used, then computation speed is improved, but fault tolerance deteriorates
Solution Approach 1:
The patent implements checkpointing mechanisms where machines periodically save their computation state to durable storage. When a machine failure occurs, the system detects the failure and restores computation from the last checkpoint, ensuring fault tolerance while minimizing computation speed impact through efficient state recovery.
Solution Approach 2:
The patent prepares for potential failures by implementing redundant computation paths and pre-computed checkpoints before failures occur. This allows the system to recover from machine failures without losing significant computation progress, maintaining both reliability and overall computation speed.
3Speed
If in-memory graph data is used, then computation speed is improved, but graph size capacity deteriorates
Solution Approach 1:
The patent segments the large graph data into smaller shards that can be stored and processed distributedly across multiple machines. Each machine maintains in-memory data structures for its assigned shard, enabling the system to handle graphs with trillions of edges by distributing the data capacity across the cluster while maintaining fast in-memory computation for each partition.
Solution Approach 2:
The patent moves from single-machine memory constraints to distributed network storage, adding the dimension of distributed data capacity. This allows the system to handle graphs exceeding the memory capacity of any single machine while maintaining computation speed through efficient distributed data access and processing.
4Quantity of substance
If distributed environment across many machines is used, then graph size capacity is improved, but system complexity increases
Solution Approach 1:
The patent implements a universal shard server architecture where each machine performs multiple functions: storing graph data, processing computations, managing checkpoints, and handling failure recovery. This multi-functionality reduces overall system complexity by eliminating the need for specialized components for each function while maintaining distributed graph size capacity.
Solution Approach 2:
The patent enables machines to autonomously manage their own computation state, checkpoint creation, and failure recovery without requiring complex centralized coordination. Each shard server independently handles its portion of the graph and can recover from failures using its own checkpoints, reducing the complexity of distributed system management while maintaining large graph processing capability.
Data Source
AI summary
Systems and methods for finding multiple shortest paths. A directed graph representing web resources and links are divided into shards, each shard comprising a portion of the graph representing multiple web resources. Each of the shards is assigned to a server, and a distance table is calculated in parallel for each of the web resources in each shard using a nearest seed computation in the server to which the shard was assigned.