Distributed Shortest Path Computation via Shard-Based Checkpointing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current graph analysis methods, such as Dijkstra and Bellman-Ford algorithms, are not scalable for large directed graphs and cannot efficiently handle machine failures, especially when dealing with graphs having trillions of edges and nodes, as they rely on shared memory and are not designed for distributed environments.
Innovation Solution
A distributed system that divides nodes into subsets (shards) and assigns them to servers, allowing for parallel computation of shortest paths across multiple machines, with mechanisms for handling machine failures through checkpointing and adaptive propagation thresholds, enabling efficient computation of single and multiple source shortest paths in extremely large graphs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If Dijkstra or Bellman-Ford algorithms are used for shortest path computation, then algorithm simplicity and ease of implementation are maintained, but scalability and parallelism are lost when dealing with graphs having trillions of edges
Solution Approach 1:
The patent divides the graph into multiple shards distributed across different machines. Each machine processes a subset of nodes and edges independently, allowing the system to handle graphs with trillions of edges by parallelizing the computation across many machines rather than requiring a single machine to process the entire graph sequentially.
2Productivity
If shared memory models are used for parallel shortest path computation, then parallelism is achieved, but reliability and fault tolerance are reduced when machines fail
Solution Approach 1:
The patent introduces a coordinator machine that manages the distributed computation across multiple worker machines. This intermediary coordinates the parallel processing, handles fault detection, and manages recovery procedures when machines fail, thereby maintaining both parallelism and reliability in the distributed system.
3Speed
If all graph data is stored in memory for fast access, then computation speed is improved, but memory requirements become prohibitive for graphs with trillions of edges
Solution Approach 1:
The patent transitions from a single-machine in-memory model to a distributed multi-machine model where graph data is stored across multiple machines on disk. This dimensional change from one machine's memory to many machines' storage allows the system to handle graphs with trillions of edges by distributing the data burden across the network file system while maintaining parallel access.
4Productivity
If Bellman-Ford algorithm is parallelized by propagating messages through all edges, then parallelism is achieved, but the number of iterations required becomes too large for scalability
Solution Approach 1:
The patent implements a threshold-based propagation mechanism where messages are only propagated when they improve the shortest path distance by more than a certain threshold. This local quality control prevents unnecessary iterations by stopping propagation when further improvements are unlikely, significantly reducing the number of iterations required while maintaining parallelism across the distributed system.
Data Source
AI summary
Systems and methods for checkpointing a computation distributed over multiple peer servers. On each server, sequentially storing checkpoints collectively representing a current state of the computation on that server as of a most recent checkpoint, each checkpoint having a checkpoint timestamp. When restarting a first server, rebuilding a most recent state of the first server from the checkpoints written by the first server through a most recent checkpoint having a most recent checkpoint timestamp, and requesting from each of the other peer servers updates from the most recent checkpoint timestamp time of the first server. On each server, in response to a first request for updates as of a particular time, deriving the requested updates from the state data in the server uncommitted to a checkpoint and the state data in checkpoints of the server that have a timestamp no earlier than the particular time of the first request, and providing the requested updates to the first server.