Distributed Graph Traversal for Top-K Path Queries at Controlled Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing graph query engines face challenges in efficiently executing shortest and cheapest path queries in distributed systems due to high memory usage and irregular access patterns, with existing algorithms like breadth-first traversal and Dijkstra's algorithm being unsuitable for large, complex property graphs.
Innovation Solution
An asynchronous distributed graph query engine that combines asynchronous pattern matching with bulk synchronous path matching algorithms, dividing queries into subjobs and using reachability maps and hop engines to manage memory and improve performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If breadth-first traversal or Dijkstra's algorithm is used for path finding, then path finding capability is achieved, but memory usage becomes excessively high and performance degrades on large property graphs
Solution Approach 1:
The patent divides the graph traversal into multiple stages and batches, processing vertices in manageable chunks rather than loading the entire graph into memory at once. This segmentation allows the system to handle large property graphs by breaking them into smaller, memory-friendly units while maintaining traversal continuity across batches.
Solution Approach 2:
The patent introduces a new dimension of processing by implementing an asynchronous distributed architecture that operates across multiple computing nodes. This distributed approach transforms the single-memory-constraint problem into a multi-node parallel processing system, effectively bypassing individual memory limitations through collective computational resources.
2Adaptability or versatility
If distributed asynchronous traversal is used, then scalability is improved, but irregular access patterns reduce execution efficiency
Solution Approach 1:
The patent implements periodic synchronization points in the asynchronous traversal where intermediate results are aggregated and processed in bulk. This periodic action allows the system to maintain asynchronous parallelism for scalability while introducing regular synchronization intervals that optimize data access patterns and improve execution efficiency through batched operations.
3Productivity
If more memory is allocated for graph storage, then query processing capability is improved, but system cost and resource consumption increase
Solution Approach 1:
The patent performs preliminary filtering and preprocessing of graph data before the main traversal operation. By pre-computing and storing only the necessary subgraphs and vertex properties required for specific queries, the system reduces the amount of memory needed during actual query execution while maintaining full query processing capability.
Data Source
AI summary
The present disclosure relates to various approaches for fast and scalable TOP K SHORTEST and CHEAPEST graph queries supporting horizontal aggregations on the group variables of a path in a distributed graph query engine. A distributed graph query processing engine may execute a graph query in a plurality of computing devices. A plurality of subjobs may be generated based at least in part on the graph query. Execution of an asynchronous pattern matching subjob of the plurality of subjobs may be initiated, and, in response to the asynchronous pattern matching subjob identifying one or more source vertices of a plurality of vertices, the execution of the asynchronous pattern matching subjob may be paused. An output context set comprising the one or more source vertices may be generated. Execution of a synchronous path matching subjob of the plurality of subjobs may be initiated, and a reachability map may be generated based at least in part on one or more matched paths between the one or more source vertices and one or more destination vertices of the plurality of vertices. The execution of the asynchronous pattern matching subjob may be resumed based at least in part on the output context set and the reachability map.