Distributed Graph Subquery Inlining and Streaming
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies face challenges in efficiently executing subqueries in distributed asynchronous graph queries, particularly in controlling memory consumption and coordinating subqueries across multiple machines, which leads to performance and scalability issues.
Innovation Solution
The implementation of efficient in-memory distributed subqueries that operate with limited memory consumption, utilizing asynchronous distributed graph traversals and subquery co-planning to remove the need for data movement between machines, and employing batching and inlining techniques to optimize subquery execution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If threads execute complete subqueries when reaching subqueries in the execution tree, then query processing is simplified, but memory consumption cannot be limited and coordination cost increases
Solution Approach 1:
The patent segments subquery execution into two distinct phases: a streaming phase that processes subqueries without materializing results, and a post-processing phase that handles aggregations and ordering. This segmentation allows the system to process subqueries efficiently without consuming excessive memory, as only necessary data is materialized in the second phase rather than everything at once.
Solution Approach 2:
The patent performs preliminary actions by executing subqueries in a streaming fashion during the first phase, processing results as they become available rather than waiting to materialize everything. This preliminary streaming processing reduces memory consumption by avoiding the need to hold all subquery results in memory simultaneously before post-processing.
2Ease of operation
If threads execute complete subqueries independently, then query processing is simplified, but coordination cost increases in distributed systems
Solution Approach 1:
The patent segments subquery execution into streaming phase and post-processing phase, where the streaming phase can execute independently on different threads without coordination, while post-processing is handled centrally. This reduces coordination overhead in distributed systems by allowing independent execution during the first phase.
Solution Approach 2:
The patent introduces an intermediary mechanism where subquery results are streamed through a common channel rather than requiring direct coordination between threads. This intermediary streaming mechanism simplifies coordination by providing a standardized interface for result delivery without requiring complex inter-thread communication protocols.
3Reliability
If subqueries are executed for each intermediate result, then query correctness is ensured, but performance deteriorates due to repeated data movement
Solution Approach 1:
The patent performs preliminary streaming execution of subqueries during the first phase, where subqueries are executed once and results are streamed forward. This preliminary action eliminates the need to re-execute subqueries for each intermediate result, significantly improving performance while maintaining correctness through the streaming approach.
Solution Approach 2:
The patent ensures continuity of useful action by streaming subquery results continuously through the execution plan rather than executing subqueries repeatedly. This continuous streaming approach maintains query correctness while eliminating redundant subquery executions, thereby improving overall performance.
4Ease of operation
If data movement between machines is required for subqueries, then subquery execution is simplified, but scalability decreases in distributed systems
Solution Approach 1:
The patent introduces streaming as an intermediary mechanism that allows subquery results to flow through the distributed system without requiring complete data movement between machines. This intermediary streaming approach enables subquery execution while maintaining scalability by processing data in a streaming fashion rather than moving entire result sets.
Data Source
AI summary
A graph processing engine is provided for executing a graph query comprising a parent query and a subquery nested within the parent query. The subquery is an existential subquery, uses a reference to one or more correlated variables from the parent query, is inlined in the parent query pattern matching, does not have a post-processing phase, does not contain any global aggregation operations, uses a reference to at most one non-correlated variable, and does not include any filters on a non-correlated variable. Executing the graph query comprises initiating execution of the parent query, responsive to the parent query matching the one or more correlated variables in an intermediate result set, executing the subquery by applying a neighbor pattern matching operator that checks for existence of an edge, and resuming execution of the parent query based on results of the neighbor pattern matching operation.