Batched Subquery Execution in Distributed Graph Queries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing graph database systems face challenges in efficiently executing subqueries in distributed asynchronous environments, leading to high memory consumption and performance issues due to the inability to effectively manage and coordinate subqueries during graph traversals.
Innovation Solution
Implementing a graph processing engine that executes subqueries in two phases: pattern matching and post-processing, with non-correlated subqueries executed before the parent query and correlated subqueries executed in a batched manner after all correlated variables are instantiated, using asynchronous distributed traversals to control memory consumption and improve scalability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If subqueries are executed in a distributed asynchronous environment using traditional approaches, then query functionality is supported, but memory consumption increases and performance deteriorates
Solution Approach 1:
The query execution is segmented into distinct phases: pattern matching phase and post-processing phase. Subqueries are executed during the pattern matching phase when intermediate results are available, while post-processing operations are deferred. This segmentation allows memory-efficient execution by processing queries in stages rather than loading all data into memory at once.
Solution Approach 2:
Non-correlated subqueries are executed in advance before the parent query's post-processing phase. By executing these subqueries preliminarily during the pattern matching phase, the system avoids needing to hold all intermediate results in memory simultaneously, thus reducing memory consumption while maintaining query functionality.
2Measurement precision
If subqueries are executed for every parent query value in correlated subqueries, then query accuracy is maintained, but execution time increases significantly
Solution Approach 1:
Multiple instantiations of correlated subqueries are merged into a single batched execution. Instead of executing the subquery separately for each parent query value, the system collects all necessary intermediate results and executes the subquery once with batched input, maintaining result accuracy while dramatically reducing execution time.
Solution Approach 2:
The system maintains continuous execution flow by executing correlated subqueries in a batched manner during the pattern matching phase rather than interrupting the main query execution repeatedly. This continuous approach preserves query accuracy while eliminating the time loss associated with repeated start-stop execution patterns.
3Adaptability or versatility
If distributed query execution is implemented across multiple machines, then scalability is improved, but coordination overhead increases
Solution Approach 1:
The coordination overhead for distributed query execution is extracted and centralized in the query compiler. The compiler analyzes the query plan, identifies subqueries, determines their execution timing (pattern matching phase vs. post-processing phase), and generates optimized execution code. This extraction eliminates the need for complex runtime coordination between distributed machines during query execution.
Solution Approach 2:
The query compiler acts as an intermediary that resolves coordination complexity before execution. By performing static analysis and generating optimized execution plans that specify exactly when and where subqueries should be executed, the compiler mediates between the distributed execution model and the need for coordination, allowing scalable execution without runtime coordination overhead.
Data Source
AI summary
A graph processing engine is provided for executing a graph query comprising a parent query and a subquery nested within the parent query. The subquery uses a reference to one or more correlated variables from the parent query. Executing the graph query comprises initiating execution of the parent query, pausing the execution of the parent query responsive to the parent query matching the one or more correlated variables in an intermediate result set, generating a subquery identifier for each match of the one or more correlated variables, modifying the subquery to include a subquery aggregate function and a clause to group results by subquery identifier, executing the modified subquery using the intermediate result set and collecting subquery results into a subquery results table responsive to pausing execution of the parent query, and resuming execution of the parent query using the subquery results table.