Hybrid In-Memory BFS-DFS Graph Query Runtime
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing relational database systems face challenges in efficiently processing graph queries due to suboptimal use of graph structure, high memory usage, and irregular access patterns, particularly when handling homogeneous and heterogeneous graphs, as they do not leverage breath-first traversal (BFS) effectively and struggle with the memory footprint and locality of depth-first traversal (DFS).
Innovation Solution
A hybrid in-memory BFS-DFS approach is integrated into relational database systems, combining the strengths of both traversal methods to optimize memory locality and reduce memory usage, allowing for pipelined execution and efficient processing of graph queries by using compressed sparse rows (CSR) representation and specialized match operators that explore a limited number of neighbors in each iteration.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If BFS traversal is used to process graph queries, then memory footprint is reduced and pipeline execution is enabled, but memory locality deteriorates and access patterns become irregular
Solution Approach 1:
The patent segments the graph traversal process into multiple iterations, where each iteration processes a limited number of neighbors (controlled by parameter K). This segmentation allows BFS to be executed in manageable chunks that fit within memory constraints while maintaining pipeline execution capability, resolving the contradiction between reduced memory footprint and maintained memory locality.
Solution Approach 2:
The patent implements periodic action by repeatedly executing the neighbor exploration process in discrete iterations. Each iteration periodically processes up to K neighbors, then pauses to allow pipeline execution and memory management. This periodic approach enables BFS to achieve reduced memory footprint while maintaining acceptable memory locality through controlled, rhythmic access patterns.
2Ease of manufacture
If DFS traversal is used to process graph queries, then memory locality is improved, but memory footprint increases and pipeline execution is hindered
Solution Approach 1:
The patent segments DFS into iterative batches where each batch processes a limited scope of neighbors. By controlling the number of neighbors explored per iteration (parameter K), the system maintains memory locality benefits of DFS while preventing unbounded memory growth, thus resolving the contradiction between memory locality and memory footprint.
Solution Approach 2:
The patent introduces dynamic control over traversal depth and breadth through the parameter K, which adjusts the number of neighbors processed in each iteration. This dynamic approach allows the system to adapt between DFS-like memory locality and BFS-like memory footprint control, resolving the contradiction by making the traversal behavior flexible rather than fixed.
3Adaptability or versatility
If SQL join operations are used for graph queries, then existing SQL engine infrastructure is utilized, but graph structure is not leveraged and hash table construction is repeated for every join
Solution Approach 1:
The patent introduces an intermediary graph processing layer that sits between the SQL engine and the graph data. This intermediary leverages the graph structure (adjacency lists, vertex/edge relationships) to optimize neighbor traversals before presenting results to the SQL engine. It eliminates repeated hash table construction by using direct graph structure lookups, thus resolving the contradiction between SQL compatibility and query execution efficiency.
Solution Approach 2:
The patent replaces the mechanical SQL join mechanism (which uses hash tables and full table scans) with a graph-specific traversal mechanism that directly exploits the graph structure. Instead of using generic join operations, the system uses graph-aware neighbor exploration that respects vertex and edge relationships, eliminating the inefficiency of repeated hash table construction while maintaining SQL engine compatibility through the intermediary layer.
4Reliability
If graph queries explore all intermediate results, then complete pattern matching is achieved, but memory usage explodes
Solution Approach 1:
The patent segments the exploration of intermediate results into iterative batches, processing only up to K neighbors per iteration. This segmentation ensures that complete pattern matching is eventually achieved (maintaining reliability) while memory usage remains bounded by the segment size, preventing memory explosion.
Solution Approach 2:
The patent applies partial action by processing only a portion of the neighbors (up to K) in each iteration rather than all neighbors at once. This partial exploration approach maintains pattern matching completeness over multiple iterations while controlling memory usage at each step, resolving the contradiction between complete matching and memory constraints.
Data Source
AI summary
An in-memory graph query runtime is integrated inside a database management system and is capable of performing simple patter-matching queries against homogeneous graphs. The runtime efficiently combines breadth-first (BFS) and depth-first (DFS) neighbor traversal algorithms to achieve a hybrid runtime that takes the best from both sides. As a result, the hybrid runtime is able to process arbitrarily large queries with a fixed amount of memory, optimizing for memory locality.