Dynamic Graph Traversal Switching Between BFS and DFS Modes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Graph database query execution faces challenges due to the combinatorial explosion effect, leading to high memory consumption and performance degradation, especially in distributed systems, where large intermediate results require extensive network communication and disk offloading, which is costly in terms of performance.
Innovation Solution
Implementing dynamic graph traversal that switches between breadth-first search (BFS) and depth-first search (DFS) modes based on available memory, using asynchronous inter-machine communication to manage intermediate results and maintain a limited memory footprint, thereby reducing the need for disk offloading and improving query execution efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If graph queries explore large parts of the graph by following edges, then connection exploration capability is improved, but memory consumption increases due to combinatorial explosion of intermediate results
Solution Approach 1:
The patent segments the graph traversal process into multiple stages (BFS stages for pattern matching, DFS stages for result generation). Each stage processes a portion of the graph and produces intermediate results that are managed separately. This segmentation allows the system to explore large graph connections while controlling memory usage by processing and discarding intermediate results in manageable chunks rather than holding all possible paths in memory simultaneously.
Solution Approach 2:
The patent dynamically switches between BFS and DFS traversal modes based on the query execution state and available resources. The system transitions from BFS (which explores breadth and generates more intermediate results) to DFS (which explores depth and consumes less memory) when memory pressure increases. This dynamic adaptation allows the system to maintain connection exploration capability while responding to changing memory conditions.
2Productivity
If distributed systems scale out to handle large graphs, then graph processing capacity is improved, but network communication overhead increases due to inter-machine coordination
Solution Approach 1:
The patent performs preliminary BFS traversal stages locally on each machine before triggering DFS stages. This preliminary action allows each node to independently process its local graph portion and generate intermediate results without immediate network communication. Only when intermediate results exceed local memory thresholds does the system initiate DFS stages that may involve remote machines, thereby minimizing unnecessary network overhead while maintaining distributed processing capacity.
3Quantity of substance
If disk offloading is used to manage large intermediate results, then memory pressure is reduced, but query execution performance degrades
Solution Approach 1:
The patent extracts and eliminates the need for disk offloading by using DFS traversal stages that generate final results directly from intermediate results in memory. Instead of spilling intermediate results to disk when memory is full, the system transitions to DFS mode which processes intermediate results and produces final query results without requiring persistent storage. This extraction of the disk offloading step maintains low memory pressure while preserving query execution performance.
Data Source
AI summary
Techniques are described for enabling in-memory execution of any-sized graph data query by utilizing both depth first search (DFS) principles and breadth first search (BFS) principles to control the amount of memory used during query execution. Specifically, threads implementing a graph DBMS switch between a BFS mode of data traversal and a DFS mode of data traversal. For example, when a thread detects that there are less than a configurable threshold number of intermediate results in memory, the thread enters BFS-based traversal techniques to increase the number of intermediate results in memory. When the thread detects that there are at least the configurable threshold number of intermediate results in memory, the thread enters DFS mode to produce final results, which generally works to move the intermediate results that are currently available in memory to final query results, thereby reducing the number of intermediate results in memory.