Parallel Graph Database Traversal with Thread and Buffer Queues
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In large-scale graph databases, query efficiency for target elements is low due to inefficient resource utilization, as existing methods sequentially execute operators, leading to high waiting times and low system resource utilization.
Innovation Solution
The method involves allocating threads to each operator in a graph traversal statement, creating buffer queues between adjacent operators to execute and share results in parallel, and transmitting token data to trigger subsequent operators, ensuring efficient execution and resource allocation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If operators are executed sequentially in graph traversal, then execution order is simple and easy to control, but query efficiency is low and system resource utilization is low
Solution Approach 1:
The graph traversal operation is divided into multiple independent operators (e.g., V, out, has, properties), and each operator is executed by a separate thread. This segmentation allows parallel execution of operators, improving query efficiency while maintaining manageable complexity through modular design
Solution Approach 2:
The execution mechanism transitions from static sequential execution to dynamic parallel execution. Threads are allocated to operators based on their execution order, and buffer queues are created between adjacent operators to enable dynamic data sharing, allowing the system to adapt to different graph traversal patterns
2Loss of time
If operators are executed sequentially, then resource allocation is simple, but waiting time is high and system resource utilization is low
Solution Approach 1:
Threads are allocated to operators in advance based on the execution order of operators. Buffer queues are pre-created between adjacent operators to store execution results. This preliminary preparation eliminates waiting time during execution, as each thread can immediately access the buffer queue when needed
Solution Approach 2:
Buffer queues serve as intermediaries between adjacent operators, enabling efficient data sharing. The buffer queue receives execution results from one operator and makes them available to the next operator, eliminating the need for complex inter-process communication and reducing waiting time
3Productivity
If parallel execution is implemented with threads and buffer queues, then query efficiency and resource utilization are improved, but implementation complexity increases
Solution Approach 1:
The thread allocation mechanism and buffer queue creation pattern are designed to be universal and can be applied to any graph traversal statement regardless of the number or type of operators. The same template-based approach works for different query patterns, simplifying implementation while maintaining high resource utilization
Data Source
Figure 1A~1B
Figure 2
Figure 3A~3B
AI summary
The present disclosure discloses a method for traversing a graph database. A graph traversal statement is obtained (S101). At least two operators contained in the graph traversal statement and an execution order of the at least two operators are determined. A thread is allocated to each operator. A buffer queue is created for each two adjacent operators. For each two adjacent operators, an operation of a former operator is executed (SI02) by a thread corresponding to the former operator of the two adjacent operators. The execution result of the former operator is written to the buffer queue corresponding to the two adjacent operators. From the buffer queue, the execution result of the former operator of the two adjacent operators is read in parallel by a thread corresponding to a latter operator of the two adjacent operators, to execute an operation of the latter operator.