Graph Database Storage Engine Separation for Deep Traversal
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current graph database systems face complexity in engineering and poor deep graph traversal performance, particularly due to the separation of graph structure and property engines leading to inefficient data access and caching.
Innovation Solution
A method and system that separate the graph structure and property in a graph database using a storage engine and compute engine, where the storage engine uses a key-value structure to store both, and the compute engine caches the graph structure for short-term and long-term queries, employing techniques like compressed sparse rows and adjacency lists to improve performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a separate storage engine and compute engine architecture is used, then engineering complexity is reduced and large-scale distributed systems can be constructed, but deep graph traversal performance deteriorates due to frequent data exchange between engines and lack of hierarchical caching
Solution Approach 1:
The patent segments the system into storage engine and compute engine that can operate independently, allowing them to be deployed separately while maintaining clear interface definitions. This segmentation enables reduced engineering complexity and facilitates large-scale distributed system construction.
Solution Approach 2:
The patent implements preliminary action by pre-caching graph structure data in the compute engine before actual query execution. The system performs advance data preparation and caching operations, so that when deep graph traversal queries are executed, the compute engine can access previously cached structure data rather than requesting it from the storage engine during query execution, thereby improving traversal performance.
2Ease of manufacture
If graph structure and graph property are stored together in a unified key-value structure, then storage simplicity is maintained, but access efficiency deteriorates because the frequently accessed graph structure cannot be cached separately
Solution Approach 1:
The patent segments the stored data into two distinct parts: graph structure data and graph property data. The storage engine maintains separate storage paths and access mechanisms for these two types of data, allowing the graph structure to be cached in the compute engine while graph properties remain in the storage engine, thereby enabling selective caching and improving access efficiency.
Solution Approach 2:
The patent extracts the graph structure data from the unified key-value storage and places it in a separate cache within the compute engine. This extraction allows the frequently accessed graph structure to be kept locally in memory, reducing access time for traversal operations while the storage engine maintains simplified key-value storage for the remaining data.
3Reliability
If uncached graph topology data is accessed from storage engine, then data consistency is maintained, but query latency increases due to repeated data exchange between storage engine and compute engine
Solution Approach 1:
The system performs preliminary action by caching graph structure data in advance in the compute engine. This pre-caching operation ensures that when queries are executed, the compute engine can access locally cached data rather than repeatedly requesting it from the storage engine, significantly reducing query latency while maintaining data consistency through controlled update mechanisms.
Solution Approach 2:
The compute engine acts as an intermediary between the storage engine and the query processing layer. It maintains a local cache of graph structure data, mediating between the persistent storage and the computational operations. This intermediary role allows the system to maintain data consistency while reducing the frequency of direct storage engine access, thereby lowering query latency.
Data Source
AI summary
The present application relates to a method and system for storing data in a graph database. The method includes: separating a key-value structure of the storage engine and separately storing a graph structure and a graph property of graph data; and reading, by the compute engine, the graph structure, caching the graph structure for a short term in a single query request scenario, and caching the graph structure for a long term in a read-only analysis scenario. The present application resolves problems that engineering is complex and deep graph traversal performance is poor when graph data is processed, reduces engineering complexity, and improves the deep graph traversal performance.