Topological Graph Changes in Database Management Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing database management systems (DBMS) face challenges in efficiently executing graph queries due to high memory usage, irregular access patterns, and lag in replication mechanisms, which can lead to unpredictable artifacts and performance issues.
Innovation Solution
The approach involves detecting and recording topological graph changes, generating in-memory graph extensions, and applying graph changes with minimal impact on DML instructions. This includes using lightweight time-stamped records to store graph changes and arranging them in a queue for concurrent applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If graph queries are executed by translating to SQL join queries and processing with existing SQL engine, then the DBMS can leverage existing SQL processing capabilities, but the graph structure is not leveraged and neighbor traversals are inefficient
Solution Approach 1:
The system segments graph processing into two distinct paths: a SQL processing path for general queries and a graph processing path for graph-specific queries. This allows each path to be optimized independently, with the graph processing path leveraging graph data structures and traversal algorithms while the SQL path handles general relational queries efficiently.
Solution Approach 2:
The system introduces an intermediary layer that detects whether a query is a graph query or a general SQL query, and routes it to the appropriate processing path. This intermediary routing mechanism enables efficient graph queries to bypass SQL processing limitations while maintaining compatibility with existing SQL infrastructure.
2Speed
If graph data is replicated in fast-access memory to improve query speed, then query execution can be performed faster, but replication lag causes unpredictable artifacts and incorrect results
Solution Approach 1:
The system performs preliminary actions by capturing graph changes at the time they occur and queuing them for later application to the in-memory graph representation. This ensures that the graph structure is updated with all necessary changes before queries are executed, eliminating replication lag issues while maintaining fast query performance.
Solution Approach 2:
The system implements feedback mechanisms where graph changes are continuously monitored, captured, and applied to the in-memory representation. This feedback loop ensures that the in-memory graph remains consistent with the actual graph state, preventing incorrect results while maintaining speed advantages.
3Reliability
If each DML instruction reconstructs the graph to properly update graph data, then the graph remains consistent, but the update process negatively impacts DML performance and slows the DBMS considerably
Solution Approach 1:
The system performs preliminary capture of graph changes during DML execution and queues them for batch processing. This preliminary action separates the change capture phase from the graph update phase, allowing DML instructions to execute quickly without immediately reconstructing the entire graph, while still maintaining consistency through queued updates.
Solution Approach 2:
The system segments graph updates into two phases: change capture phase where DML instructions are processed and changes are recorded, and graph update phase where the in-memory graph is reconstructed with all captured changes. This segmentation allows DML execution to proceed quickly while graph consistency is maintained through batched updates.
4Reliability
If transactions must see their own uncommitted changes before replication, then transactional consistency is achieved, but transactions are limited to using only the replica where changes were made
Solution Approach 1:
The system performs preliminary capture of graph changes during transaction execution and queues them for application to the in-memory graph representation. This preliminary action allows transactions to access and see their own uncommitted changes immediately in the in-memory graph, while the actual replication to other replicas occurs asynchronously without limiting transaction access.
Data Source
AI summary
Techniques are described for applying topological graph changes and traversing the modified graph. In an implementation, a set of compile processes schedules the graph changes caused by a DML (Data Manipulation Language) statement. Based on the requested graph operation in a received query for graph, a set of graph operation processes generate extensions to the graph that capture the changes to the graph by the DML. The received graph operation(s) are then performed by traversing both the existing graph and the generated extensions.