Graph Database Partitioning for Query Processing Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Graph databases face inefficiencies in query processing across multiple machines due to the need for increased storage space, leading to slower query execution compared to single-machine processing, especially when handling large datasets.
Innovation Solution
A method and system for partitioning a graph database that determines node types and places new nodes into partitions based on property values for mapped nodes and connection patterns for connected nodes, minimizing the number of partitions involved in queries and transactions, utilizing a domain-specific partitioning scheme that prioritizes efficient query processing across machines.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If graph database is partitioned across multiple machines to handle larger datasets, then storage capacity increases, but query processing speed decreases
Solution Approach 1:
The graph database is segmented into multiple partitions distributed across different machines. Each partition contains a subset of nodes and edges, allowing the system to handle larger datasets by dividing the storage burden across multiple machines while maintaining independent query processing within each partition.
Solution Approach 2:
The patent applies local quality by creating hot partitions and cold partitions with different characteristics. Hot partitions are optimized for frequent query access with smaller size and faster processing, while cold partitions are optimized for storage efficiency. This allows frequently accessed data to be processed quickly on single machines while less frequently accessed data is stored across multiple machines.
2Adaptability or versatility
If more partitions are created to distribute data across machines, then storage scalability improves, but query processing complexity increases
Solution Approach 1:
The system performs preliminary action by pre-computing and storing metadata about each partition, including the nodes and edges it contains. This metadata is cached and used to quickly determine which partitions are relevant for a given query, avoiding the need to scan all partitions and reducing query processing complexity as the system scales.
Solution Approach 2:
The patent introduces an intermediary layer in the form of a coordinator or router that receives queries, determines the relevant partitions using stored metadata, and routes queries to the appropriate machines. This intermediary abstracts the complexity of multi-machine query processing from the actual query execution, making the system more scalable.
3Quantity of substance
If data is distributed across multiple machines, then storage capacity increases, but cross-machine query processing becomes slower
Solution Approach 1:
The patent creates hot partitions that are kept smaller and optimized for fast query processing on single machines, while cold partitions are larger and optimized for storage. By keeping frequently accessed data in hot partitions that can be processed locally without cross-machine communication, the system minimizes query execution time for important operations while still providing large storage capacity through cold partitions.
Solution Approach 2:
The system performs preliminary action by pre-computing query results or intermediate results and caching them in the appropriate partitions. This allows frequently accessed query results to be retrieved locally from cache rather than requiring cross-machine data retrieval, significantly reducing query execution time for repeated operations.
Data Source
AI summary
A method and apparatus for partitioning a graph database that includes nodes connected to one another is presented. The method entails determining a node type indicating a type of node that is to be used as mapped nodes, and upon receiving a new node, placing the new node in a partition according to one or more property values associated with the new node if it is a mapped node, and placing the new node in a partition according to its connections to other nodes if it is not a mapped node. The apparatus includes modules containing computer-executable instructions to perform the above method.