Distributed Graph Database RDMA Communication Layer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current techniques for storing and processing graph data across multiple devices suffer from high latency and scalability issues due to cross-device computation, which limits their ability to handle large datasets efficiently.
Innovation Solution
A distributed graph database utilizing a Remote Direct Memory Access (RDMA)-based communication layer and a shared memory layer with Fast Remote Memory (FaRM) to enable low-latency messaging and transactional processing across a cluster of devices, allowing for efficient distribution and execution of graph data structures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If graph data is stored and processed across multiple devices using current techniques, then data storage capacity is increased, but processing latency increases and scalability is limited
Solution Approach 1:
The patent segments the graph database into multiple partitions distributed across different devices in a cluster. Each device stores a portion of the graph data (vertices and edges) in its local memory, allowing parallel processing while maintaining data locality. This segmentation enables the system to scale storage capacity across multiple devices without proportionally increasing cross-device communication overhead.
Solution Approach 2:
The patent introduces an RDMA (Remote Direct Memory Access) communication layer as an intermediary between devices. This intermediary enables direct memory access between devices without involving the CPU or operating system of remote devices, dramatically reducing communication latency. The RDMA layer acts as a mediator that allows fast data transfer while minimizing processor intervention and system call overhead.
2Loss of time
If graph data is partitioned to avoid cross-device computation, then processing latency is reduced, but scalability is limited
Solution Approach 1:
The patent implements dynamic data partitioning and load balancing mechanisms that adapt to changing workload patterns and data distribution. The system can dynamically repartition graph data across devices based on access patterns, data size, and computational requirements. This dynamic approach allows the system to scale efficiently while maintaining low latency by keeping frequently accessed data local and distributing less critical data across the cluster.
Solution Approach 2:
The patent transitions from traditional single-device or simple multi-device architectures to a multi-dimensional distributed architecture using RDMA. This adds the dimension of high-speed inter-device communication, enabling the system to scale across multiple devices while maintaining performance characteristics similar to single-device systems. The RDMA dimension allows the system to overcome the traditional trade-off between partitioning and scalability.
3Productivity
If a cluster of devices is used for distributed processing, then data storage and processing capability are increased, but coordination overhead and complexity increase
Solution Approach 1:
The patent implements self-service mechanisms where each device autonomously manages its local data partitions and processing tasks. Devices independently handle local queries and computations without requiring constant coordination with other devices. The system uses decentralized conflict resolution and automatic data replication strategies that reduce the need for complex centralized coordination, allowing each device to serve itself while contributing to the overall distributed system.
Data Source
AI summary
A distributed graph database that enables scaling and efficient processing is described. The distributed graph database can, for example, scale up to petabytes of data to enable transactional processing of graph data with low latency and low processing overhead. The distributed graph database can include a cluster of devices and a remote direct memory access (RDMA)-based communication layer to perform low latency messaging between devices of the cluster of devices. Additionally, the distributed graph database can include a shared memory layer that provides one or more data structures, a transaction layer to facilitate query processing, and a graph database layer stored in computer-readable media and executed on a processor to implement a graph data model. In at least one example, the graph data model can be mapped to the one or more data structures.