Lockless Distributed Storage with NVRAM Cache and RDMA
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing distributed data storage systems face challenges in maintaining data integrity and consistency across multiple servers without centralized control or direct communication, particularly in managing data blocks and stripes in a highly distributed and redundant manner.
Innovation Solution
The implementation of a method using Non-Volatile Random-Access Memory (NVRAM) cache and remote direct memory access (RDMA) to accumulate and transfer data blocks, forming stripes, while preventing inconsistencies by querying and updating shared data structures using atomic compare-and-swap commands, allowing servers to maintain system-wide data integrity without locks or direct server communication.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If centralized control is used to manage data integrity and consistency across multiple servers, then data consistency is improved, but system complexity and single point of failure risk increase
Solution Approach 1:
Each server independently manages its own data blocks by querying and updating shared data structures using atomic compare-and-swap commands. Servers autonomously ensure data consistency without requiring centralized control or coordination with other servers, eliminating the need for complex centralized management while maintaining reliability
Solution Approach 2:
A shared data structure acts as an intermediary between servers, storing metadata about data block locations. Servers use atomic compare-and-swap commands to read and update this shared structure, enabling coordinated data management across distributed servers without direct server-to-server communication or centralized control
2Reliability
If direct server communication is used to coordinate data writes, then data consistency is improved, but network overhead and communication latency increase
Solution Approach 1:
Each server independently queries and updates the shared data structure using atomic compare-and-swap commands without needing to communicate with other servers. This self-service approach eliminates inter-server communication overhead while maintaining data consistency through local atomic operations on shared metadata
Solution Approach 2:
The patent replaces the mechanical approach of direct server-to-server communication with a memory-based approach using atomic compare-and-swap commands on a shared data structure. This substitution eliminates network communication latency while achieving the same data consistency goal through atomic memory operations
3Reliability
If locks are used to prevent data inconsistencies, then data integrity is improved, but throughput and system performance decrease
Solution Approach 1:
The patent replaces the mechanical locking mechanism with atomic compare-and-swap commands on a shared data structure. This substitution allows multiple servers to simultaneously query and update data block locations without blocking each other, maintaining data integrity through atomic operations while enabling parallel processing and high throughput
Solution Approach 2:
The system uses dynamic atomic compare-and-swap commands that allow servers to check and update data block locations in a single atomic operation. This dynamic approach replaces static locks with flexible atomic memory operations that can execute concurrently across multiple servers without blocking, improving both integrity and throughput
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach enables high-performance, highly distributed redundant storage by ensuring data integrity and consistency across multiple servers, eliminating the need for centralized control and direct server communication, and optimizing storage operations such as writing, reading, and parity calculations.
Implementation Method 1
accumulating data blocks by a server in a Non-Volatile Random-Access Memory (NVRAM) cache that is accessible to the servers and to the storage devices, by sending the data blocks from the server to the NVRAM cache using remote direct memory access
Implementation Method 2
making a second query to the shared data structure using remote direct memory readout, so as to re-obtain the storage location of the data block; and aborting writing of the data block if the storage location differs between the first and second queries
Data Source
AI summary
A method for data storage includes, in a system that includes multiple servers and multiple multi-queue storage devices that communicate over a network, accumulating data blocks by a server in a Non-Volatile Random-Access Memory (NVRAM) cache that is accessible to the servers and to the storage devices, by sending the data blocks from the server to the NVRAM cache using remote direct memory access, so as to form one or more stripes. The stripes are transferred from the NVRAM cache to respective server-specific queues on the storage devices. Another server is prevented from causing inconsistency in the data blocks, by querying a data structure, which is shared and modified by the servers using remote direct memory access, and which specifies storage locations of the data blocks.