RDMA Cache Coherency for NVMe Storage Clusters
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional storage systems face difficulties in maintaining local cached data coherency among multiple host devices accessing the same logical storage devices, especially when not all hosts have RDMA capability, leading to performance issues and configuration challenges in large-scale clusters.
Innovation Solution
Implementing RDMA-based local cached data coherency using NVMe over InfiniBand, where each host device can directly access and invalidate cache metadata of other host devices, maintaining coherency without remote threads, and utilizing MPIO drivers to manage connectivity and caching operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional caching methods are used without RDMA, then host devices can access storage systems, but local cached data coherency cannot be maintained among multiple host devices
Solution Approach 1:
The patent introduces RDMA (Remote Direct Memory Access) as an intermediary mechanism that enables direct memory access between host devices through the storage system. This intermediary allows cache metadata to be shared and synchronized across multiple hosts without requiring complex peer-to-peer communication protocols, thereby maintaining data coherency while managing system complexity.
Solution Approach 2:
The patent combines the caching functionality with the existing storage system infrastructure by integrating cache metadata management into the storage system's host connectivity information. This merging approach allows multiple host devices to share a unified view of cache state through the storage system, eliminating the need for separate distributed cache coherence protocols.
2Reliability
If RDMA is implemented for all host devices, then local cached data coherency is maintained, but implementation cost and configuration complexity increase
Solution Approach 1:
The patent applies RDMA selectively only to host devices that require local caching functionality, rather than requiring all host devices to have RDMA capability. Host devices without RDMA can still access the storage system through conventional methods, while RDMA-enabled hosts benefit from enhanced cache coherency. This local quality approach allows the system to optimize for performance where needed without mandating expensive hardware upgrades across the entire infrastructure.
3Reliability
If remote threads are used to maintain cache coherency, then data consistency can be achieved, but IO processing performance decreases
Solution Approach 1:
The patent replaces the mechanical system of remote threads with RDMA-based direct memory access operations. Instead of using software threads that require context switching and CPU intervention to maintain cache coherency, the system uses hardware-supported RDMA operations that can directly read and write cache metadata in the background without blocking IO processing threads, thereby maintaining data consistency while preserving high IO performance.
Data Source
AI summary
A first host device establishes connectivity to a logical storage device of a storage system. The first host device obtains from the storage system host connectivity information identifying at least a second host device that has also established connectivity to the logical storage device, caches one or more extents of the logical storage device in a memory of the first host device, and maintains local cache metadata in the first host device regarding the one or more extents of the logical storage device cached in the memory of the first host device. In conjunction with processing of a write operation of the first host device involving at least one of the one or more cached extents of the logical storage device, the first host device invalidates corresponding entries in the local cache metadata of the first host device and in local cache metadata maintained in the second host device.