RDMA Shared KV Store Deduplication for Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face inefficiencies in deduplication processes, particularly in managing data blocks across multiple servers and storage devices, as they often require processor involvement for data-path operations, leading to increased latency and resource utilization.
Innovation Solution
The implementation of a distributed data storage system using a shared Key-Value (KV) store accessible via Remote Direct Memory Access (RDMA), allowing servers to deduplicate data blocks without executing code on storage controller CPUs, through in-line and background deduplication processes, and utilizing Network Interface Controllers (NICs) for hash calculations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If traditional deduplication processes are used with processor involvement for data-path operations, then data blocks can be managed across multiple servers and storage devices, but latency increases and processor resources are over-utilized
Solution Approach 1:
The patent extracts the deduplication processing from the storage controller's CPU by implementing a shared data structure accessible via RDMA. Servers perform deduplication lookups directly against the shared structure without executing code on storage controller processors, thereby removing the bottleneck and reducing latency while maintaining distributed deduplication functionality.
Solution Approach 2:
The patent introduces a shared data structure as an intermediary between servers and storage controllers. This intermediary enables servers to perform deduplication operations independently by directly accessing the shared structure through RDMA, eliminating the need for processor involvement in data-path operations and reducing system latency.
2Productivity
If processor code execution is required for deduplication operations, then data blocks can be managed across multiple servers and storage devices, but processor load increases
Solution Approach 1:
The patent extracts deduplication processing from storage controller processors by implementing a shared data structure that servers can access directly via RDMA. This removes the processor execution requirement for data-path operations, significantly reducing processor load while maintaining the ability to manage data blocks across the distributed system.
Solution Approach 2:
The patent enables servers to perform deduplication operations independently by directly accessing the shared data structure through RDMA. This self-service mechanism eliminates the need for storage controller processor involvement in data-path operations, reducing overall processor load while maintaining distributed deduplication functionality.
3Loss of time
If shared data structure is accessed via RDMA without processor code execution, then processor load is reduced and latency is minimized, but system complexity increases
Solution Approach 1:
The patent implements a universal shared data structure that serves multiple functions: it stores deduplication metadata, enables atomic operations, and provides RDMA access for servers. This multi-functional approach reduces the need for separate specialized structures and processors, managing system complexity while achieving low-latency deduplication.
Data Source
AI summary
A method for data storage includes, in a system that includes multiple servers, multiple multi-queue storage devices and at least one storage controller that communicate over a network, storing data blocks by the servers on the storage devices. A shared data structure, which is accessible to the servers using remote direct memory access and which maps hash values calculated over the data blocks to respective storage locations of the data blocks on the storage devices, is maintained. The data blocks stored on the storage device are deduplicated, by looking-up the shared data structure by the servers without executing code on a processor of the storage controller.