Persistent Memory Buffering for Low-Latency Cluster Writes
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing method for storing write data into a storage cluster is lengthy and complex, resulting in delays due to network instabilities, resource contention, and internal network delays within the storage cluster.
Innovation Solution
The data storage method involves dividing the storage process into two stages: first storing data in a persistent memory and then sequentially writing it into the storage cluster, using a persistent memory with a constructed index table to manage data storage conditions and support efficient reading and writing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If write data is directly stored into the storage cluster through the central node, then the storage process is straightforward, but the storage delay is lengthy and complex due to network instabilities, resource contention, and internal network delays
Solution Approach 1:
The patent segments the storage process into two independent phases: Phase 1 stores data locally in the DPU's persistent memory and provides immediate acknowledgment to the server, while Phase 2 asynchronously writes data to the storage cluster. This segmentation isolates the server from storage cluster delays, improving both reliability and reducing perceived storage delay.
Solution Approach 2:
The persistent memory in the DPU acts as an intermediary buffer between the server and the storage cluster. It temporarily holds write data and enables the server to receive immediate feedback without waiting for the slow storage cluster operations, effectively decoupling the server from storage delays while maintaining data integrity.
2Reliability
If the server waits for storage confirmation before issuing next write request, then data consistency is maintained, but storage throughput is reduced due to the lengthy feedback process
Solution Approach 1:
The system performs preliminary action by storing data in the DPU's persistent memory and providing immediate acknowledgment to the server before the data is actually written to the storage cluster. This preliminary storage action allows the server to proceed with subsequent operations without waiting for cluster confirmation, thereby improving throughput while maintaining consistency through the durable persistent memory.
Solution Approach 2:
The persistent memory enables continuous write operations by the server without interruption for storage confirmations. While Phase 1 continues seamlessly with immediate acknowledgments, Phase 2 handles the asynchronous completion of storage operations, ensuring both continuity of server operations and eventual data consistency.
3Reliability
If all write data is immediately written to the storage cluster, then data durability is ensured, but the storage process becomes complex and delayed due to multi-replica distribution and feedback loops
Solution Approach 1:
The patent extracts the complexity of multi-replica distribution and feedback loops from the server-side storage process and relocates it to the DPU's Phase 2 operations. The server only needs to interact with the simple Phase 1 interface, while the DPU handles the complex storage cluster interactions, thereby reducing overall system complexity and improving performance.
Solution Approach 2:
The persistent memory creates a copy of the write data locally in the DPU, providing an immediate durable storage copy that satisfies the server's durability requirements. This local copy enables the server to proceed without waiting for the complex cluster write operations, while the original data copy in the cluster is still being written in the background.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
The present disclosure provides a data storage method and device. The method includes: receiving a write data request transmitted by a server, where the write data request is used to request to store target data into a storage cluster, and the storage cluster is used to store and manage data to support writing and reading of the data; storing the target data into a persistent memory in a data processing unit, and dynamically recording, in a constructed index table, a data storage condition in the persistent memory, where the index table supports the server to read all or part of the target data; and sequentially reading the target data from the persistent memory and writing into the storage cluster until all the target data indicated by the write data request is written. According to the present disclosure, data storage is divided into two stages by using the persistent memory, where the first stage is to store the target data in the persistent memory, and the second stage is to take out the target data from the persistent memory and store the target data into the storage cluster, which can implement efficient storage of the target data.