Distributed Storage Data Synchronization via Persistent Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage systems, data loss occurs when a network host fails during the caching period, as cached data is not written to persistent storage in time, leading to incomplete data synchronization and service disruptions.
Innovation Solution
Implementing a DAX-enabled file system with software-simulated persistent memory that directly writes data into simulated persistent memory, creating multiple copies for immediate transmission and caching, and using asynchronous writeback mechanisms to ensure data is stored in persistent storage without affecting access efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is cached in memory before being written to persistent storage, then data access efficiency is improved, but data reliability deteriorates because cached data may be lost if the system fails during the caching period
Solution Approach 1:
The data writing process is segmented into two independent paths: a synchronous path that immediately writes data to persistent storage for reliability, and an asynchronous path that writes to cache for performance. This segmentation allows the system to achieve both data reliability and access efficiency simultaneously.
Solution Approach 2:
The system performs preliminary writing to persistent storage before allowing the operation to complete, ensuring data is safely stored even if the system fails during the subsequent caching operation. This preliminary action guarantees data reliability while the async cache write provides performance benefits.
2Reliability
If data is immediately written to persistent storage, then data reliability is improved, but data access efficiency deteriorates due to increased I/O operations
Solution Approach 1:
The I/O operations are segmented into essential synchronous writes to persistent storage and optional asynchronous writes to cache. This allows the system to maintain data reliability through guaranteed persistent storage writes while improving access efficiency by deferring non-critical cache writes.
Solution Approach 2:
The system creates a copy of the data for caching purposes while the original is written to persistent storage. This copying approach allows the persistent storage write to complete for reliability, while the cache copy provides performance optimization without blocking the critical path.
3Reliability
If a remote backup network host is activated to take over services, then service continuity is improved, but service restoration time increases due to data synchronization delays
Solution Approach 1:
Data is preliminarily synchronized to the remote backup host through asynchronous replication before failover is needed. This preliminary action ensures the backup host already has current data, enabling immediate service restoration without synchronization delays during the failover event.
Solution Approach 2:
Data replication to the remote backup host continues asynchronously in the background during normal operation, maintaining continuous data synchronization. This continuous useful action ensures the backup host is always ready for failover, eliminating service restoration delays.
Data Source
AI summary
A distributed storage system and a data synchronization method are used with a network. The system includes a first network host and a second network host. A first file system directly writes data generated by the first network host into a first software-simulated persistent memory. The data in the first software-simulated persistent memory is stored into a first remote block device and cached, respectively. The cached data is stored into a first persistent storage by asynchronous writeback mechanisms. The first remote block device transmits the received data to the second software-simulated persistent memory through the network. The data transmitted to the second software-simulated persistent memory is cached, and the cached data is stored into a second persistent storage by asynchronous writeback mechanisms. The second network host replaces the first network host to provide services when the first network host is out of service.