Asynchronous Virtual Disk Backup Batching Mechanism
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage arrays, maintaining data redundancy is challenging due to the difficulty in determining the order of transactions across different storage cells with varying timestamps and network performance, which complicates ensuring that a redundant storage device is a true copy of the primary storage device at a specific point in time.
Innovation Solution
A batching mechanism is implemented for asynchronous backup of virtual disks, where all writes to a storage cell are logged in a batch, and a consistency point is established in the IO stream to terminate the first batch before starting the second, ensuring that writes up to this point are copied to a redundant storage device, while subsequent writes are logged in another batch.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If asynchronous backup is implemented to reduce delays during normal operation, then productivity is improved, but reliability deteriorates because it becomes difficult to ensure the redundant storage device is a true copy at a definable point in time
Solution Approach 1:
The system performs preliminary actions by establishing consistency points before the actual backup transfer. A consistency point is defined as a specific state in the IO stream where all writes up to that point are logged in a batch. This allows the system to capture a consistent snapshot of data before transferring it to the redundant storage device, ensuring reliability while maintaining asynchronous operation for productivity.
Solution Approach 2:
The batch mechanism acts as an intermediary between the IO stream and the redundant storage device. Instead of directly transferring individual writes, the system groups writes into batches that are transferred as units. This intermediary structure ensures that only completed writes are transferred together, maintaining data consistency while enabling efficient asynchronous backup operations.
2Ease of operation
If timestamps on different storage cells are used to determine transaction order, then ease of operation is improved, but measurement precision deteriorates due to clock skew and network performance variations
Solution Approach 1:
Instead of relying on potentially inaccurate timestamps to determine transaction order, the system creates a copy of the IO stream state at consistency points. This copy contains all writes that completed up to that point, providing an accurate and reliable ordering without depending on timestamp precision or clock synchronization between storage cells.
Solution Approach 2:
The IO stream is segmented into discrete batches separated by consistency points. Each batch contains a complete set of writes that completed before the consistency point. This segmentation allows the system to process and transfer data in manageable units while maintaining accurate ordering, avoiding the need for precise timestamp comparison across distributed storage cells.
Data Source
AI summary
Systems and methods for asynchronous backup of virtual disks in a distributed storage array are disclosed. An exemplary method may comprise receiving an IO stream at the virtual disks in the distributed storage array, writing data to one or more storage cells of the virtual disks in the distributed storage array, and logging completed writes in a first batch for each of the one or more storage cells. The method also comprises establishing a consistency point in the IO stream, terminating logging to the first batch for each of the one or more storage cells at the consistency point and logging completed writes after the consistency point to a new batch for each of the one or more storage cells. The method also comprises transferring writes logged in the first batch to at least one remote storage so that the at least one remote storage is crash consistent.