On-demand Volume Restore Using Cloud Block Map Caching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing snapshot restore processes from remote object stores to on-demand volumes result in prolonged application downtime due to the unavailability of data until the entire restore process is complete, leading to hours of inaccessibility and potential timeouts.
Innovation Solution
Implement an on-demand restore process that provides clients with access to both already restored and not yet restored data by caching metadata and using a cloud block map to cache not yet restored data within local storage, allowing low latency access during the restore process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional snapshot restore processes are used from remote object stores, then data is restored to on-demand volumes, but application downtime is prolonged due to unavailability of data until the entire restore process is complete
Solution Approach 1:
The system performs preliminary actions by creating an on-demand volume and populating it with metadata and cloud block map information before the actual data restore is complete. This allows the volume structure and indexing to be prepared in advance, so that when data blocks are restored, they can be immediately accessed without waiting for the entire restore process to finish.
Solution Approach 2:
The restore process is segmented into independent operations: metadata restoration, cloud block map population, and data block restoration. Clients can access already-restored metadata and cloud block map data while data block restoration continues in the background, allowing partial data availability during the restore process.
2Loss of time
If data is made accessible during the restore process, then application downtime is reduced, but data integrity may be compromised if clients access not yet restored data
Solution Approach 1:
The cloud block map serves as an intermediary layer between clients and the restore process. It tracks the restore status of each data block and provides clients with information about which blocks are available and which are still being restored. This intermediary mechanism allows clients to access available data while preventing access to incomplete or inconsistent data, thus maintaining data integrity while enabling partial availability.
3Quantity of substance
If snapshots are stored in remote object stores for backup, then storage costs are reduced, but restore speed is slowed due to remote access
Solution Approach 1:
The system performs preliminary actions by restoring metadata and cloud block map information to local storage before data block restoration begins. This allows the indexing structure to be available locally, enabling faster access to restore status information and reducing the need for repeated remote object store accesses during the restore process.
Solution Approach 2:
The restore process is segmented such that metadata and cloud block map restoration can proceed in parallel with data block restoration. By separating these operations, the system can utilize local storage for indexing operations while simultaneously retrieving data blocks from remote storage, improving overall restore throughput.
Data Source
AI summary
Techniques are provided for caching data during an on-demand restore using a cloud block map. A client may be provided with access to an on-demand volume during a restore process that copies backup data from a snapshot within a remote object store to the on-demand volume stored within local storage. In response to receiving a request from the client for a block of the backup data not yet restored from the snapshot to the on-demand volume, the block may be retrieved from the snapshot in the remote object store. The block may be cached within a cloud block map stored within the local storage as a cached block. The client may be provided with access to the cached block.