Checkpoint Metafile Format for Directory Restore Across Storage Tiers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional storage systems struggle with restoring data at a granularity between full volume and single file restore, fail when objects are stored across different storage tiers, and lack efficient methods for directory-level restore operations that can handle hierarchical directory structures.
Innovation Solution
A data format utilizing base and instance metafiles for tracking progress and checkpoint support during storage operations, enabling multi-level directory restore across various storage tiers, allowing selective restoration of directories and subdirectories, and supporting on-demand access and efficient rehydration of data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If full volume restore operation is performed, then data can be restored, but existing volume data is destroyed and restoration cost increases
Solution Approach 1:
The restore operation is segmented into directory-level units rather than treating the entire volume as a single unit. The system identifies and restores only specific directories and subdirectories that need recovery, leaving other parts of the volume intact. This segmentation enables selective restoration without destroying existing data.
Solution Approach 2:
The invention extracts only the necessary data portions (specific directories and subdirectories) from the snapshot for restoration, rather than extracting the entire volume. This extraction approach allows recovery of lost data while preserving existing volume data that does not need restoration.
2Ease of operation
If single file restore operation is performed, then specific data can be restored, but directory structure and subdirectories cannot be restored together
Solution Approach 1:
The system merges the capabilities of single file restore with directory-level restore operations. By processing restore operations at the directory level while maintaining the ability to select specific directories, it combines the precision of single-file restoration with the structural integrity of directory-level restoration, enabling both specific data recovery and complete directory restoration in one operation.
Solution Approach 2:
The restore operation mechanism is designed to be universal, handling multiple restore scenarios (single file, directory, subdirectory) through a unified approach. The same directory-level restore mechanism can restore individual files within a directory or entire directory structures, providing multi-functional capability.
3Loss of substance
If data is restored from archival storage tier, then cost is reduced, but restoration time increases due to data movement requirements
Solution Approach 1:
The system performs preliminary identification of data locations across storage tiers before initiating restoration. By pre-mapping which data resides in archival storage and which is already in standard storage, the system can optimize the restoration sequence to minimize data movement time while maximizing cost savings from archival tier utilization.
Solution Approach 2:
The restoration process dynamically adapts to the distribution of data across storage tiers. The system adjusts restoration strategies based on real-time detection of data locations, moving data from archival to standard storage only when necessary and optimizing the restoration sequence to reduce overall restoration time while maintaining cost efficiency.
4Ease of repair
If restore operation is restarted after failure, then interrupted restoration can be completed, but already restored data may be re-processed causing duplicate charges
Solution Approach 1:
The system implements feedback mechanisms through checkpoint tracking that monitor the restoration progress and record which data has been successfully restored. This feedback information is used to guide subsequent restore operations, allowing the system to identify already restored data and skip it during resumption, thereby preventing duplicate processing and charges.
Solution Approach 2:
The system discards (skips) already restored data during operation resumption and only processes unrestored data. By tracking restoration state through checkpoints, the system recovers the operation at the correct point without re-processing completed tasks, eliminating duplicate charges while maintaining the ability to complete interrupted restoration operations.
Data Source
AI summary
Techniques are provided for a data format for efficient management of checkpoint support. The data format corresponds to a base metafile and a set of instance metafiles used to track storage operations such as a directory restore operation. The base metafile and the set of instance metafiles can be used to resume the storage operation from where the storage operation left off in the event of a failure. The base metafile and the set of instance metafiles can be used to track progress of the storage operation processing objects stored within an object store of a cloud storage environment.