Data Restore Module for Parallel Downloads and Persistent Checkpoints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Restoring a large data footprint from a backup storage can take several hours, requiring efficient management of network resources and avoiding network bottlenecks, while also needing a checkpointing mechanism to track progress and avoid duplicate Input/Output operations during interruptions.
Innovation Solution
A data restore system that includes a backup data storage with metadata and object stores, utilizing concurrent data block downloads and a checkpointing mechanism to track progress and minimize duplicate I/O operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If sequential data restoration is used, then network resources are conserved, but recovery time objective (RTO) increases and network pipes are not fully utilized
Solution Approach 1:
The patent segments the backup storage into metadata store and object store, and further segments data into data blocks with unique block identifications. This segmentation enables parallel processing of metadata retrieval and data block downloads, fully utilizing network pipes while maintaining organized data restoration.
Solution Approach 2:
The system performs preliminary actions by first retrieving metadata information containing URLs for data blocks before initiating data download. This preliminary metadata retrieval enables concurrent data block downloads to proceed efficiently without waiting for sequential processing.
2Productivity
If checkpointing mechanism is not implemented, then system complexity is reduced, but duplicate I/O operations occur during interruptions
Solution Approach 1:
The checkpointing mechanism implements feedback by continuously tracking the progress of restore operations and persisting this information. When interruptions occur, the system reads the checkpoint information to resume operations from the correct position, preventing duplicate I/O operations and maintaining restoration efficiency.
3Productivity
If metadata and data blocks are fetched separately, then data integrity is maintained, but network pipes are not fully engaged and RTO increases
Solution Approach 1:
The system ensures continuity of useful action by concurrently performing metadata retrieval and data block downloads. The network pipes remain continuously engaged as data blocks are downloaded using URLs from metadata without interruption, maximizing network utilization while maintaining data integrity through the established fetch sequence.
Data Source
AI summary
A data restore system is provided. The data restore system includes a backup data storage configured to store data for a client and a data restore module configured to receive a restore trigger from the client and to initiate restore operation for selected data from the backup data storage in response to the received trigger. The data restore module is further configured to receive information regarding the selected data to be restored and access a metadata store to receive metadata information for the selected data and provide the metadata information and the downloaded data blocks to a controller to facilitate sorting of the downloaded data blocks based on the files they belong to and store the downloaded restored data to a target data storage. The data restore module is further configured to interact with the checkpointing module to track the progress of restore operation in persistent storage and to minimize rework when restore operation is restarted from interrupt.