Snapshot Copy Resume Tracking Across Object Store Endpoints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Snapshot copy operations across multiple objects in an object store are inefficient due to the lack of information about which objects have been copied, making it difficult to pause, resume, or handle failures effectively.
Innovation Solution
Implement a snapshot copy process that tracks the state of snapshots using a state machine and utilizes a metadata object diffing mechanism to identify and copy only objects not already present at the destination endpoint, allowing for resumable operations and parallel processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If snapshot copy operations are performed across multiple objects in an object store, then backup functionality is provided, but efficiency is reduced due to lack of tracking information about copied objects
Solution Approach 1:
The system performs preliminary actions by creating metadata objects at the destination endpoint before actual data copying begins. These metadata objects contain tracking information about which objects have been copied, enabling efficient resumption after failures and avoiding redundant copying operations.
Solution Approach 2:
The system implements feedback mechanisms through metadata objects that continuously track the state of snapshot copy operations. By comparing metadata at source and destination endpoints, the system receives feedback about which objects have been successfully copied and which require re-copying, improving overall operation efficiency.
2Ease of operation
If snapshot copy operations are performed across multiple objects, then backup functionality is provided, but the ability to pause and resume operations is compromised
Solution Approach 1:
The system introduces metadata objects as intermediary structures that mediate between the snapshot copy process and the tracking requirements. These metadata objects store state information about copied objects, enabling the system to pause and resume operations without requiring complex external tracking mechanisms.
3Reliability
If all objects are copied during snapshot operations, then complete backup is achieved, but time is wasted copying objects already present at destination
Solution Approach 1:
The system extracts only the necessary copying operations by comparing metadata objects at source and destination endpoints. It identifies and extracts only those objects that need to be copied, excluding objects that already exist at the destination, thereby reducing redundant copying time while maintaining backup completeness.
Data Source
AI summary
Techniques are provided for implementing a snapshot copy operation between endpoints. One or more snapshots (e.g., snapshots of an on-premise volume) is stored within a source endpoint, such as a source bucket of an object store. A post operation is executed to copy objects comprising snapshot data of a snapshot from the source endpoint to a destination endpoint. A get operation and a tracking object such as a cookie is used to track progress of copying the objects from the source endpoint to the destination endpoint. The tracking object is used to restart the copying of the objects from a point where the copying left off (e.g., in the event there is a failure) without having to restart from the beginning.