Snapshot Copy Tracking for Resumable Object-Store Transfers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Snapshot copy operations across multiple objects in an object store are inefficient due to the lack of information about which objects have been copied, making it difficult to pause, resume, or handle failures effectively.
Innovation Solution
A snapshot copy process that tracks the state of snapshots using a state machine and identifies objects to be copied by comparing metadata indicators, allowing for resumable operations and efficient copying of only missing data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If snapshot copy operations are performed across multiple objects in an object store, then data backup and recovery functionality is provided, but the operation becomes inefficient due to lack of tracking information about copied objects
Solution Approach 1:
The system performs preliminary actions by creating a tracking object before the snapshot copy operation to record the state of each object. This tracking object is initialized with all objects in the source endpoint, and as objects are copied, their states are updated. This preliminary setup enables efficient resumption and tracking without requiring complex real-time tracking mechanisms during the actual copy operation.
Solution Approach 2:
The system implements feedback mechanisms through the tracking object that continuously monitors and records the copy state of each object. The tracking object provides feedback information about which objects have been copied and their current states, enabling the system to make informed decisions about resumption, failure handling, and optimization of subsequent copy operations.
2Ease of operation
If complete snapshot copy operations are performed without tracking, then simplicity is maintained, but the ability to resume or handle failures is compromised
Solution Approach 1:
The system segments the snapshot copy operation into individual object-level operations, each with its own state tracking. By dividing the overall copy operation into discrete unit operations that can be independently tracked and managed, the system achieves both simplicity in execution and reliability in failure handling. Each object's copy state is independently recorded, allowing selective resumption without complexity.
Solution Approach 2:
The system creates a copy of the operational state information in the tracking object rather than copying all data objects. This copying of metadata about object states enables the system to track and manage copy progress without actually copying the entire snapshot data, maintaining simplicity while enabling reliable failure handling and resumption.
3Quantity of substance
If all snapshot data objects are copied without identifying missing data, then completeness is ensured, but copying efficiency is reduced due to redundant data transfer
Solution Approach 1:
The system performs preliminary tracking of object copy states before the actual data transfer. By recording which objects have been successfully copied and their current states in advance, the system can identify missing data objects and transfer only those, eliminating redundant data transfer while ensuring completeness through systematic tracking of all source objects.
Solution Approach 2:
The system applies partial action by copying only the subset of objects that are missing or have changed, rather than copying all objects in the snapshot. The tracking object enables the system to identify exactly which objects need to be copied, performing only the necessary partial copy operation that maintains completeness while optimizing transfer efficiency.
Data Source
AI summary
Techniques are provided for implementing a snapshot copy operation between endpoints. One or more snapshots (e.g., snapshots of an on-premise volume) is stored within a source endpoint, such as a source bucket of an object store. A post operation is executed to copy objects comprising snapshot data of a snapshot from the source endpoint to a destination endpoint. A get operation and a tracking object such as a cookie is used to track progress of copying the objects from the source endpoint to the destination endpoint. The tracking object is used to restart the copying of the objects from a point where the copying left off (e.g., in the event there is a failure) without having to restart from the beginning.