Object-Store Snapshot Service Migration to Reduce Cloud Egress
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing archival storage systems incur substantial cloud egress costs and workload processing inefficiencies when replicating data from a cloud cluster to an on-premises snapshot storage service, primarily due to the design of file systems that are not optimized for long-term storage of large quantities of snapshots.
Innovation Solution
A service migration technique is employed to move a snapshot storage service from a primary site to a secondary site during a planned failover event, involving the storage of snapshot data and metadata in an object store, followed by instantiation at the secondary site to resume operations, while handling pending replication transfers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data is replicated from cloud cluster to on-premises snapshot storage service, then data availability is improved, but cloud egress costs increase substantially
Solution Approach 1:
The patent uses object store replication to create copies of snapshot data across multiple geographic locations. Instead of replicating data from cloud to on-premises, the system stores snapshots as objects in a cloud object store and uses the object store's built-in geographic replication capability to distribute copies across regions, eliminating costly cloud egress while maintaining data availability.
Solution Approach 2:
The object store acts as an intermediary between the cloud cluster and on-premises storage. Snapshots are first stored as objects in the object store, which then serves as a shared repository that both cloud and on-premises systems can access. This intermediary approach eliminates direct cloud-to-on-premises data transfer and the associated egress costs.
2Speed
If file systems are used for long-term storage of snapshots, then immediate access requests are supported, but storage efficiency and retrieval performance deteriorate
Solution Approach 1:
The patent changes the storage parameter from traditional file system format to object store format. Object stores are optimized for long-term archival storage with different performance characteristics than file systems. This parameter change allows the system to efficiently store large quantities of snapshots for long-term retention while maintaining retrieval capability when needed, as object stores are specifically designed for this use case rather than for rapid sequential access patterns.
Solution Approach 2:
The patent segments the storage system into two distinct layers: hot storage for active file system operations and cold storage in the object store for archival snapshots. This segmentation allows each layer to be optimized for its specific function - file systems for immediate access and object stores for efficient long-term storage - rather than trying to make a single system excel at both.
3Reliability
If snapshot storage service is migrated during planned failover, then service continuity is improved, but migration complexity increases
Solution Approach 1:
The snapshot storage service is designed as a stateless service that stores all data as objects in the object store. This copying approach means the service itself doesn't need to be physically migrated - it can be re-instantiated at the secondary site and immediately access the copied snapshot objects already present in the object store, greatly simplifying the failover process.
Solution Approach 2:
The system performs preliminary actions by continuously maintaining snapshot objects in the object store before failover is needed. During planned failover, the snapshot data is already prepared and available in the object store at the secondary site, so the service can be quickly re-instantiated without complex data transfer or synchronization operations during the actual failover event.
Data Source
AI summary
A service migration technique moves (migrates) a snapshot storage service of an archival storage system from a primary site to a secondary site in accordance with a planned failover event. The snapshot storage service is configured to provide storage and retrieval of large amounts of snapshots (e.g., recovery points) of application workloads stored as objects on one or more buckets of an object store. As part of finalization of the recovery points (RPs) at the primary site, the snapshot storage service stores snapshot data and metadata of the RPs (e.g., as service state) in the object store. The snapshot storage service may then be migrated to the secondary site, wherein migration involves terminating the service at the primary site and instantiating the service at the secondary site. The instantiated storage service instance may then rebuild its service state from the object store, and resume storage and retrieval operations directed to RPs serviced by the instance.