High Frequency Snapshot Hydration via LTSS and B+ Tree Indexing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current file systems are not designed for long-term storage and retrieval of large quantities of snapshots, limiting the ability to achieve rapid recovery of failover data within short Recovery Point Objectives (RPO) intervals, such as minutes or seconds, which is critical for disaster recovery scenarios.
Innovation Solution
A technique that utilizes a long-term storage service (LTSS) to store High Frequency Snapshots (HFSs) as recovery points in cloud storage, employing a key-value database to store metadata with HFS identifiers, logical timestamps, and logical offsets, and creates a searchable index metadata structure (B+ tree) for efficient retrieval and hydration of snapshots, enabling rapid recovery of failover data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If snapshots are periodically replicated to backup/archival storage systems, then data loss tolerance is reduced to support restore operations, but recovery time is increased due to periodic replication intervals
Solution Approach 1:
The system dynamically adjusts snapshot replication frequency based on data change rates and recovery requirements. High-frequency snapshots are replicated more frequently when data changes rapidly, while low-frequency snapshots use periodic replication. This dynamic approach allows the system to achieve both low data loss tolerance and fast recovery time by adapting replication intervals to actual data modification patterns.
Solution Approach 2:
The snapshot replication process is segmented into two distinct pathways: high-frequency snapshot replication for critical recent data changes, and low-frequency snapshot replication for historical data. This segmentation allows the system to prioritize replication resources for the most critical data while still maintaining comprehensive backup coverage, thereby reducing overall recovery time without sacrificing data loss tolerance.
2Duration of action of stationary object
If large quantities of snapshots are maintained for long-term storage, then data retention is improved, but file system performance and storage efficiency deteriorate
Solution Approach 1:
The storage system is segmented into two distinct layers: a file system layer for active, frequently accessed snapshots, and an object storage layer for long-term archival of low-frequency snapshots. This segmentation allows the system to maintain high performance for active operations while efficiently retaining large quantities of historical snapshots in the object storage layer without degrading overall system productivity.
Solution Approach 2:
Low-frequency snapshots are extracted from the file system and stored separately in object storage. This extraction removes the performance burden of managing large numbers of historical snapshots from the file system, allowing it to maintain high productivity for active data operations while the extracted snapshots are preserved long-term in the object storage layer.
3Speed
If snapshots are stored in object storage with metadata indexing, then retrieval speed is improved, but system complexity increases
Solution Approach 1:
A metadata indexing service acts as an intermediary between the object storage layer and the retrieval operations. This intermediary maintains indexes of snapshot metadata (including high-frequency and low-frequency snapshot identifiers, timestamps, and locations), enabling fast retrieval without requiring complex direct access mechanisms in the object storage system itself. The intermediary absorbs the complexity of rapid retrieval while keeping the object storage layer relatively simple.
Data Source
AI summary
A technique enables recovery of failover data used to generate one or more High Frequency Snapshots (HFSs) at a source and replicated to a target for storage and recovery. The target is illustratively an intermediary repository embodied as a long-term storage service (LTSS) configured to organize and store the HFSs as recovery points (RPs) in an object store. The LTSS stores a HFS identifier (ID), a logical offset in an object of the object store storing data of the HFS, and a logical timestamp associated with each replicated HFS as a key of a segment descriptor in a key-value database configured to store metadata describing the failover data of the HFS stored as one or more objects in the object store. Upon recovery of the failover data, the technique enables identification of the HFS stored in the object store and creation of a HFS index metadata structure (B+ tree) to extract the identified HFS as a RP.