Snapshot Indexing With Deferred B+ Tree Retrieval
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional file systems and backup/archival systems are not designed for long-term storage and retrieval of large quantities of snapshots, leading to increased retrieval times as the number of snapshots grows due to the need to traverse extensive metadata for recent data access.
Innovation Solution
An indexing technique using a B+ tree with a large branching factor is employed to create an index data structure that efficiently retrieves snapshot data from an archival storage system, allowing deferred indexing of immutable snapshots and leveraging log-structured formats for write-heavy operations, enabling efficient retrieval independent of the number of snapshots.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If conventional file systems process data indexing together with storage layout and data storage to support immediate access requests, then recently stored data can be immediately retrieved, but retrieval time increases as the number of snapshots increases due to traversing greater metadata
Solution Approach 1:
The patent segments the indexing process from the storage process by using log-structured formats that separate data writing from index construction. Data is written sequentially to logs first, then indexed separately in background processing, eliminating the need to traverse extensive metadata during data retrieval operations.
Solution Approach 2:
The patent performs preliminary actions by pre-processing data into log-structured formats during the storage phase, organizing data with inherent structural properties that enable efficient later retrieval without requiring extensive metadata traversal at access time.
2Reliability
If conventional backup/archival systems are designed for immediate on-demand data availability, then data can be retrieved quickly, but the systems cannot efficiently maintain large quantities of snapshots for long-term storage
Solution Approach 1:
The patent implements dynamic snapshot management where the system can efficiently create, maintain, and retrieve a large number of snapshots over time. The log-structured format allows snapshots to be added dynamically without degrading retrieval performance, enabling the system to handle massive numbers of snapshots while maintaining data availability.
Solution Approach 2:
The patent changes the fundamental parameters of how snapshots are stored and indexed by using log-structured formats with deferred indexing. This parameter change allows the system to scale to large quantities of snapshots while maintaining efficient retrieval through bounded retrieval times that do not increase with snapshot count.
3Ease of operation
If conventional file systems traverse extensive metadata to support live access to recent data, then immediate access is enabled, but retrieval time increases as the number of snapshots increases
Solution Approach 1:
The patent extracts the metadata traversal requirement from the data retrieval process by using log-structured formats where data location information is inherently embedded in the log structure itself, rather than requiring separate metadata traversal. This extraction eliminates the performance degradation associated with increasing snapshot counts.
Data Source
AI summary
An indexing technique provides an index data structure for efficient retrieval of a snapshot from a long-term storage service (LTSS) of an archival storage system. The snapshot is generated from typed data of a logical entity, such as a virtual disk (vdisk). The data of the snapshot is replicated to a frontend data service of the LTSS sequentially and organized as one or more data objects for storage by a backend data service of LTSS in an object store of the archival storage system. Metadata associated with the snapshot (i.e., snapshot metadata) is recorded as a log and persistently stored on storage media local to the frontend data service. The snapshot metadata includes information describing the snapshot data, e.g., a logical offset range of a snapshot of the vdisk and, thus, is used to construct the index data structure. Notably, construction of the index data structure is deferred until after the entirety of the snapshot data has been replicated and received by the frontend data service.