Sparse Metadata Segment Tree for Backup Workloads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current backup solutions face high space and metadata overhead costs, especially in high-frequency backups and incremental replication, due to inefficient metadata processing and storage, which is exacerbated by the wear on solid-state disks from frequent metadata updates.
Innovation Solution
Implementing a sparse metadata segment tree structure to reduce metadata processing and storage overhead by caching only updated metadata segments, using a Merkle tree for efficient data segment fingerprinting and aligning I/O operations with existing data segment boundaries to minimize read-modify-write operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional metadata segment trees are used to track backup data, then complete data coverage is achieved, but metadata overhead and storage space cost increase significantly
Solution Approach 1:
The patent applies local quality by making metadata segments sparse rather than dense. Instead of creating metadata segments that cover all data segments uniformly, the system creates metadata segments only where needed to track changes. This allows the metadata structure to have different densities in different regions - sparse in unchanged areas and dense in changed areas - thereby reducing overall metadata overhead while maintaining complete data coverage tracking capability
Solution Approach 2:
The patent segments the metadata tracking structure into variable-sized metadata segments that can be independently created and managed. Rather than using a fixed uniform metadata structure, the system divides the backup space into segments and creates metadata segments only for regions containing changed data. This segmentation approach allows the system to cover complete data space while minimizing the quantity of metadata by only creating segments where necessary
2Productivity
If frequent metadata updates are performed for high-frequency backups, then backup freshness is improved, but SSD wear increases
Solution Approach 1:
The patent extracts only the changed portions of data into separate metadata segments rather than updating entire metadata structures. When a backup operation occurs, the system identifies only the modified data blocks and creates metadata segments solely for these changed regions. This extraction approach allows frequent backups to be performed while minimizing write operations to metadata storage, thereby reducing SSD wear even as backup frequency increases
Solution Approach 2:
The patent implements periodic consolidation of sparse metadata segments. Instead of continuously writing metadata updates at every backup operation, the system accumulates changes and periodically consolidates the sparse metadata segments into a unified structure. This periodic action reduces the frequency of metadata write operations to SSD, extending device lifespan while still maintaining the capability to support high-frequency backup operations
3Measurement precision
If differencing operations are performed on file system trees to identify changes, then change detection accuracy is improved, but I/O overhead increases
Solution Approach 1:
The patent performs preliminary action by pre-organizing backup data into fixed-size data segments with associated metadata segments before differencing operations are needed. Each data segment is assigned a unique identifier and stored with its metadata in a structured format. This preliminary organization allows subsequent change detection to operate on pre-segmented data rather than entire file systems, significantly reducing I/O overhead while maintaining accurate change detection capability through the segment identifier comparisons
Data Source
AI summary
Embodiments are directed to a method of minimizing latency and input/output (I/O) operations in a data storage system by defining a sparse metadata segment tree to identify changed data blocks, wherein a full version of the tree is stored in a memory and modified versions of the tree are stored in cache memory, and using the sparse metadata segment tree to perform at least one data storage application including file verification, file replication, file restores, and file system snapshots.