File System Recovery via Partitioned Metadata Segmentation
Solution Overview
Problem
Large-scale data storage systems face challenges in fault tolerance and recovery due to the potential loss of entire collections when a part of the storage system fails, leading to long recovery times and system unavailability.
Innovation Solution
The system is partitioned into multiple independent, fault-contained units, allowing for localized failure management and quick recovery of affected partitions using a restore-at-replica process, where each partition includes metadata for the entire file system, enabling self-identification and independent operation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If data is stored as a single large collection to ease management and improve deduplication, then ease of operation and deduplication efficiency are improved, but reliability and recovery time are worsened when failures occur
Solution Approach 1:
The patent divides the storage system into multiple independent partitions, each capable of operating autonomously. When a failure occurs, only the affected partition needs to be recovered while others remain operational. This segmentation resolves the contradiction by maintaining management simplicity at the system level while introducing fault isolation at the partition level, thus improving reliability without sacrificing ease of operation.
2Productivity
If data is stored as a single large collection, then deduplication efficiency is improved, but recovery time is worsened when failures occur
Solution Approach 1:
By segmenting the storage into independent partitions that each maintain their own deduplication domains, the system allows parallel recovery operations. When a failure occurs, only the affected partition needs to be restored rather than the entire collection, dramatically reducing recovery time while preserving deduplication efficiency within each partition.
3Manufacturing precision
If the collection is taken offline to repair data or metadata, then data integrity is improved, but system availability is worsened
Solution Approach 1:
The partitioned architecture enables independent repair operations on failed partitions while other partitions remain online and accessible. This segmentation allows the system to maintain data integrity through proper repair procedures without requiring complete system shutdown, thus preserving system availability during maintenance operations.
Solution Approach 2:
Each partition contains its own metadata and can perform self-diagnosis and self-repair operations. When a partition fails, it can be recovered independently using its own metadata structures without requiring the entire system to be taken offline, enabling continuous operation of healthy partitions while maintaining data integrity in the recovered partition.
4Quantity of substance
If more storage devices are added to increase capacity, then storage capacity is improved, but reliability is worsened due to increased failure likelihood
Solution Approach 1:
By organizing storage devices into independent partitions where each partition's metadata is self-contained, the system isolates failures to specific partitions. This segmentation allows the addition of more storage devices to increase capacity while limiting the impact of any single device failure to only its associated partition, thus maintaining overall system reliability despite increased scale.
Data Source
AI summary
According to one embodiment, recovering file systems without a replica is described. In one example, a method includes discovering an error in a first storage partition of the file system containing a current version of a namespace metadata, finding a second storage partition containing a most recent copy of the namespace metadata, and restoring the found copy of the namespace metadata from the second storage partition to the first storage partition for use as a current version of the namespace metadata.