Deduplicated File System Fast Failover via Memory Pool Architecture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deduplicated file systems face challenges in achieving fast failover and recovery due to the need to maintain extensive metadata, leading to lengthy recovery times that result in client timeouts and disruptions, especially when traditional hard disk drives are used for metadata access.
Innovation Solution
Implementing a memory pool architecture with both volatile huge pages and persistent memory devices across cluster nodes, allowing for the separation and rapid retrieval of metadata, thereby reducing the reliance on hard disk drives for recovery and enabling faster restarts and failovers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If traditional hard disk drives are used for metadata access, then storage capacity is sufficient, but recovery time becomes excessively long (10-20 minutes)
Solution Approach 1:
The patent replaces mechanical hard disk drive access with volatile memory (huge pages) and persistent memory devices for metadata storage. This substitution eliminates mechanical access limitations and achieves significantly faster metadata retrieval speeds, reducing recovery time from 10-20 minutes to 2-5 minutes.
Solution Approach 2:
The patent introduces a hierarchical memory architecture with multiple tiers (volatile memory, persistent memory, and storage pool) to access metadata from different dimensional layers. This allows the system to retrieve frequently accessed metadata from high-speed memory tiers while maintaining capacity for extensive metadata across all tiers.
2Reliability
If extensive metadata is maintained for deduplication operations, then deduplication functionality is improved, but recovery complexity increases
Solution Approach 1:
The patent segments the extensive metadata into different storage tiers based on access frequency and criticality. Frequently accessed metadata (such as fingerprints index) is stored in volatile memory and persistent memory, while less frequently accessed metadata is stored in the storage pool. This segmentation simplifies recovery by allowing the system to restore critical functionality from high-speed memory tiers without requiring immediate access to the entire metadata set.
Solution Approach 2:
The patent maintains pre-computed metadata structures (fingerprints index, tree structures) in high-speed memory tiers before failures occur. This preliminary preparation ensures that upon failure, the system can immediately retrieve these pre-organized structures from memory rather than having to reconstruct them from raw data, significantly reducing recovery complexity and time.
3Speed
If metadata is isolated from user namespace, then recovery speed improves, but system complexity increases
Solution Approach 1:
The patent introduces a memory pool architecture as an intermediary layer between the user namespace and the storage pool. This memory pool manages volatile memory and persistent memory resources, providing a unified interface for metadata access while isolating metadata management from user processes. The intermediary handles memory allocation, persistence, and recovery operations, simplifying the overall system while enabling fast recovery.
Data Source
AI summary
A memory tier is established in a cluster system having a deduplicated file system. The memory tier includes memory pages configured as huge pages, where writes to the huge pages are exported in a device file that is outside of a user process namespace within which processes of the deduplicated file system run. At least a portion of metadata generated by the deduplicated file system is written to the memory tier. The portion of metadata includes an index of fingerprints corresponding to data segments stored by the deduplicated file system to a storage pool. A determination is made that an instance of the deduplicated file system has failed. A new instance of the deduplicated file system is started to recover file system services by loading the index of fingerprints from the device file.