Self-Describing Replica Files for Distributed Storage Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face performance issues and slow recovery due to the use of journals for filesystem durability, which result in increased downtime and reduced lifespan of storage devices, as they require additional writes and cache flushes, and necessitate full filesystem scans for consistency checks.
Innovation Solution
The implementation of self-describing replica files with chronology data and metadata links, which allow for atomic persistence and updates, enabling efficient recovery without journal replay or full filesystem scans, thereby reducing downtime and storage wear.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If journals are used to provide filesystem durability, then crash consistency is improved, but recovery speed deteriorates due to required journal replay and full filesystem scans
Solution Approach 1:
The patent extracts the durability information from the journal system and embeds it directly into the replica files themselves. Each replica file contains chronology data and metadata links that self-describe its version and relationship to parent files, eliminating the need for external journal replay during recovery.
Solution Approach 2:
Replica files become self-describing by incorporating chronology data and metadata links within their structure. This allows them to independently verify their consistency state without requiring external journal information or full filesystem scans, enabling faster recovery.
2Reliability
If journals are used for filesystem durability, then crash consistency is improved, but storage device lifespan deteriorates due to additional writes and cache flushes
Solution Approach 1:
The patent removes the separate journal writing mechanism and integrates durability tracking directly into the replica file structure. This eliminates redundant write operations to journal files and reduces the total number of cache flushes required, thereby extending storage device lifespan.
Solution Approach 2:
The patent combines the durability tracking function previously performed by separate journal files with the replica files themselves. By embedding chronology data and metadata links within replica files, the system merges data storage and durability verification into a single operation, reducing total write amplification.
3Reliability
If full filesystem scans are performed for consistency checks, then version consistency is improved, but productivity deteriorates due to increased downtime
Solution Approach 1:
The patent extracts the consistency verification information from the entire filesystem scope and localizes it within individual replica files through embedded metadata links and chronology data. This allows consistency checks to be performed on individual files rather than requiring full filesystem scans.
Solution Approach 2:
Instead of performing excessive full filesystem scans for consistency checks, the patent applies partial action by checking only the specific replica files that need verification using their self-describing metadata. This selective approach maintains version consistency while minimizing the scope and time of consistency checks.
4Reliability
If multiple journal writes and cache flushes are performed, then durability is improved, but device wear increases
Solution Approach 1:
The patent merges the durability assurance mechanism with the data storage operation itself by embedding chronology data and metadata links within replica files. This combination eliminates the need for separate journal writes and reduces cache flush operations, thereby reducing write amplification and storage device wear while maintaining durability.
Solution Approach 2:
Replica files self-verify their durability state through embedded chronology data and metadata links, eliminating the need for external journal validation. This self-service approach reduces the number of write and flush operations required to ensure durability, thereby reducing storage device wear.
Data Source
AI summary
The embodiments relate to performing a write operation in a filesystem. Replica files embedded with respective sets of data are stored are on persistent storage local to replica nodes of the filesystem. The storage includes atomically updating the replica files to the replica nodes. Each set of data has information for self-characterizing each replica file. A map stored at persistent storage local to a remote node is updated. The map maintains information for identifying each replica file stored at each replica node.