File Metadata Layout With Relative References for Append-Only Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional file system metadata structures are incompatible with append-only storage devices like zoned namespace (ZNS) due to their zoned architecture, leading to increased latency and computational waste, and existing solutions require separate storage devices, which limits scalability and efficiency.
Innovation Solution
Organize file system metadata with a write buffer to append layer by layer, using relative references instead of absolute references, enabling atomic write operations and compatibility with append-only storage devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If conventional file system metadata structures are used on append-only storage devices, then storage capacity is increased, but write latency increases and operational efficiency deteriorates
Solution Approach 1:
The metadata structure is segmented into multiple layers (e.g., layer 0, layer 1, layer 2) where each layer contains specific types of metadata. This segmentation allows the system to append only the necessary layer to the append-only storage device, rather than writing the entire metadata structure, thereby reducing write latency while maintaining full storage capacity.
Solution Approach 2:
A translation layer is introduced as an intermediary between the conventional file system metadata structure and the append-only storage device. This translation layer converts absolute references in metadata to relative references, enabling the metadata to be stored in an append-only manner without requiring random writes or overwrites, thus resolving the latency issue while preserving storage capacity.
2Measurement precision
If conventional file system metadata structures with absolute references are used, then metadata accuracy is maintained, but compatibility with append-only storage deteriorates
Solution Approach 1:
The reference mechanism in metadata is changed from absolute references to relative references. This parameter change allows the metadata to be stored in an append-only fashion on ZNS devices while maintaining the ability to accurately locate and access metadata entries through relative positioning within the appended layers.
3Ease of operation
If FTL is used to support file system activity, then storage functionality is improved, but device latency and cost increase
Solution Approach 1:
The FTL functionality is extracted and replaced by the layer-based metadata structure directly integrated with the append-only storage device. The translation layer performs the necessary reference conversions and metadata management functions that were previously handled by FTL, eliminating the need for a separate FTL component and reducing associated latency and costs.
4Loss of time
If ZNS interface is used to eliminate FTL complexity, then device latency is reduced, but file system compatibility deteriorates
Solution Approach 1:
The translation layer serves as an intermediary that bridges the ZNS interface and conventional file systems. It translates between the layer-based metadata structure required by ZNS devices and the traditional file system metadata format, enabling file system compatibility while maintaining the low-latency benefits of the ZNS interface.
Solution Approach 2:
The metadata reference mechanism is changed from absolute to relative references, and the metadata is organized into appendable layers. This parameter change enables direct integration with ZNS devices while the translation layer maintains compatibility with conventional file system operations, resolving the compatibility issue.
Data Source
Figure 1
Figure 2A
Figure 2B
AI summary
The techniques disclosed herein enable systems to store file system metadata on append-only storage devices by the introduction of a metadata layout and relative references. To achieve this, a system analyzes a file metadata structure to select an approach to storing the metadata structure on the storage device. If the metadata structure is above a threshold size, it is separated into layers that are then appended to the storage device from lowest to highest. An offset is retrieved for appended layers and provided to upper-level layers that refer to them until the full structure is stored. For storing smaller pieces of metadata that refer to each other, the system configures upper-level nodes with relative references that express the location of metadata relative to the node. This enables atomic write operations for smaller metadata structures and/or portions of larger structures.