Metadata Cache for Distributed Storage Reliability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In computing systems with virtual machines, maintaining metadata to prevent data loss from storage devices requires frequent access operations, leading to inefficiencies and potential errors, especially in distributed systems where multiple changes occur with each user operation affecting real storage devices.
Innovation Solution
The approach involves reading substantially all metadata associated with data recovery into memory when a virtual storage device is made available, allowing changes to be made without accessing real storage devices for read operations and writing metadata only when necessary to ensure data integrity in case of failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If metadata is maintained on real storage devices to protect against data loss, then data reliability is improved, but the number of storage access operations increases leading to system inefficiency
Solution Approach 1:
The patent introduces a metadata cache as an intermediary layer between the file system and real storage devices. This cache stores metadata in memory, allowing the system to access metadata without frequent reads from and writes to real storage devices. The cache acts as a mediator that reduces the number of storage access operations while maintaining data reliability through selective flushing of metadata changes to the storage device.
2Productivity
If metadata is read into memory to reduce storage access operations, then system efficiency is improved, but memory usage increases
Solution Approach 1:
The patent applies local quality by caching only metadata in memory rather than all data. The metadata cache selectively stores file system metadata structures (such as inodes, directory entries, and allocation information) in memory, while leaving actual file data on storage devices. This approach optimizes memory usage by storing only the necessary metadata portion locally in memory, reducing the memory footprint compared to caching all data.
3Reliability
If multiple metadata changes are made with each user operation, then data integrity is maintained, but the number of write operations to storage devices increases
Solution Approach 1:
The patent implements preliminary action by pre-allocating and pre-structuring metadata in the cache before actual data operations occur. The metadata cache is initialized with the necessary file system structures and metadata formats, allowing subsequent operations to make changes in memory without immediate writes to storage devices. This preliminary preparation enables batch processing of metadata changes, reducing the frequency of write operations to storage devices while maintaining data integrity through controlled flushing.
Data Source
AI summary
In a reliable distributed computing system, techniques where user actions or events affecting real system devices (RSD's), or causing the system to make changes to metadata, require fewer accesses to RSD's. When a virtual system device (VSD) is mounted, substantially all the metadata, or at least that associated with data recovery, is read into memory. Changes to that metadata can be made substantially without any metadata accesses to RSD's. No metadata accesses to RSD's are made for read operations, as the RSD would remain unchanged. Metadata is flushed from memory upon write operations. Metadata about data locations can be maintained in memory, so as to be available if any RSD becomes corrupted or otherwise comes to lose data.