Metadata Log Bucket Segmentation for Storage Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face inefficiencies in metadata management, particularly in handling metadata requests and updates, leading to increased latency and resource consumption due to the need to load and cache entire metadata pages for small updates or reads, which can result in performance degradation.
Innovation Solution
The proposed solution involves optimizing metadata management by recording metadata updates as deltas in a metadata log, allowing for efficient reading and writing of small portions of metadata pages without loading the entire page, and using a metadata log structure that includes an active and inactive bucket set for volatile memory to manage metadata changes, enabling direct access to the latest data without caching entire pages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If entire metadata pages are loaded and cached for small updates or reads, then metadata access reliability is ensured, but system latency increases and resource consumption increases
Solution Approach 1:
The metadata page is segmented into multiple buckets, where each bucket stores a specific portion of metadata. Instead of loading the entire page, only the required bucket is loaded from non-volatile storage to volatile memory, significantly reducing access latency while maintaining data reliability.
Solution Approach 2:
The necessary metadata information is extracted from the full metadata page and stored in a compact log structure in volatile memory. This extraction allows the system to access only the essential metadata changes without loading redundant data, improving access speed while preserving data integrity.
2Reliability
If entire metadata pages are loaded and cached for small updates or reads, then metadata access reliability is ensured, but resource consumption increases
Solution Approach 1:
The metadata page is segmented into multiple buckets, where each bucket stores a specific portion of metadata. Instead of loading the entire page, only the required bucket is loaded from non-volatile storage to volatile memory, significantly reducing access latency while maintaining data reliability.
Solution Approach 2:
The system performs partial loading of metadata by only retrieving the specific bucket containing the required metadata from non-volatile storage, rather than loading the entire metadata page. This partial action reduces memory consumption and energy usage while maintaining access reliability.
3Productivity
If metadata updates are recorded as deltas in a metadata log with active and inactive bucket sets, then read and write efficiency is improved, but device complexity increases
Solution Approach 1:
The metadata log uses dynamic bucket sets that can be switched between active and inactive states. When a bucket is being written, it belongs to the active set; when finished, it moves to the inactive set. This dynamic management allows concurrent read and write operations without complex locking mechanisms, improving efficiency while keeping the structure manageable.
Solution Approach 2:
The system maintains a log copy of metadata changes in volatile memory that mirrors the structure of non-volatile storage. This copying approach allows efficient read operations from the volatile memory log while writes are persisted to non-volatile storage, improving productivity without requiring complete structural redesign.
Data Source
AI summary
Processing a read request to read metadata from an entry of a metadata page may include: determining whether the metadata page is cached; responsive to determining the metadata page is cached, obtaining the first metadata from the cached metadata page; responsive to determining the metadata page is not cached, determining whether the requested metadata is in a metadata log of metadata changes stored in a volatile memory; and responsive to determining the metadata is the metadata log of metadata changes stored in the volatile memory, obtaining the requested metadata from the metadata log. Processing a write request that overwrites an existing value of a metadata page with an updated value may include: recording a metadata change in the metadata log that indicates to update the metadata page with the updated value; and performing additional processing during destaging that uses the existing value prior to overwriting it with the updated value.