Owner-Based Metadata Page Staging for Lower-Latency Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems face inefficiencies in managing metadata updates due to increased contention and latency caused by holding exclusive locks on metadata pages while waiting for staging areas to become full, leading to adverse performance impacts.
Innovation Solution
A novel approach that collects and accumulates metadata pages in a staging area without holding exclusive locks, allowing access during the accumulation period, and performs large sequential writes out of band, reducing contention and latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If exclusive locks are held on metadata pages while waiting for staging areas to become full, then data consistency is maintained, but contention and latency increase
Solution Approach 1:
The system performs preliminary actions by writing metadata updates to a staging area before they are fully committed. The metadata update is staged with a staging sequence number, allowing the write operation to begin immediately without waiting for the staging area to be full, thus reducing latency while maintaining consistency through subsequent commit operations.
Solution Approach 2:
A staging area acts as an intermediary between the metadata update source and the final committed storage. This intermediary buffer allows updates to be queued and processed in batches without blocking the incoming write operations, reducing contention on the metadata pages while ensuring data consistency through ordered commit operations.
2Reliability
If exclusive locks are held on metadata pages during staging accumulation, then data integrity is ensured, but performance deteriorates
Solution Approach 1:
The locking mechanism transitions from static (holding locks throughout the entire staging process) to dynamic (acquiring and releasing locks at specific checkpoints). The system acquires locks only when committing staged metadata updates to the final destination, rather than holding them during the entire accumulation period, thus improving performance while maintaining data integrity.
Solution Approach 2:
The system performs preliminary staging of metadata updates without holding exclusive locks, then performs a subsequent commit operation with proper locking. This separates the high-frequency staging operations from the lower-frequency committed operations, improving overall system performance while ensuring data integrity during commits.
3Device complexity
If metadata pages are staged sequentially, then simplicity is maintained, but write throughput decreases
Solution Approach 1:
The staging area is segmented into multiple regions or buckets that can be written to independently and in parallel. Different metadata updates can be staged to different segments simultaneously, increasing write throughput while maintaining the conceptual simplicity of the staging process. Each segment can be managed independently with its own lock and commit sequence.
Data Source
AI summary
In at least one embodiment, processing can include: receiving a write that writes content to a logical address; recording an entry for the write in a log; flushing the entry from the log; recording, in a metadata log, an update to a metadata page; flushing the update from the metadata log; generating an updated version of the metadata page including the update; staging the updated version of the metadata page to a staging area of a plurality of partitions, comprising: selecting a partition of the staging area based: on partition owner of the partition, a page owner of the metadata page, a type of the metadata page, and a second type of the first partition; selecting a zone of the partition based on a processing node performing said staging; and writing the updated version of the metadata page to the zone of the partition.