Metadata Destaging with Read Locks and Tablet Generation IDs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing metadata management in data storage systems involves complex write locks that cause delays and resource-intensive processing, particularly due to non-idempotent metadata changes and the need for markers to prevent duplicate writes, leading to reduced system performance and potential corruption.
Innovation Solution
Implementing a technique that uses read locks instead of write locks for destaging metadata changes, accompanied by tablet generation identifiers (TGIDs) to manage metadata, allowing concurrent reads and writes, and avoiding duplicate writes by limiting changes to those with higher TGIDs, thus simplifying processing and reducing contention.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If write locks are used to hold pages during destaging, then data consistency is ensured, but read request latency increases and system performance decreases
Solution Approach 1:
The patent segments the locking mechanism into read locks and write locks, allowing read operations to acquire read locks that do not block other read operations, while write operations acquire write locks. This segmentation enables concurrent read access during destaging operations, reducing read request latency while maintaining data consistency through proper lock management.
Solution Approach 2:
The patent introduces an intermediary mechanism (the lock manager and tablet generation identifiers) that mediates between read and write operations. The lock manager coordinates read and write locks, while TGIDs serve as intermediaries to track which metadata changes have been applied to pages, allowing the system to ensure consistency without blocking reads during destaging.
2Reliability
If write locks are used to prevent concurrent access during destaging, then data consistency is maintained, but device complexity and processing resource consumption increase
Solution Approach 1:
The patent segments the complex lock management into distinct read lock and write lock pathways with different behaviors. Read locks allow concurrent access and do not block destaging, while write locks provide exclusive access when needed. This segmentation simplifies the overall management complexity compared to a unified write-lock approach.
Solution Approach 2:
The patent implements self-service mechanisms where the system automatically tracks which metadata changes have been applied to pages using tablet generation identifiers stored with the pages. This automated tracking eliminates the need for complex manual marker management and reduces processing resource consumption.
3Reliability
If markers are placed in tablets to prevent duplicate writes, then corruption from non-idempotent changes is avoided, but processing resources are consumed and complexity increases
Solution Approach 1:
The patent implements a self-service mechanism where tablet generation identifiers automatically track which metadata changes have been applied to pages. The system uses these TGIDs to determine whether a page needs updating, eliminating the need for manual marker placement and removal. This automated approach prevents duplicate writes of non-idempotent changes while reducing complexity and processing resource consumption.
Solution Approach 2:
The patent introduces tablet generation identifiers as intermediaries between the metadata change tracking system and the actual write operations. These TGIDs mediate the decision-making process by providing a simple comparison mechanism to determine whether a page should be updated, replacing the complex marker management system.
4Reliability
If write locks are used to ensure all metadata changes are accounted for, then data consistency is maintained, but productivity and throughput decrease
Solution Approach 1:
The patent segments the destaging process into phases with different locking requirements. During the initial destaging phase, read locks are used which allow concurrent access and do not block productivity. The locking is selectively applied only where necessary to ensure consistency, rather than blocking all operations throughout the entire destaging process.
Solution Approach 2:
The patent implements dynamic lock management where the type and duration of locks are adjusted based on the specific operation being performed. Read locks are used during destaging to maintain consistency while allowing concurrent operations, whereas write locks are used only when actual modifications occur. This dynamic approach optimizes productivity while maintaining metadata consistency.
Data Source
AI summary
A technique of managing metadata includes receiving metadata changes into active tablets. When the active tablets are filled, the active tablets are changed to frozen tablets, whereupon previously frozen tablets are freed and changed to the active tablets. The technique further includes destaging metadata changes from the frozen tablets to pages in backend storage under read locks. Tablet generation identifiers (TGIDs) are assigned to successive generations of tablets, and a TGID of the frozen tablets is stored with the pages that are written as part of destaging from the frozen tablets.