Volatile Memory Metadata Buffering for Storage Write Performance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional storage systems face limitations in accommodating IO traffic due to the expensive and limited nature of non-volatile memory used for journals, which cannot handle more than a few seconds of IO traffic, leading to inefficiencies in metadata management and write performance.
Innovation Solution
A method where each data container in volatile memory is associated with a metadata page, with metadata changes being written to these containers until they are full, then switched to a new set, using a hash function to determine target containers and merge changes with stored metadata pages from the storage array, allowing efficient aggregation and writing of metadata changes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If non-volatile memory is used for journals to ensure data persistence, then data safety is improved, but storage capacity is limited and cost increases
Solution Approach 1:
The patent segments the journal storage function into two parts: volatile memory containers for metadata change buffering and non-volatile storage array for persistent storage. This segmentation allows the system to use inexpensive volatile memory for frequent writes while maintaining data safety through eventual persistence to non-volatile storage.
Solution Approach 2:
The patent introduces data containers in volatile memory as an intermediary between the host system and the storage array. These containers act as a buffer that decouples the high-speed write operations from the persistent storage requirements, allowing the system to handle bursts of IO traffic without immediately writing to non-volatile memory.
2Productivity
If non-volatile memory journals are used to acknowledge writes quickly, then write performance is improved, but the journal cannot accommodate more than a few seconds of IO traffic
Solution Approach 1:
The patent implements a dynamic buffering system where data containers in volatile memory can be rapidly allocated and switched to accommodate varying IO traffic patterns. The system dynamically manages container allocation based on current write workload, allowing it to handle bursts of traffic that exceed traditional journal capacity while maintaining quick acknowledgment performance.
3Stability of the object's composition
If metadata changes are immediately written to storage array, then data consistency is improved, but write performance and IO handling efficiency deteriorate
Solution Approach 1:
The patent performs preliminary buffering of metadata changes in volatile memory containers before writing to the storage array. This preliminary action allows the system to aggregate and organize metadata changes efficiently, then write them to storage in optimized batches, maintaining data consistency while improving write performance through reduced I/O operations.
Data Source
AI summary
A technique is directed to processing metadata changes. The technique involves designating a first tablet to ingest metadata changes and a second tablet to destage previously ingested metadata changes, the first tablet being partitioned into a first reserved space and a first regular space that form a first memory pool, and the second tablet being partitioned into a second reserved space and a second regular space that form a second memory pool. The technique further involves, while the first tablet is designated to ingest metadata changes and the second tablet is designated to destage previously ingested metadata changes, ingesting metadata changes into the first tablet and destaging the previously ingested metadata changes from the second tablet. The technique further involves, when the first tablet becomes full, performing a switch operation that designates the second tablet to ingest metadata changes and the first tablet to destage previously ingested metadata changes.