Persistent Cache Metadata Aggregation for Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Transaction logs in data storage systems are burdensome, causing performance issues due to memory copies, thread management, and resource contention, and lead to delays and metadata inconsistency, especially with the adoption of fast storage devices like SSDs and NVMe.
Innovation Solution
Implementing a persistent cache that receives and aggregates metadata changes in a single, atomic transaction, storing them successfully or not, and flushing them to non-volatile storage, eliminating the need for transaction logs and ensuring metadata consistency without replay requirements.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If transaction logs are used to store metadata changes, then metadata consistency is maintained, but system performance deteriorates due to memory copies, thread management, and resource contention
Solution Approach 1:
The patent extracts the transaction logging function from the traditional file system path and places it directly in the storage device's memory. By writing metadata changes directly to the storage device's memory instead of using a separate transaction log structure, the system eliminates the need for complex thread management and resource locking while maintaining metadata consistency. This extraction resolves the contradiction by removing the performance bottleneck while preserving reliability.
Solution Approach 2:
The patent introduces the storage device's memory as an intermediary between the host file system and the persistent storage medium. This intermediary buffer allows metadata changes to be staged and then atomically committed to the storage device, eliminating the need for traditional transaction logs and their associated overhead. The intermediary memory space resolves the contradiction by providing a high-speed buffer that maintains consistency without the performance penalties of traditional logging mechanisms.
2Reliability
If transaction logs are used to ensure metadata consistency, then data integrity is maintained, but system restart time increases due to replay requirements
Solution Approach 1:
The patent performs preliminary actions by writing metadata changes directly to the storage device's memory during normal operation, where they remain in a staged but committed state. This preliminary action ensures that when the system restarts, the metadata is already in its correct final state and does not require replay. The preliminary commitment of changes to the storage device's memory resolves the contradiction by maintaining data integrity while eliminating restart delays.
3Speed
If fast storage devices like SSDs and NVMe are adopted, then data transfer speed improves, but metadata management becomes a limiting factor due to transaction log overhead
Solution Approach 1:
The patent merges the metadata management function with the storage device's native memory structures. By combining the transaction logging functionality with the storage device's existing memory management mechanisms, the system eliminates the separate transaction log layer and its associated complexity. This merging resolves the contradiction by allowing fast storage devices to operate at full speed without the metadata management overhead that previously limited their performance.
Data Source
AI summary
A technique for managing metadata in a data storage system includes receiving a set of data to be stored in a file system of the data storage system and calculating values of metadata blocks that the file system will use to organize the set of data in the file system. The technique aggregates values of the metadata blocks and sends them in a single, atomic transaction to a persistent cache. The transaction either succeeds, in which case the persistent cache stores the values of all of the metadata blocks, or it fails, in which case the persistent cache stores none of the values of the metadata blocks. Over time, the persistent cache flushes the values of the metadata blocks to a set of non-volatile storage devices that back the metadata blocks in the data storage system.