Log-Structured Storage Metadata Mapping for Write Amplification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Log-structured distributed file systems face increased write amplification and wear-leveling due to frequent data compaction and metadata management, which consumes resources and poses risks of data loss, especially in SSDs where metadata consistency is crucial.
Innovation Solution
The system employs persistent memory, such as phase change memory (PCM), resistive RAM (ReRAM), or magnetoresistive RAM (MRAM), to store metadata and mapping tables instead of volatile DRAM, allowing in-place overwrites and maintaining data at original physical locations while mapping original logical block addresses to new ones, thereby reducing redundant protection and storage needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data compaction is performed frequently to efficiently utilize storage capacity, then storage utilization is improved, but write amplification increases
Solution Approach 1:
The patent introduces a log-structured layer as an intermediary between the host and the storage media. This log structure absorbs write operations sequentially, avoiding frequent random writes to the storage media. The log acts as a buffer that decouples the host's write operations from the storage media's physical structure, reducing write amplification while maintaining efficient storage utilization through periodic log compaction.
2Reliability
If metadata is stored in non-volatile memory to ensure consistency and reliability, then data reliability is improved, but resource consumption increases
Solution Approach 1:
The patent merges the metadata storage with the log structure itself. Instead of maintaining separate metadata structures, the log entries contain all necessary mapping information (logical block addresses to physical block addresses) directly within the log entries. This consolidation eliminates redundant metadata storage, reducing resource consumption while maintaining reliability through the inherent persistence of the log-structured storage.
3Ease of manufacture
If data is written sequentially in a circular fashion without in-place overwrites, then storage simplicity is improved, but scattered data chunks with holes are created
Solution Approach 1:
The patent performs preliminary compaction of the log structure periodically. Before new data is written, the system compacts the log by removing invalid entries and reorganizing valid data contiguously. This preliminary action prevents the accumulation of scattered data chunks with holes, maintaining efficient storage utilization while preserving the simplicity of sequential write operations.
Data Source
AI summary
One embodiment provides a system which facilitates organization of data. During operation, the system identifies an original data chunk stored in a non-volatile memory of a storage device, wherein the original data chunk is a logical chunk which includes original logical block addresses. The system stores a first mapping of the original logical block addresses to original physical block addresses in a first data structure. The system assigns new logical block addresses to be included in a new data chunk. The system creates, in a second data structure based on an order of the assigned new logical block addresses, a mapping of the new logical block addresses to valid original logical block addresses. The system stores, based on the first data structure and the second data structure, a second mapping of the new logical block addresses to the original physical block addresses.