Logging Metadata Modifications in Data Storage Systems
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional methods for updating metadata tables in data storage systems are costly in terms of I/O performance and disk space, and often require locking out other I/O operations or storing large amounts of data, which is inefficient, especially in systems like RAID 5.
Innovation Solution
A log-based approach is implemented where metadata modifications are logged instead of directly writing to the metadata structures, using a log string data structure with a type, length, and string data field, and collected log strings are periodically committed to disk, allowing for efficient space utilization and lower disk thrashing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional methods write metadata directly to disk, then data integrity is maintained, but I/O performance deteriorates and disk space requirements increase
Solution Approach 1:
The patent segments the metadata update process into two distinct parts: (1) logging metadata modifications to a log structure on disk, and (2) periodically replaying the log to update the actual metadata tables. This segmentation allows the system to maintain data integrity through logging while improving I/O performance by batch-processing updates rather than writing immediately for each modification.
Solution Approach 2:
The patent performs preliminary action by logging all metadata modifications in advance before actually applying them to the metadata tables. The log structure records all changes with timestamps and operation details, allowing the system to batch-process and replay these changes later, thereby improving I/O performance while ensuring no data loss occurs.
2Reliability
If traditional methods write metadata to disk, then data persistence is ensured, but disk space requirements increase
Solution Approach 1:
The patent uses a copying approach where instead of writing full metadata tables to disk, it creates a compact log structure that copies only the essential modification information (operation type, affected blocks, timestamps). This log structure is much smaller than the actual metadata tables, significantly reducing disk space requirements while maintaining data persistence through the replay mechanism.
3Manufacturing precision
If read-modify-write operations are used for sub-sector level changes, then data accuracy is maintained, but I/O performance deteriorates
Solution Approach 1:
The patent applies preliminary action by logging all sub-sector level modifications in advance with precise operation details (read, write, delete operations on specific blocks). This allows the system to maintain data accuracy through the complete log record while improving I/O performance by batch-processing these operations during log replay rather than performing read-modify-write cycles for each individual change.
Data Source
AI summary
A method, system, apparatus, and computer-readable medium for logging metadata modifications in a storage system are provided. According to one method, modifications to storage system metadata are detected and described in a log string data structure. The log strings are collected and periodically committed to a mass storage device in the data storage system. The log strings may be collected in a log buffer data structure that includes the collected log strings and one or more unique, constantly increasing timestamps. The time stamps are written at the end of each sector of the log buffer. The log buffer may be committed to the mass storage device when the log buffer is full, when a predefined period of time has elapsed since the previous commitment of the log strings to disk, or in a manner designed to ensure that one write of the log buffer is always outstanding. A logger module is operative to collect the log strings, build the log buffer, and to commit the log buffer to disk.