Buffered Atomic Write Interface for Distributed Storage Metadata
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional filesystems and block storage systems are inefficient in handling scattered metadata updates in distributed storage systems, leading to performance issues and increased complexity due to the need for journaling or logging to ensure consistency across multiple disk volumes.
Innovation Solution
The system employs a buffered atomic write interface that allows for atomic read-modify-write operations on non-volatile buffers, enabling efficient storage of metadata chunks smaller than block size across multiple disk volumes without the need for journaling, using a block device firmware interface to manage metadata storage and handle updates in a background process.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional filesystems use journaling or logging to ensure consistency across multiple disk volumes, then reliability is improved, but device complexity and overhead increase
Solution Approach 1:
The patent separates metadata storage from content storage into distinct storage units, allowing independent optimization of each. Metadata is stored in metadata storage units with specific block addresses, while content is stored in content storage units, enabling targeted consistency mechanisms without affecting the entire storage system.
Solution Approach 2:
The patent introduces a buffer storage unit as an intermediary between the host and metadata storage units. This buffer temporarily holds metadata blocks and manages the atomic write operations, acting as a mediator that simplifies consistency management by providing a staging area for coordinated updates across multiple volumes.
2Reliability
If traditional filesystems use journaling or logging to ensure consistency, then reliability is improved, but I/O performance deteriorates due to overhead
Solution Approach 1:
The patent performs preliminary actions by pre-allocating buffer storage units and pre-establishing the metadata block structure before actual write operations. The buffer is prepared in advance to receive metadata, and the atomic write mechanism is pre-configured, reducing the overhead during actual I/O operations by eliminating runtime consistency checks.
Solution Approach 2:
The patent enables continuous metadata updates by using a buffer that can accept write operations without requiring immediate consistency verification. The atomic write operation continuously maintains consistency by atomically updating multiple metadata blocks together, allowing I/O operations to proceed without interruption from logging overhead.
3Adaptability or versatility
If metadata chunks smaller than block size are scattered across metadata storage unit, then adaptability is improved, but device complexity increases due to alignment requirements
Solution Approach 1:
The patent extracts the alignment and sizing complexity from the metadata storage system by using a buffer as an intermediate layer. The buffer handles the conversion between variable-sized metadata chunks and fixed-size blocks in the metadata storage unit, allowing scattered metadata of various sizes to be stored without requiring the storage system itself to be complex or aligned to specific boundaries.
4Reliability
If atomic read-modify-write commands are used for background process writing, then reliability is improved, but I/O performance deteriorates due to additional operations
Solution Approach 1:
The patent uses a buffer as a copy of the metadata storage unit's interface. The buffer receives and processes write operations, then atomically transfers the consolidated data to the metadata storage unit in the background. This copying mechanism allows the host to proceed with I/O operations without waiting for the slower atomic read-modify-write operations to complete, improving performance while maintaining reliability.
Data Source
AI summary
A storage command is received that is directed to a distributed storage system. Based on the storage command, a metadata update is targeted to a logical block address of metadata storage of the distributed storage system. The metadata update includes a chunk of metadata that is smaller than a block addressed by the logical block address. An atomic write command is sent to a block device interface. The command includes the chunk of metadata, the logical block address, and an offset within the block defining where the chunk of metadata is to be stored. Via the block device interface, the atomic write command is stored in a non-volatile buffer that has faster performance than the metadata storage. The chunk of metadata of the atomic write command is written from the non-volatile buffer to the block in the metadata storage via a background process using an atomic read-modify-write command.