Storage System Metadata Backup and Data Write Concurrency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems experience excessively long write operation times due to the requirement that metadata must be successfully backed up before data blocks can be written to storage devices.
Innovation Solution
A data processing method in a storage system where the client sends data blocks and metadata to a data storage node and sends parity blocks and metadata to a parity storage node, allowing for concurrent processing and reduced write operation times by eliminating the need to wait for metadata backup success.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the storage node waits for metadata backup success before writing data blocks, then metadata reliability is improved, but write operation time increases excessively
Solution Approach 1:
The system performs metadata backup in advance before the actual data writing operation. The client sends metadata to the storage node prior to writing data blocks, so that when data writing is initiated, the metadata is already in place, eliminating the need to wait for metadata backup during the write operation.
Solution Approach 2:
The writing process is divided into separate phases: metadata preparation phase and data block writing phase. These phases are decoupled so that metadata can be prepared independently and in parallel with data generation, allowing the data writing process to proceed without being blocked by metadata backup status.
2Reliability
If metadata is backed up to a separate storage node, then system reliability is improved, but device complexity increases
Solution Approach 1:
The storage node is designed to perform multiple functions: it serves as both a data storage node and a metadata storage node. By making the storage node universal, the system avoids adding separate dedicated metadata storage infrastructure, thereby improving reliability through redundancy while controlling device complexity.
3Productivity
If the client sends both data blocks and metadata to the storage node, then write operation time is reduced, but network traffic increases
Solution Approach 1:
The client combines metadata and data block transmissions into a single network operation. Instead of sending metadata and data blocks in separate network round trips, both are transmitted together in one request, reducing the number of network interactions and improving overall write operation speed despite the increased data volume.
Data Source
AI summary
A data processing method in a storage system and the storage system, and relates to the field of data storage technologies. A client first sends a data block in a stripe and metadata of the data block to a data storage node. The client then sends the metadata of the data block and a parity block to a parity storage node, to back up the metadata of the data block on the parity storage node. When the metadata of the data block is successfully backed up, the data block is written into a corresponding storage device based on a storage location indicated by the metadata. This reduces write operation time.