DPU Erasure Coding for Balanced Distributed Storage Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current distributed storage systems face challenges in data loss due to node failures, inefficient use of storage space with multi-replica backup mechanisms, and unbalanced performance across storage nodes, leading to delayed data reconstruction and reduced overall cluster performance.
Innovation Solution
A DPU-based storage system with an erasure acceleration module and OSD fast hash table lookup module is introduced, which performs erasure coding and decoding operations on a DPU to optimize data distribution and retrieval, using FPGA and SoC modules for efficient data block management and storage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multi-replica backup mechanism is used to ensure data reliability, then data reliability is improved, but storage space consumption increases and capacity utilization rate decreases
Solution Approach 1:
The patent extracts the data reliability function from traditional multi-replica backup and implements it through erasure coding, where data is divided into fragments and distributed across multiple storage nodes with mathematical redundancy, achieving the same reliability without storing complete duplicate copies
Solution Approach 2:
The patent changes the redundancy parameter from fixed multi-replica (e.g., 3 copies) to flexible erasure coding parameters (k data blocks + m parity blocks), allowing dynamic adjustment of the trade-off between storage efficiency and data reliability based on actual needs
2Quantity of substance
If erasure coding is used to improve storage efficiency and reliability, then data redundancy is reduced, but data reconstruction speed decreases due to unbalanced I/O pressures and network transmission
Solution Approach 1:
The patent performs preliminary actions by pre-calculating and pre-distributing data fragments and parity blocks across storage nodes during the data writing phase, and by pre-establishing the mapping relationships between data blocks and storage locations, so that during reconstruction, the system can quickly locate and retrieve the necessary blocks without delay
Solution Approach 2:
The patent introduces an intermediary metadata layer that tracks the location and state of all data fragments and parity blocks, enabling the reconstruction process to efficiently identify which blocks need to be retrieved from which nodes, thereby balancing the I/O pressure and network transmission load
3Reliability
If data is segmented and distributed across multiple nodes for fault tolerance, then fault tolerance is improved, but data reconstruction delay increases due to unbalanced storage devices and network transmission
Solution Approach 1:
The patent performs preliminary actions by pre-distributing data and parity blocks across multiple nodes during the initial write operation, and by pre-establishing the mapping relationships between data blocks and storage locations, so that during reconstruction, the system can quickly locate and retrieve the necessary blocks without delay
Solution Approach 2:
The patent implements dynamic load balancing during data reconstruction by monitoring the real-time status of storage nodes and dynamically adjusting the retrieval strategy, selecting optimal paths and nodes based on current I/O pressure and network conditions, thereby minimizing reconstruction delay
Data Source
AI summary
A storage system, a data storage method, a data read method, and a storage medium are provided by the present application, and belong to the technical field of data storage. A central processing unit (CPU) of a main storage node of the system, after determining to-be-stored data in response to a data storage operation of a client sends the to-be-stored data to a data processing unit (DPU) of the main storage node; an erasure acceleration module of the DPU performs an erasure coding operation on the to-be-stored data to obtain a plurality of data blocks and a plurality of check blocks, determines corresponding address information of target storage devices according to state information of each storage device in an object storage device (OSD) fast hash table lookup module, and sends the data blocks or the check blocks of the to-be-stored data to a target storage node for storage.