Dynamic Erasure Coding Redundancy for Distributed Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage systems using erasure coding, data migration and backward migration due to faulty nodes significantly impact performance, as they require extensive data redistribution and new node addition, leading to performance bottlenecks.
Innovation Solution
A data storage method that dynamically adjusts the EC redundancy ratio by allowing the storage client to generate fewer EC blocks and parity blocks based on available non-faulty nodes, excluding faulty nodes from data storage and avoiding unnecessary data migration, thereby reducing the number of generated blocks and improving system performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data migration and backward migration are performed to replace faulty storage nodes, then storage reliability is maintained, but system performance deteriorates due to extensive data redistribution
Solution Approach 1:
The patent dynamically adjusts the EC redundancy ratio based on the number of available non-faulty storage nodes. When faulty nodes are detected, the system reduces the redundancy ratio from the original N+M configuration to match the current number of available nodes, eliminating the need for data migration while maintaining storage reliability through adaptive parameter adjustment.
2Reliability
If new storage nodes are added to replace faulty nodes, then storage reliability is improved, but device complexity increases due to node addition and data redistribution
Solution Approach 1:
The patent changes the redundancy ratio parameter dynamically based on the actual number of available storage nodes. Instead of adding new nodes or performing complex data migration, the system adjusts the EC coding parameters (N and M values) to match the current cluster size, thereby maintaining reliability without increasing device complexity.
3Reliability
If full EC stripe with N+M blocks is generated, then data persistence is ensured, but loss of time increases due to generating and storing additional parity blocks
Solution Approach 1:
The patent applies partial EC encoding by generating only the necessary number of EC blocks and parity blocks based on the current number of available storage nodes, rather than always generating the full N+M blocks. This reduces the time required for data storage while maintaining sufficient redundancy for data persistence through dynamic parameter adjustment.
Data Source
AI summary
A storage client needs to store to-be-written data into a distributed storage system, and storage nodes corresponding to a first data unit assigned for the to-be-written data by a management server are only some nodes in a storage node group. When receiving a status of the first data unit returned by the management server, the storage client may determine quantities of data blocks and parity blocks needing to be generated during EC coding on the to-be-written data. The storage client stores the generated data blocks and parity blocks into some storage nodes designated by the management server in a partition where the first data unit is located. Accordingly, dynamic adjustment of an EC redundancy ratio is implemented, and the management server may exclude some nodes in the partition from a storage range of the to-be-written data based on a requirement, thereby reducing a data storage IO amount.