Storage Proxy Switching Between Erasure Coding and Replication
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Erasure coding (EC) systems with fewer than the EC-based data-plus-parity count are vulnerable to data loss and service disruptions due to node failures, as they lack sufficient fragments for reconstruction and distribution during failures.
Innovation Solution
A distributed data storage system with a storage proxy that manages EC configurations, intercepts client requests, and switches between EC and replication modes to ensure data resilience, using a 4+2 EC scheme with a storage proxy that creates EC and replication virtual disks to handle node failures and outages.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If erasure coding is configured with fewer storage nodes than the EC-count (data plus parity fragments), then storage capacity is optimized and storage savings are achieved, but the system becomes vulnerable to data loss and service disruptions when nodes fail
Solution Approach 1:
The system dynamically switches between EC mode and replication mode based on the number of available storage nodes. When nodes are available, EC is used for optimal storage efficiency. When node failures occur and the number of available nodes drops below the EC-count, the system automatically transitions to replication mode to ensure data availability, thereby resolving the contradiction between storage optimization and reliability
Solution Approach 2:
The system changes the protection parameter from EC fragments to replicated copies when the number of available nodes is insufficient for EC reconstruction. This parameter change allows the system to maintain data availability by using a different data protection mechanism that does not require a minimum number of nodes equal to the EC-count
2Productivity
If the number of storage nodes is reduced below the EC-count, then storage efficiency is improved, but the system cannot distribute EC fragments during node failures
Solution Approach 1:
The system dynamically adjusts its operation mode based on the number of available storage nodes. When the number of nodes is sufficient, EC is used for high storage efficiency. When node failures reduce the available nodes below the EC-count, the system automatically switches to replication mode, enabling the system to adapt to failure conditions while maintaining operational capability
3Quantity of substance
If EC is used with fewer than data-plus-parity storage nodes, then storage space is optimized, but the system experiences loss of service during node failures
Solution Approach 1:
The system dynamically transitions between EC mode and replication mode based on the number of available storage nodes. When nodes are available, EC provides storage space optimization. When node failures occur and available nodes fall below the EC-count, the system automatically switches to replication mode to maintain service continuity, thereby resolving the contradiction between storage optimization and ease of operation
Solution Approach 2:
The storage proxy acts as an intermediary that manages the transition between EC mode and replication mode. It intercepts client requests and determines the appropriate mode based on the number of available nodes, enabling seamless switching to maintain service continuity while optimizing storage space when conditions permit
Data Source
AI summary
A distributed data storage system using erasure coding (EC) provides advantages of EC data storage while retaining high resiliency for EC data storage architectures having fewer data storage nodes than the number of EC data-plus-parity fragments. An illustrative embodiment is a three-node data storage system with EC 4+2. Incoming data is temporarily replicated to ameliorate the effects of certain storage node outages or fatal disk failures, so that read and write operations can continue from/to the storage system. The system is equipped to automatically heal failed EC write attempts in a manner transparent to users and/or applications: when all storage nodes are operational, the distributed data storage system automatically converts the temporarily replicated data to EC storage and reclaims storage space previously used by the temporarily replicated data. Individual hardware failures are healed through migration techniques that reconstruct and re-fragment data blocks according to the governing EC scheme.