Erasure-Coded Storage Proxy for Node-Limited Cluster Resilience
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Erasure coding (EC) systems with fewer than data-plus-parity storage nodes face challenges in maintaining resiliency and data integrity during node failures, leading to potential data loss and service disruptions.
Innovation Solution
A distributed data storage system with a storage proxy that manages EC and replication modes, ensuring data resilience by distributing EC fragments across multiple nodes and switching to replication mode during failures, using a 4+2 EC configuration with a storage proxy that intercepts client requests and handles failure scenarios.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If erasure coding is configured with fewer storage nodes than data-plus-parity fragments, then storage capacity is improved, but system resiliency deteriorates
Solution Approach 1:
The system dynamically switches between EC mode and replication mode based on real-time node availability. When all nodes are available, EC mode provides optimal storage efficiency. When node failures occur, the system automatically transitions to replication mode to ensure data resiliency, and then heals back to EC mode when nodes are restored.
Solution Approach 2:
The storage proxy acts as an intermediary between client applications and the storage cluster. It transparently manages the complexity of mode switching, fragment distribution, and failure handling, allowing clients to access storage resources without needing to understand the underlying EC or replication mechanisms.
2Device complexity
If storage nodes are limited to fewer than EC-count, then device complexity is reduced, but data integrity deteriorates
Solution Approach 1:
The system performs preliminary actions by pre-distributing EC fragments across available nodes before failures occur. When a node fails, the system has already positioned fragments optimally on remaining nodes, enabling immediate reconstruction without complex real-time calculations during failure events.
Solution Approach 2:
The system changes operational parameters by switching between EC configuration (for normal operation) and replication configuration (for failure recovery). This parameter change allows the same physical infrastructure to provide different protection levels based on runtime conditions, maintaining data integrity despite node limitations.
3Productivity
If EC fragments are distributed across fewer nodes, then storage efficiency is improved, but service availability deteriorates
Solution Approach 1:
The system dynamically adjusts fragment distribution strategies based on node availability. During normal operation, fragments are distributed to maximize storage efficiency. During failures, the system dynamically repositions fragments and switches to replication mode to maintain service availability, then heals back to optimal EC distribution when nodes are restored.
Data Source
AI summary
A distributed data storage system using erasure coding (EC) provides advantages of EC data storage while retaining high resiliency for EC data storage architectures having fewer data storage nodes than the number of EC data-plus-parity fragments. An illustrative embodiment is a three-node data storage system with EC 4+2. Incoming data is temporarily replicated to ameliorate the effects of certain storage node outages or fatal disk failures, so that read and write operations can continue from/to the storage system. The system is equipped to automatically heal failed EC write attempts in a manner transparent to users and/or applications: when all storage nodes are operational, the distributed data storage system automatically converts the temporarily replicated data to EC storage and reclaims storage space previously used by the temporarily replicated data. Individual hardware failures are healed through migration techniques that reconstruct and re-fragment data blocks according to the governing EC scheme.