Composite Data Fragment Copies for Dual-Node Failure Tolerance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Contemporary cloud-based data storage systems face challenges in tolerating dual-node failures due to vulnerabilities in optimized triple mirroring schemes, where data loss occurs if two nodes with complete chunk copies fail, making the system more susceptible to data loss.
Innovation Solution
Implementing a preliminary protection scheme that writes three composite copies of data fragments across different nodes, ensuring no duplicate copies are stored on the same node, allowing the system to tolerate dual-node failures by distributing data fragments across distinct nodes, and performing erasure coding once the chunk is full or after a predefined time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of energy
If optimized triple mirroring stores two complete chunk copies and one composite copy with data fragments distributed among nodes, then internal data traffic is reduced, but the system cannot tolerate dual-node failure and data vulnerability increases
Solution Approach 1:
The chunk data is segmented into multiple data fragments (k fragments) which are then distributed across different nodes. Instead of storing complete copies, the system stores these fragmented portions on different nodes, ensuring that no single node holds a complete copy that could cause data loss if failed. This segmentation approach maintains data redundancy while reducing internal data traffic.
Solution Approach 2:
Different nodes store different portions of the data (specific data fragments) rather than identical complete copies. Each node has a specialized role storing specific fragments, creating a distributed architecture where data is locally optimized across the network. This allows the system to tolerate node failures while minimizing data movement.
2Reliability
If triple mirroring stores three complete chunk copies to different nodes, then dual-node failure is tolerated, but internal data traffic increases significantly
Solution Approach 1:
The complete chunk data is divided into k data fragments, which are then distributed across multiple nodes. Instead of replicating entire chunks three times (creating 3n data movement), the system distributes fragments such that each node stores only a portion, reducing total internal data traffic while maintaining the ability to reconstruct complete data from any three nodes.
Solution Approach 2:
The system transitions from a vertical replication model (storing complete copies) to a horizontal fragmentation model (distributing data fragments across nodes). This dimensional shift in data organization allows the same reliability level to be achieved with reduced data movement by utilizing the network's distributed nature more efficiently.
3Device complexity
If two nodes containing complete chunk copies fail in optimized triple mirroring, then data loss occurs, but the system structure remains simple
Solution Approach 1:
By segmenting the chunk into k data fragments and distributing them across nodes, the system ensures that no single node or small group of nodes holds complete data. This segmentation prevents data loss from dual-node failure while maintaining relatively simple storage operations, as each node still stores data in a straightforward manner, just different portions.
Solution Approach 2:
The system introduces data fragments as intermediaries between complete chunks and node storage. These fragments act as building blocks that can be reassembled from any sufficient subset of nodes, providing a buffer that protects against node failures while keeping the overall system structure manageable through standardized fragment handling procedures.
Data Source
AI summary
The disclosed technology generally describes a preliminary (e.g., triple mirroring) data protection scheme that operates by writing data as redundant (e.g., three) composite copies made up of copies of data fragments to different nodes of a data storage system. The data fragments are distributed such that any two nodes can fail yet a complete set of data remains among the remaining data fragments. Later, erasure encoding creates redundant coding fragments that are written to the nodes of a data storage system in a distributed manner along with one copy of the data fragments, such that any two nodes can fail but the complete data can still be recovered. Redundant data fragments are then deleted.