Erasure-Coded Data Replication for Space-Efficient Node Failure Protection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data protection methods are inefficient in terms of data transfer and storage space when creating multiple replicas to ensure availability, especially in dynamic cloud storage systems where data-generating and storage nodes frequently fail, leading to excessive data transfer and storage requirements.
Innovation Solution
A system that identifies a redundancy number for failed storage nodes and uses the Inter-Planetary File System protocol to divide data objects into sections, allocating replicas and erasure codes across available storage nodes in a way that minimizes data transfer and storage by creating erasure codes only as needed, allowing for efficient recreation of data files even if some nodes fail.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If multiple full replicas of data objects are created and stored across cloud storage nodes to ensure availability against node failures, then data reliability is improved, but data transfer volume and storage space requirements increase significantly
Solution Approach 1:
The patent divides data objects into multiple sections and distributes these sections across different storage nodes. Instead of creating complete replicas of entire data objects, the system segments the data and stores sections independently, reducing the storage space required for each replica while maintaining data availability through distributed reconstruction capability.
Solution Approach 2:
The patent transforms the replication approach by changing the parameter of data representation from complete copies to erasure-coded sections. By applying erasure coding algorithms, the system converts data into a different parameter form (encoded sections) that requires less storage space per node while still enabling full data reconstruction when sufficient nodes are available.
2Reliability
If multiple full replicas of data objects are created and transferred across the network to cloud storage nodes, then data reliability is improved, but network data transfer time and bandwidth consumption increase
Solution Approach 1:
The patent segments data objects into smaller sections before transfer and distribution. This segmentation reduces the time required to transfer each section across the network compared to transferring complete data object replicas, while still achieving the same level of data availability through distributed storage and erasure coding reconstruction.
Solution Approach 2:
By transforming data into erasure-coded sections, the patent changes the parameter of data representation to enable more efficient network transfer. The encoded sections can be transmitted faster and in parallel across multiple nodes, reducing overall data transfer time while maintaining the ability to reconstruct original data for availability.
3Ease of operation
If complete replicas of data objects are stored on each storage node, then ease of data retrieval is improved, but storage cost and resource consumption increase
Solution Approach 1:
The patent divides data objects into sections and distributes these segments across storage nodes rather than storing complete replicas at each node. This segmentation enables efficient data retrieval by allowing the system to reconstruct complete data objects from distributed sections, reducing storage resource consumption while maintaining retrieval capability through coordinated access to multiple nodes.
Solution Approach 2:
The patent applies erasure coding to transform data into encoded sections with different parameters than the original data. These encoded sections occupy less storage space per node while still enabling complete data reconstruction. The parameter change from full replicas to erasure-coded segments reduces storage resource consumption while preserving data accessibility.
Data Source
AI summary
A system identifies a data object, stored in a client, for replication. The system identifies a redundancy number associated with a protected amount of failed storage nodes. The system determines whether the total count of available storage nodes is at least two greater than the redundancy number. The system determines a distribution number that is at most the total count minus the redundancy number, in response to a determination that the total count of available storage nodes is at least two greater than the redundancy number. The system creates erasure code for the data object. The system allocates the distribution number of sections of a replica of the data object to a corresponding distribution number of available storage nodes and the redundancy number of sections of the erasure code to the redundancy number of available storage nodes, which excludes the corresponding distribution number of available storage nodes.