Dynamic Redundancy in Storage Systems via Erasure Coding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Fault-tolerant storage systems face inefficiencies in data archiving due to the need for all servers to confirm chunk storage, leading to prolonged archiving times and potential failures if one server is slow or unavailable.
Innovation Solution
Implement a dynamic redundancy method that splits data fragments into chunks and generates additional chunks using erasure codes, allowing confirmation from a subset of storage devices sufficient to ensure redundancy, thereby skipping confirmation from slowest servers and reducing archiving time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all servers must confirm chunk storage, then data reliability is improved, but archiving time increases significantly
Solution Approach 1:
The system applies partial action by requiring confirmation from only a subset of servers (fifth number of storage devices) rather than all servers (fourth number of storage devices). This allows the archiving process to complete with sufficient redundancy without waiting for all servers, thereby reducing archiving time while maintaining acceptable data reliability.
Solution Approach 2:
The system dynamically adjusts the redundancy level based on the data fragment type. For last data fragments, higher redundancy (third number of additional data chunks) is generated and stored on more devices. For non-last data fragments, lower redundancy (second number of additional data chunks) is sufficient. This dynamic approach optimizes both reliability and archiving speed.
2Reliability
If confirmation from all storage devices is required, then data redundancy is ensured, but productivity decreases
Solution Approach 1:
The system uses partial action by accepting confirmation from a subset of storage devices rather than all devices. The fifth number of storage devices providing confirmation is sufficient to ensure the required level of redundancy, eliminating the need to wait for all fourth number of storage devices and thereby improving archiving speed.
Solution Approach 2:
The system changes the redundancy parameter dynamically based on data fragment characteristics. By adjusting the number of additional data chunks generated and the number of storage devices required for confirmation, the system optimizes the balance between reliability and productivity for different types of data fragments.
3Reliability
If higher redundancy is applied to all data fragments, then data reliability improves, but storage requirements increase
Solution Approach 1:
The system applies local quality by differentiating redundancy levels based on the specific characteristics of data fragments. Last data fragments receive higher redundancy (third number of additional data chunks) because they are critical for data stream completion. Non-last data fragments receive lower redundancy (second number of additional data chunks) as they can be recovered from previous fragments. This localized approach optimizes storage requirements while maintaining necessary reliability.
Solution Approach 2:
The system dynamically adjusts redundancy parameters based on data fragment type. By generating different numbers of additional data chunks for different fragment types, the system ensures high reliability where needed (last fragments) while minimizing storage consumption where lower redundancy suffices (non-last fragments).
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
This approach enhances archiving operation speeds by avoiding waiting for confirmation from all servers, ensuring data redundancy with fewer confirmations, and minimizing storage requirements for increased service availability.
Implementation Method 1
using erasure codes to generate one or both of the second number of additional data chunks and the third number of additional data chunks
Data Source
AI summary
A method and system for dynamic redundancy in storage systems is described. The method may include receiving a data fragment from a data stream of user data to be archived. The method may further include splitting the data fragment into a first number of data chunks. The method may also include, in response to determining that the data fragment is not a last data fragment in the data stream, generating a second number of additional data chunks based upon, at least in part, the first number of data chunks. The method may additionally include, in response to determining that the data fragment is the last data fragment in the data stream, generating a third number of additional data chunks based upon, at least in part, the first number of data chunks.