Adaptive Erasure Coding for Deduplicated Data Protection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data storage systems lack an efficient method to balance data protection and storage efficiency, as data deduplication reduces redundancy, making some unique data less protected, while erasure coding increases storage requirements and costs.
Innovation Solution
Combining data deduplication with erasure coding, dynamically adapting erasure code generation based on the value of unique data, using rateless erasure codes to selectively increase or decrease redundancy, and distributing erasure code symbols accordingly, allowing for adaptive protection levels without recomputing entire code sets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If data deduplication is used to reduce storage requirements, then storage efficiency is improved, but data protection against loss is worsened
Solution Approach 1:
The patent applies local quality by differentiating protection levels for different unique data based on their value. High-value unique data receives erasure code protection while low-value unique data does not, creating non-uniform protection across the data set. This resolves the contradiction by providing enhanced protection only where needed rather than uniformly across all deduplicated data.
Solution Approach 2:
The patent changes the protection parameter dynamically based on data value assessment. By adjusting whether erasure code protection is applied to each unique data item based on its value, the system optimizes the balance between storage efficiency and data protection. This parameter change approach allows the system to adapt protection levels rather than maintaining a fixed state.
2Reliability
If erasure coding is used to enhance data protection, then data reliability is improved, but storage requirements and costs increase
Solution Approach 1:
The patent applies erasure code protection selectively to only those unique data items identified as high-value, rather than uniformly to all deduplicated data. This local application of protection reduces the overall storage overhead compared to universal erasure coding while maintaining protection for the most critical data items.
Solution Approach 2:
The patent implements partial action by applying erasure code protection to only a subset of unique data items (those with high value) rather than all deduplicated data. This partial application achieves sufficient data protection for critical items without the full storage cost of universal protection.
3Reliability
If uniform erasure code protection is applied to all unique data, then data protection is improved, but storage overhead increases for all data regardless of value
Solution Approach 1:
The patent implements non-uniform protection by assessing the value of each unique data item and applying erasure code protection only to high-value items. This creates a quality-based differentiation where protection level corresponds to data value, reducing overall storage overhead while maintaining protection for critical data.
Solution Approach 2:
The patent dynamically changes the protection parameter (whether to apply erasure coding) based on the assessed value of each unique data item. This parameter change allows the system to optimize the trade-off between protection and overhead by adjusting protection levels according to data importance rather than using a fixed uniform approach.
Data Source
AI summary
Example apparatus and methods combine erasure coding with data deduplication to simultaneously reduce the overall redundancy in data while increasing the redundancy of unique data. In one embodiment, an efficient representation of a data set is produced by deduplication. The efficient representation reduces duplicate data in the data set. Redundancy is then added back into the data set using erasure coding. The redundancy that is added back in adds protection to the unique data associated with the efficient representation. How much redundancy is added back in and what type of redundancy is added back in may be controlled based on an attribute (e.g., value, reference count, symbol size, number of symbols) of the unique data. Decisions concerning how much and what type of redundancy to add back in may be adapted over time based, for example, on observations of the efficiency of the overall system.