Deduplicated Storage Redundancy Switching for Space Efficiency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing data protection schemes in storage clusters face inefficiencies in storage space utilization due to varying levels of data redundancy, particularly when combining replication and erasure coding, leading to redundant information storage and suboptimal use of storage resources.
Innovation Solution
A technique that dynamically assigns data blocks to slice and block services based on data protection schemes, allowing for deduplication and erasure coding while maintaining data integrity, and employs periodic garbage collection to optimize storage based on current protection schemes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If data blocks are replicated across multiple storage nodes to ensure data integrity, then data availability and reliability are improved, but storage space efficiency deteriorates due to redundant information storage
Solution Approach 1:
The system dynamically changes the redundancy parameter based on data block characteristics and storage conditions. By adjusting the number of replicas or erasure coding parameters according to data importance, access patterns, and available storage capacity, the system optimizes the balance between data integrity and storage space utilization without compromising reliability
Solution Approach 2:
The redundancy scheme transitions from static to dynamic, allowing the system to adaptively select between replication and erasure coding based on real-time conditions. This dynamic approach enables the system to maintain data integrity guarantees while optimizing storage space by choosing the most efficient redundancy method for each data block
2Quantity of substance
If different data protection schemes are applied to different data blocks, then storage efficiency is improved, but system complexity increases due to managing multiple redundancy schemes
Solution Approach 1:
The system implements a universal redundancy management framework that handles both replication and erasure coding through a common architecture. This multi-functional approach allows the same system components to manage different data protection schemes, reducing overall system complexity while maintaining storage space efficiency
Solution Approach 2:
An intermediary layer is introduced between the storage nodes and the redundancy management logic. This mediator abstracts the complexity of managing multiple redundancy schemes by providing a unified interface for data block assignment, tracking, and recovery operations across different protection schemes
3Quantity of substance
If erasure coding is used instead of duplication to improve storage efficiency, then storage space utilization is improved, but data recovery complexity increases due to encoded data reconstruction
Solution Approach 1:
The system performs preliminary organization of data blocks into erasure coding groups and pre-computes recovery information. By preparing the data structure and encoding relationships in advance, the system reduces the complexity of data reconstruction operations, making erasure coding more manageable while maintaining storage space efficiency
Data Source
Figure 1
Figure 2
Figure 3A~3B
AI summary
A technique is configured to provide various data protection schemes, such as replication and erasure coding, for data blocks of volumes served by storage nodes of a cluster configured to perform deduplication of the data blocks. Additionally, the technique is configured to ensure that each deduplicated data block complies with data redundancy guarantees of the data protection schemes, while improving storage space of the storage nodes. In order to satisfy the data integrity guarantees while improving available storage space, the storage nodes perform periodic garbage collection for data blocks to optimize storage in accordance with currently applicable data protection schemes