Tiered Storage Data Migration with Adaptive Erasure Coding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current data storage systems face inefficiencies in maximizing storage capacity and space utilization due to the limitations of existing erasure code technologies, particularly in multi-level medium storage systems where different performance layers (like DRAM, SCM, and NAND flash) are used without adaptive erasure code ratios.
Innovation Solution
A method is introduced where data is stored at a high-performance medium layer using a low erasure code ratio to reduce write amplification and then migrated to a low-performance medium layer using a high erasure code ratio, with the erasure code ratios adjusted based on performance differences and system conditions, allowing for flexible calculation of check units to optimize storage space utilization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is stored at high-performance medium layer with low erasure code ratio, then write amplification is reduced and storage speed is improved, but storage space utilization decreases
Solution Approach 1:
The storage system is segmented into multiple medium layers (first medium layer and second medium layer) with different performance characteristics. Data is stored at different layers based on access patterns and importance, allowing the system to achieve both high speed for frequently accessed data and high space utilization for less frequently accessed data through differentiated erasure code ratios at each layer
Solution Approach 2:
The erasure code ratio parameter is changed based on the medium layer being used. The first medium layer uses a first erasure code ratio (N:M) while the second medium layer uses a second erasure code ratio (X:Y) where N/X ≠ M/Y. This parameter adaptation allows optimization of both write amplification and storage space utilization for different storage scenarios
2Quantity of substance
If data is migrated from high-performance medium layer to low-performance medium layer, then storage cost is reduced and capacity increased, but access latency increases
Solution Approach 1:
Data is pre-stored at the high-performance first medium layer with appropriate erasure coding protection. When migration conditions are met (such as data becoming cold or less frequently accessed), the system proactively migrates data to the second medium layer, ensuring that hot data remains fast while cold data occupies cheaper capacity
Solution Approach 2:
The system dynamically adjusts data placement between medium layers based on access patterns and system conditions. The erasure code ratio is dynamically adapted when migrating data between layers, allowing the system to maintain optimal performance and space utilization as data transitions between hot and cold states
3Device complexity
If same erasure code ratio is used across all medium layers, then system complexity is reduced, but storage space utilization and performance are suboptimal
Solution Approach 1:
Different erasure code ratios are applied to different medium layers based on their specific characteristics. The first medium layer uses erasure code ratio N:M while the second medium layer uses ratio X:Y, with the condition that N/X ≠ M/Y. This local optimization at each layer maximizes storage space utilization and performance without requiring complex cross-layer coordination
Data Source
AI summary
In a data storage method, a storage system comprises first and second medium layers for data storage, wherein the performance of the first medium layer is different from the performance of the second medium layer. Based on the performance difference between the two layers, the storage system stores data in the first medium layer and the second medium layer based on different erasure code ratios. The different erasure code ratios correspond to different write amplification, and result in different storage space utilization.