Adaptive File Storage Using Dynamic Erasure Coding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing erasure code technologies in distributed file systems face challenges in balancing storage overheads, restoration costs, and coding speed, with current solutions either having high storage overheads or high restoration costs when nodes fail.
Innovation Solution
An adaptive file storage method that determines the 'cold' or 'hot' attribute of files based on access frequency and time intervals, using different erasure code technologies for coding and transcoding storage processing to optimize storage overheads and restoration costs dynamically.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If product code algorithm is used for erasure coding, then coding speed is high and restoration costs are low, but storage overheads are high
Solution Approach 1:
The system dynamically selects between different erasure coding algorithms (product code for hot data, LRC for cold data) based on data access patterns. This dynamic adaptation allows the system to optimize for coding speed when data is frequently accessed while reducing storage overheads when data is rarely accessed, resolving the contradiction between coding speed and storage overheads.
Solution Approach 2:
The system changes the coding parameters by selecting different algorithms based on data temperature. Product code algorithm parameters are used for hot data to maximize coding speed, while LRC algorithm parameters are used for cold data to minimize storage overheads, thus resolving the contradiction through parameter adaptation.
2Quantity of substance
If LRC algorithm is used for erasure coding, then storage overheads are low, but restoration costs are high when multiple nodes fail
Solution Approach 1:
The system dynamically switches between LRC and product code algorithms based on the failure scenario and data temperature. When multiple nodes fail, the system can switch to product code algorithm which provides better restoration capabilities despite higher storage overheads, while normally using LRC for low overheads. This dynamic adaptation resolves the contradiction between storage overheads and restoration costs.
Solution Approach 2:
The system adapts coding parameters by selecting different algorithms based on reliability requirements. LRC parameters are used for normal operation to minimize storage overheads, while product code parameters are activated when multiple node failures are detected or anticipated, thus resolving the contradiction between storage overheads and restoration costs through parameter changes.
3Device complexity
If a single EC technology is used, then implementation is simple, but comprehensive performance cannot be optimized across multiple dimensions
Solution Approach 1:
The system segments the erasure coding implementation into multiple algorithm components (product code and LRC) and selectively applies each segment based on data temperature and failure scenarios. This segmentation allows the system to achieve high comprehensive performance by using the appropriate algorithm for each situation while keeping the overall implementation manageable through clear segmentation logic.
Solution Approach 2:
The system implements a universal erasure coding framework that can perform multiple functions by selecting between different algorithms. The framework provides both high coding speed and low storage overheads capabilities, as well as both low restoration costs and high reliability capabilities, depending on the selected algorithm. This multi-functionality resolves the contradiction between implementation complexity and comprehensive performance.
Data Source
AI summary
An adaptive file storage method and apparatus is disclosed. The method includes determining a cold and hot attribute of a file, and performing coding storage processing or transcoding storage processing on the file according to the cold and hot attribute of the file. Therefore, a requirement of the cold and hot attribute of the file for storage overheads and restoration costs can be fully considered. In addition, the used coding technology has high reliability and a high coding speed. Therefore, comprehensive performance in multiple dimensions of storage overheads, restoration costs, reliability, and an coding speed can be improved.