Erasure Code Block Updates With Lower I/O in Multi-Mirror Storage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed storage systems, the recalculation of code blocks in erasure coding systems can be performance-intensive due to the need for numerous I/O operations, especially when storage locations are unavailable, impacting overall system performance.
Innovation Solution
A method is developed to determine the operation cost of generating an updated code block based on either the plurality of data blocks or the existing code and data blocks, choosing the method with the lowest read operation cost to efficiently regenerate the code block.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If code block recalculation is performed using traditional erasure coding methods, then data redundancy and reliability are maintained, but system performance deteriorates due to numerous I/O operations
Solution Approach 1:
The system pre-calculates and stores operation costs for different code block regeneration methods before actual data updates occur. When a data block needs updating, the system has already determined the most efficient regeneration path, avoiding runtime performance penalties while maintaining data redundancy through proper code block regeneration.
2Manufacturing precision
If code block regeneration reads from multiple data blocks, then accurate code block updates are achieved, but I/O operation count increases
Solution Approach 1:
The system dynamically selects between two different code block regeneration methods based on real-time conditions: reading from multiple data blocks when available for accurate regeneration, or reading from existing code blocks when fewer I/O operations are needed. This dynamic adaptation maintains code block accuracy while minimizing I/O operations based on current system state.
Solution Approach 2:
The system changes the operational parameters of code block regeneration by calculating and comparing operation costs for different methods. It selects the method with the lower operation cost, thereby changing from a fixed regeneration approach to a flexible one that adapts to current data block availability and performance requirements.
3Reliability
If storage locations are unavailable during code block regeneration, then system availability is reduced, but data protection is maintained through reconstruction
Solution Approach 1:
The system pre-calculates operation costs for code block regeneration methods, including scenarios where storage locations may be unavailable. By having regeneration strategies predetermined and cost-analyzed in advance, the system can quickly switch to alternative methods when storage locations are unavailable, maintaining data protection while minimizing impact on system availability.
Data Source
AI summary
Certain embodiments described herein are generally directed to generating an updated version of an existing code block in response to a data update to an existing data block of a plurality of data blocks in a distributed multi-mirror erasure coding system. For example, in some embodiments, a first operation cost to generate the updated version of the existing code block by a first method based on the plurality of data blocks and the data update is determined. In certain embodiments, a second operation cost to generate the updated version of the existing code block by a second method based on the existing code block, the existing data block, and the data update is determined. In some embodiments, if the first operation cost is less than or equal to the second operation cost, the first method is used. Otherwise, the second method is used.