Dynamic Erasure Coding for Faster Distributed Storage Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large-scale distributed storage systems face high degraded read latency and longer reconstruction times due to the need to fetch multiple data and parity blocks during recovery in erasure-coded systems, which results in increased disk and network traffic, making it inefficient to maintain high reliability and availability.
Innovation Solution
Implementing a dynamic erasure coding system that adapts by selecting between a fast recovery code for frequently accessed data and a compact code for less frequently accessed data, with the ability to convert between these codes based on access patterns and system states, using upcoding and downcoding operations that update parity blocks to optimize storage overhead and recovery cost.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If a single erasure code is used to provide high reliability at lower storage cost, then storage overhead is reduced, but recovery cost and reconstruction time increase due to fetching multiple data and parity blocks
Solution Approach 1:
The patent segments the storage system into multiple code stripes, where each stripe can be independently encoded with different erasure codes. This allows the system to apply fast recovery codes to specific stripes that require quick reconstruction while using compact codes in other stripes for optimized storage overhead, thereby resolving the contradiction between storage efficiency and recovery speed.
Solution Approach 2:
The system dynamically selects and switches between different erasure codes (e.g., from compact code to fast recovery code) based on real-time workload conditions, failure patterns, and access frequencies. This dynamic adaptation allows the system to optimize for storage overhead during normal operation and switch to optimizing for recovery speed when failures occur or during degraded reads.
2Reliability
If replication is used to guarantee high reliability and availability, then fault tolerance is improved, but storage cost increases significantly at massive scales
Solution Approach 1:
The patent changes the parameters of erasure codes (such as the ratio of data blocks to parity blocks, and the structure of code stripes) to achieve different trade-offs between reliability and storage overhead. By adjusting these parameters dynamically, the system maintains high fault tolerance while significantly reducing storage costs compared to traditional replication methods.
3Reliability
If multiple data and parity blocks are fetched during recovery in an erasure-coded system, then data integrity is maintained, but disk and network traffic increase resulting in high degraded read latency
Solution Approach 1:
The system performs preliminary actions by pre-computing and caching parity blocks or intermediate computation results during normal operation. When a failure occurs, these pre-computed values are already available, eliminating the need to fetch and compute them during recovery, thus maintaining data integrity while significantly reducing read latency during degraded operations.
Solution Approach 2:
The patent introduces intermediary computation structures and caching mechanisms that act as mediators between the stored data and the recovery process. These intermediaries hold pre-computed parity information or reconstruction data, allowing the system to maintain data integrity through proper erasure code reconstruction while avoiding the need to fetch all original data and parity blocks during recovery.
Data Source
AI summary
Embodiments relate to a system with multiple erasure codes, and selecting and encoding for a write file with one of the codes to mitigate costs associated with storage recovery. The codes include a fast recovery code for frequently accessed data and a higher storage efficiency code for less frequently accessed data. State data is tracked to ascertain frequency of access to the file. One of the erasure codes is dynamically selected based on the tracked data, with the focus of the code select to lower recovery costs, and the data is encoded with the selected erasure code. Accordingly, the original coding of the write file is subject to change based on the tracked state data.