Erasure Coded Shard Write Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data storage systems face challenges in achieving optimal write latency while maintaining a predetermined level of data reliability, as existing methods are limited by the slowest storage device in a cluster, leading to delayed write completion and reduced performance.
Innovation Solution
Implementing an overprovisioned number of erasure coded shards, where additional shards beyond the threshold required for reliability are written, allowing faster devices to complete writes first and reducing overall latency without compromising data reliability.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the minimum number of erasure coded shards required for data reliability is written, then data reliability is maintained, but write latency increases due to waiting for the slowest storage device
Solution Approach 1:
The patent applies partial action by writing only the minimum number of shards (k) required for data reconstruction, rather than writing all encoded shards (n). This allows the system to achieve the required reliability level without waiting for slower storage devices to complete writes of unnecessary shards, thereby reducing write latency while maintaining data reliability.
Solution Approach 2:
The patent changes the parameter of shard count from the fixed minimum (k) to an overprovisioned number (n) where n > k. By adjusting this parameter, the system can trade off between reliability and write latency, allowing faster devices to complete writes first and reducing overall write latency while maintaining the ability to reconstruct data from any k shards.
2Reliability
If all erasure coded shards are written to ensure data reliability, then data reliability is maximized, but write completion time increases
Solution Approach 1:
The patent implements partial action by writing only k shards out of n total encoded shards, where k is the minimum required for data reconstruction. This partial writing approach reduces write completion time significantly while maintaining data reliability, as the system does not need to wait for all n shards to be written before considering the write operation complete.
Solution Approach 2:
The patent applies preliminary action by pre-calculating which k shards are sufficient for data reconstruction and prioritizing their write completion. The system can determine write completion as soon as k shards are successfully written, without waiting for the remaining n-k shards, thereby reducing write completion time while ensuring data reliability.
3Reliability
If the number of shards is increased to provide redundancy, then data reliability improves, but write latency increases due to additional write operations
Solution Approach 1:
The patent applies partial action by utilizing only the necessary portion of encoded shards (k out of n) for write completion determination. Although n > k provides enhanced redundancy and reliability, the system only waits for k shards to complete writing, thereby achieving high reliability without the full write latency penalty of writing all n shards.
Solution Approach 2:
The patent changes the shard configuration parameter from EC k/k (no redundancy) to EC k/n where n > k (with redundancy). This parameter change allows the system to achieve better data reliability through overprovisioning while controlling write latency by only requiring k shard completions rather than all n shards.
Data Source
AI summary
A data storage system includes a plurality of Data Storage Devices (DSDs). A write command is sent to each DSD of the plurality of DSDs to each store one or more erasure coded shards of an overprovisioned number of shards. The overprovisioned number of shards is generated from an erasure coding on data to provide at least a predetermined level of data reliability. Write complete indications are received for a threshold number of shards less than the overprovisioned number of shards, with each write complete indication indicating that one or more shards of the overprovisioned number of shards has been stored in a DSD. It is determined that the data has been written with at least the predetermined level of data reliability after receiving write complete indications for the threshold number of shards, but before receiving write complete indications for all of the overprovisioned number of shards.