Redundant Write Quorums for Lower Storage Tail Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In storage systems with multiple devices, write tail latency occurs due to the need for all storage devices to complete write operations, leading to increased latency and potential failure, especially when using replication or erasure coding for data durability.
Innovation Solution
The storage system generates redundant write requests to reduce write tail latency by using replication or erasure coding, allowing the operation to be considered complete when a minimum number of write requests succeed, even if some devices take longer, thereby utilizing the quickest responses to complete the write operation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If all storage devices must complete write operations for data durability, then data reliability is improved, but write tail latency increases
Solution Approach 1:
The system performs more write operations than the minimum required for data durability. Instead of waiting for all n storage devices to complete writes, the system writes to n+k devices using erasure coding, where k additional writes provide redundancy that allows the system to tolerate failures and reduces tail latency by having extra completion options.
Solution Approach 2:
The system changes the parameter of write completion threshold from requiring all n devices to requiring only a quorum (n/2 + 1 or similar threshold). This parameter change allows the system to declare write completion earlier, reducing tail latency while maintaining data durability through the redundancy provided by erasure coding.
2Reliability
If replication or erasure coding is used to ensure data durability, then reliability is improved, but the number of write requests increases
Solution Approach 1:
The system segments the write operation into multiple independent write requests distributed across n+k storage devices. Each device receives a portion of the data or encoded chunks, and the segmentation allows parallel execution of writes, reducing overall completion time while maintaining durability through the distributed nature of the segments.
Solution Approach 2:
The system creates redundant copies of data through replication or erasure coding. Instead of writing to a single device, the data is copied and distributed across multiple devices, with k additional copies beyond the minimum n required for reconstruction. This copying provides fault tolerance and reduces tail latency.
3Reliability
If wait for all storage devices to complete writes, then data integrity is ensured, but overall write operation time increases
Solution Approach 1:
The system implements feedback mechanisms to monitor write completion status across storage devices. When a quorum of writes (threshold T) is confirmed complete, the system immediately declares the write operation successful and can proceed with subsequent operations, rather than waiting for all devices. This feedback-driven early termination maintains integrity while improving throughput.
Solution Approach 2:
The system performs preliminary write operations to n+k storage devices with erasure coding applied in advance. By pre-distributing the data and encoded chunks across multiple devices before needing to confirm completion, the system creates a buffer that allows early declaration of success once a quorum is reached, improving throughput without sacrificing integrity.
Data Source
AI summary
In order to reduce write tail latency, a storage system generates redundant write requests when performing a storage operation for an object. The storage operation is determined to be effectively complete when a minimum number of write requests have completed. For example, the storage system may generate twelve write requests and also generate four redundant write requests for a total of sixteen write requests. The storage system considers the object successfully stored once twelve of the sixteen writes complete successfully. To generate the redundant writes, the storage system may use replication or erasure coding. For replication, the storage system may issue a redundant write request for each of n chunks being written. For erasure coding, the storage system may use rateless codes which can generate unlimited number of parity chunks or use an n+k+k′ erasure code which generates an additional k′ encoded chunks, in place of an n+k erasure code.