Lost Write Handling with Storage Completion Notifications
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Distributed storage systems face issues with delayed or lost responses to access requests due to network failures or congestion, leading to unpredictable outcomes when retrying or ignoring non-responded requests, especially for non-idempotent operations.
Innovation Solution
Implement an execution clock to timestamp storage operations, maintain counters for pending operations, and suspend subsequent accesses until it is determined that a response is not expected, using notification mechanisms to ensure completion status is known before resuming operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If timeouts are used to handle unexpected failures in distributed storage systems, then the system can detect failed operations, but the requesting storage system cannot determine whether the operation was eventually performed, leading to unpredictable outcomes when retrying
Solution Approach 1:
The patent implements a feedback mechanism where the storage system sends notifications back to the requesting system about the completion status of operations. When an operation is eventually performed (even after timeout), the system notifies the requester, allowing it to determine whether retry is necessary. This resolves the information loss problem by providing explicit feedback about operation outcomes.
Solution Approach 2:
The patent introduces an intermediary notification mechanism that mediates between the storage system and the requesting system. Instead of the requester directly guessing whether an operation succeeded, the intermediary notification system provides authoritative information about operation completion, enabling reliable decision-making about retries.
2Loss of time
If the system waits for responses within a determined time, then response timing is predictable, but network congestion or component failures cause responses to be delayed or lost beyond the determined time
Solution Approach 1:
The patent performs preliminary actions by establishing notification mechanisms in advance that can report operation completion even after timeouts. The system prepares tracking and notification infrastructure before failures occur, enabling it to handle delayed responses reliably without sacrificing timing predictability for successful operations.
Solution Approach 2:
The patent implements beforehand cushioning by creating a safety net of notification mechanisms that cushion against the harmful effects of lost or delayed responses. Even when responses are delayed beyond determined time due to network congestion or failures, the cushioning notification system ensures completion status is eventually communicated, preventing data loss or incorrect retry decisions.
3Reliability
If non-responded requests are retried, then the system attempts to recover lost operations, but it may lead to problems if the operation was already performed, especially when operations are not idempotent
Solution Approach 1:
The notification mechanism provides feedback about whether an operation has already been performed before a retry is attempted. By checking completion status through notifications, the system可以避免 executing operations that have already succeeded, preventing harmful side effects of incorrect retries on non-idempotent operations.
4Object-generated harmful factors
If the system ignores non-responded requests, then it avoids potential errors from incorrect retries, but it may miss operations that were actually performed but not acknowledged
Solution Approach 1:
Instead of ignoring non-responded requests, the system uses feedback notifications to determine whether operations were actually performed. The notification mechanism provides the missing information about operation completion status, allowing the system to reliably decide whether to retry or ignore, thus avoiding both incorrect retries and missed operations.
Data Source
AI summary
A method for managing lost content change requests (CCRs), the method includes (i) maintaining, by compute entities of a second storage system (SSS), states of executions of multiple CCRs of multiple categories and of multiple SSS reception times; (ii) receiving, from a first storage system (FSS), at a failure indication reception time (FIRT), a failure indication that is indicative of a certain category of the multiple categories. The certain category is associated with a failure of the FSS to receive a response, from the SSS, to a certain CCR of the certain category, that was presumably sent from the FSS to the SSS; (iii) determining, based on states of executions that are related at least to the certain category, when the SSS has completed to execute all CCRs of at least the certain category that were received by the SSS up to the FIRT; and (iv) transmitting, by the SSS, a SSS completion indication that indicates that the SSS has completed to execute all tasks that were received by the SSS up to the FIRT.