EOS System Transaction Pruning for Storage Exhaustion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Storage systems face resource exhaustion and increased recovery time due to the need to log and replay non-idempotent operations, which are not inherently idempotent and require explicit logging to ensure exactly once semantics.
Innovation Solution
Implementing an exactly once semantics (EOS) system where non-idempotent operations are wrapped in transactions with unique identifiers, allowing for periodic checkpoint acknowledgments that enable pruning of logged transactions, reducing storage consumption and minimizing replay time by ignoring acknowledged transactions during recovery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If non-idempotent operations are logged on storage resources until completion is acknowledged, then exactly once semantics is ensured, but storage consumption increases and recovery time increases
Solution Approach 1:
The patent changes the state of logged transactions from 'pending' to 'acknowledged' by updating a parameter (acknowledgement status) that allows the system to differentiate between transactions requiring full logging and those that can be pruned. This parameter change enables the resolution of the contradiction by allowing storage space to be freed while maintaining exactly once semantics through the tracking of acknowledged transactions.
Solution Approach 2:
The patent implements discarding of logged transactions after they are acknowledged, recovering storage space that would otherwise be consumed indefinitely. The system maintains a record of acknowledged transactions to ensure they are not replayed during recovery, allowing safe discarding of the original log entries. This directly addresses the storage consumption issue while preserving reliability.
2Reliability
If non-idempotent operations are logged on storage resources until completion is acknowledged, then exactly once semantics is ensured, but recovery time increases
Solution Approach 1:
The patent extracts the acknowledgement status information from the full transaction log, separating the essential recovery information (which transactions were acknowledged) from the detailed operation logs. This extraction allows the system to quickly identify which transactions need replay during recovery without having to process or validate every logged transaction, significantly reducing recovery time while maintaining exactly once semantics.
Solution Approach 2:
During recovery, the system discards (skips) transactions that are identified as already acknowledged, recovering time that would be spent replaying these transactions. The acknowledged transaction records are retained to guide this selective discarding, enabling fast recovery by focusing only on unacknowledged transactions that actually need replay.
3Quantity of substance
If periodic checkpoint acknowledgments are implemented, then storage resources can be pruned, but system complexity increases
Solution Approach 1:
The patent implements periodic checkpoint acknowledgments where the system periodically updates the state of transactions from pending to acknowledged. This periodic action creates natural opportunities to prune old log entries while maintaining a manageable record of acknowledged transactions. The periodic nature of this operation simplifies the complexity by establishing a regular, predictable maintenance rhythm rather than requiring continuous complex management of transaction states.
Data Source
AI summary
A flow control technique prevents exhaustion of storage resources in an exactly once semantics (EOS) system of a storage input/output stack executing on a node of a cluster. An EOS server may service transactions sent by an EOS client and issue replies with results to the EOS client. In order to replay the transactions during normal operation after recovery from a crash, the EOS server persistently stores the transactions in the storage resources until an acknowledgement of completion is received from the EOS client for each pending transaction. The EOS client may issue a checkpoint acknowledgement, e.g., as a prune record, after a periodic interval that marks the completion of all pending transactions issued prior to the record. The EOS server need only log the prune record (rather than each pending transaction) to thereby prevent exhaustion of the storage resources, while also minimizing logging overhead of the server. In response to the crash and during replay of the transactions, the EOS server may employ the prune records to ignore those transactions that have already been acknowledged by the EOS client, thereby reducing time required for replay of the transactions.