Global Retry Constraints in Distributed Applications
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In distributed applications using service-oriented architecture, retry storms can occur due to excessive retry attempts, leading to system overload and congestion collapse, especially when individual resources make decisions about retries independently without considering overall system activity.
Innovation Solution
Implementing global or transaction-wide constraints on retry attempts using token buckets or a retry manager to limit the total number of retries across all resources involved in a transaction, ensuring that the entire transaction is considered as a single unit, rather than individual resources making independent decisions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If individual resources make independent retry decisions to handle transient disruptions, then the system can recover from brief failures, but retry storms and congestion collapse can occur when too many retries are attempted in quick succession
Solution Approach 1:
The patent introduces a coordinator resource that acts as an intermediary between individual resources and the retry mechanism. This coordinator receives retry requests from multiple resources, manages a shared retry budget, and decides whether to allow or deny each retry attempt. By centralizing retry control, the system prevents retry storms while maintaining the ability to recover from transient failures.
Solution Approach 2:
The system implements feedback mechanisms where the coordinator monitors the state of shared resources and the number of retry attempts made. Based on this feedback, the coordinator dynamically adjusts retry decisions, allowing retries when resources are available and denying them when the system is approaching capacity limits, thereby preventing congestion collapse.
2Object-affected harmful factors
If a fail fast policy is used to avoid retry storms, then system overload is prevented, but negative end-user experiences occur more frequently due to abandoned recoverable transactions
Solution Approach 1:
The patent implements dynamic retry management where the retry policy is not fixed but adapts based on current system conditions. The coordinator adjusts the number of allowed retries and the timing of retry attempts based on resource availability and system load, enabling the system to be more aggressive during low-load periods and more conservative during high-load periods.
Solution Approach 2:
The system performs preliminary assessment of retry requests by the coordinator before allowing them to proceed. The coordinator evaluates whether a retry should be permitted based on current system state, resource availability, and retry history, making a decision in advance that prevents both premature failures and excessive retries.
3Object-affected harmful factors
If global constraints are applied to limit total retries across all resources, then retry storms are prevented, but the complexity of managing coordinated retry logic increases
Solution Approach 1:
The patent segments the retry management function into separate components: individual resources that generate retry requests, a coordinator that manages the shared retry budget, and a decision-making layer that evaluates each retry request. This segmentation allows each component to remain relatively simple while the system as a whole achieves sophisticated retry control.
Solution Approach 2:
The coordinator resource serves multiple functions: it receives retry requests from various resources, manages the shared retry budget, makes authorization decisions, and provides feedback to requesting resources. This multi-functionality consolidates complexity into a single component rather than distributing it across multiple resources.
Data Source
AI summary
It is determined that a workflow comprising inter-resource requests of a distributed application is to be initiated. In response to detecting, at a first resource during execution of the workflow, that a triggering condition for retrying a request is met, and that a workflow-level retry budget of the workflow indicates that a retry is permitted, the retry is attempted and the budget is modified to indicate that the retry has been attempted. In response to detecting, at another resource, that a triggering condition for retrying another request is met, and that the workflow-level retry budget of the workflow indicates that a retry is not permitted, an indication that the workflow has failed is generated.