High Availability Resource Restart Counter Management
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
High Availability (HA) systems face inefficiencies in resource restart management, where resources may remain unstable despite reaching the MAX_RESTARTS limit, leading to unnecessary manual intervention and prolonged system resource wastage due to continuous bouncing.
Innovation Solution
Implementing a mechanism to detect stability conditions, such as tracking runtime or performing stability tests, to reset the RESTART_COUNTER, allowing stable resources to be automatically restarted and preventing continuous bouncing by defining a finite MAX_RESTARTS limit.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If MAX_RESTARTS is set to a large value to reduce administrator intervention, then the frequency of manual intervention is reduced, but the resource will bounce for a longer time for unrecoverable failures
Solution Approach 1:
The system performs preliminary stability testing before allowing restarts. The stability counter is incremented only after the resource has been stable for a predetermined period, preventing premature restart attempts and reducing unnecessary administrator intervention while maintaining quick response to actual failures
Solution Approach 2:
The system implements feedback mechanisms by monitoring resource stability over time and adjusting restart permissions accordingly. The stability counter provides feedback on resource health, enabling automatic restart decisions that balance reducing administrator intervention with preventing prolonged bouncing
2Loss of time
If MAX_RESTARTS is set to a small value to prevent continuous bouncing, then the bounce time is reduced, but manual intervention is required even when the resource would function well if restarted
Solution Approach 1:
The system performs preliminary stability verification by tracking the stability counter. Only resources that have demonstrated stability for a predetermined period have their restart counter reset, ensuring that restarts are performed only when likely to succeed, thereby reducing MAX_RESTARTS without increasing manual intervention
Solution Approach 2:
The system enables self-service automatic restarts for stable resources by automatically resetting the restart counter when stability conditions are met. This eliminates the need for manual administrator intervention for resources that have stabilized, while maintaining strict limits on resources that continue to fail
3Reliability
If the RESTART_COUNTER is not reset after reaching MAX_RESTARTS, then continuous bouncing is prevented, but stable resources cannot be automatically restarted
Solution Approach 1:
The system performs preliminary stability monitoring before resetting the restart counter. By tracking whether the resource has been stable for a predetermined period, the system ensures that counter resets only occur for genuinely stable resources, preventing continuous bouncing while restoring automatic restart capability when appropriate
Solution Approach 2:
The stability counter provides feedback on resource health status. When this counter reaches the predetermined stability threshold, it triggers a reset of the restart counter, enabling automatic restarts for stable resources while maintaining protection against continuous bouncing of unstable resources
Data Source
AI summary
Techniques are provided for managing a resource in a High Availability (HA) system. The techniques involve incrementing a count when a particular type of remedial action is performed on a resource, so that the count that reflects how often the particular type of remedial action has been performed for the resource. When it is determined that the resource has been in stable operation, the count is automatically reduced. After a failure, the count is used to determine whether to attempt to perform the particular type of remedial action on the resource. Examples of remedial actions include restarting the resource, and relocating the resource to another node of a cluster. By using the count, the system insures that a faulty resource does not get constantly “bounced”. By reducing the count when a resource has become stable, there is less likelihood that failure of otherwise stable resources will require manual intervention.