Adaptive Circuit Breaker Thresholds for Service Reliability
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing circuit breaker configurations for dependency calls are ad-hoc and static, lacking rationale and target metrics, leading to ineffective failure management that can cause significant business impact due to prolonged service failures.
Innovation Solution
Adaptive circuit breaker management systems that measure integration reliability between services to determine dynamic thresholds, using exponential distribution to model failure probability and adjust configurations based on changing load patterns and criticality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If static default values are used for circuit breaker configurations, then device complexity is reduced and ease of operation is improved, but reliability deteriorates due to inability to adapt to changing load patterns and failure rates
Solution Approach 1:
The circuit breaker configuration transitions from static default values to dynamic adaptive thresholds. The failure threshold is continuously adjusted based on measured integration reliability from historical data, allowing the system to adapt to changing load patterns and failure rates while maintaining service resilience
Solution Approach 2:
The system implements feedback by measuring integration reliability between services and using this information to adjust circuit breaker thresholds. The measured reliability data feeds back into the configuration process, creating a closed-loop system that continuously optimizes failure thresholds based on actual system behavior
2Measurement precision
If ad-hoc failure settings are used without target metrics, then ease of manufacture is improved, but measurement precision deteriorates due to lack of basis to verify correctness
Solution Approach 1:
The manual configuration process is replaced with an automated system that measures integration reliability and calculates appropriate failure thresholds. Instead of manually setting thresholds without metrics, the system automatically computes them based on measured reliability data, eliminating the need for expert judgment while improving accuracy
Solution Approach 2:
The circuit breaker configuration system performs self-configuration by automatically measuring integration reliability and determining appropriate failure thresholds without requiring manual intervention. The system serves itself by generating configurations based on its own measured performance data
3Adaptability or versatility
If static circuit breaker configurations are used, then adaptability deteriorates due to inability to account for changing load patterns, but device complexity is reduced
Solution Approach 1:
The circuit breaker configuration adapts dynamically to changing load patterns by continuously measuring integration reliability and adjusting failure thresholds accordingly. The system transitions from static to dynamic configuration, allowing it to respond to varying system conditions while maintaining simplicity through automated measurements
Data Source
AI summary
Systems and methods for adaptively managing services using circuit breakers are disclosed. A disclosed method includes: measuring an integration reliability of interaction between a first service executed at a first device and a second service executed at a second device during a past time period, the first service being dependent on the second service for execution; determining an adaptive threshold for a circuit breaker regarding a first time period based on the integration reliability; and executing the first service based on a request within the first time period by calling the second service through the circuit breaker, wherein the circuit breaker trips to stop the calling of the second service when a failure rate of interaction between the first service and the second service in the first time period exceeds the adaptive threshold.