Resilience Testing Controller for Distributed Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing resilience testing systems, such as Chaos Monkey, are not customizable for individual services or compute groups and face challenges when dealing with large and complex distributed networks, leading to issues like API rate limiting and inability to configure testing frequency or probability for specific services.
Innovation Solution
A resilience testing system that maintains its own database of testing parameters for each service, allowing for customizable scheduling and frequency of tests without relying on container management systems, enabling more flexible and targeted resilience testing in distributed networks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing resilience testing systems (e.g., Chaos Monkey) are used, then basic resilience testing can be performed, but customization for individual services or compute groups is not possible and API rate limiting occurs in large distributed networks
Solution Approach 1:
The system segments the distributed network into individual compute groups, each with its own resilience testing configuration. The controller divides the large network into manageable units that can be tested independently with customized parameters, avoiding the complexity of managing testing for the entire network as a single unit.
Solution Approach 2:
The patent introduces a new dimension of organization by implementing a controller that maintains a database of testing parameters for each compute group. This adds a hierarchical layer (controller → compute groups → compute instances) that enables customization without increasing overall system complexity.
2Reliability
If resilience testing is performed frequently to ensure reliability, then detection of network failures improves, but system performance and availability may be degraded
Solution Approach 1:
The system dynamically adjusts testing parameters based on service importance and current system state. The controller can modify testing frequency and probability of failures for different compute groups, allowing more frequent testing of critical services while reducing testing intensity for less critical ones, thus maintaining reliability without uniformly degrading performance.
Solution Approach 2:
The system applies partial testing action by allowing configuration of testing probability less than 100%. This enables resilience testing to be performed at controlled intensities, balancing detection capability with system performance requirements.
3Stability of the object's composition
If centralized control is used to manage testing parameters, then consistency across services is improved, but the burden on container management systems increases
Solution Approach 1:
The patent extracts the testing parameter management functionality from the container management system and places it in a separate controller. This separation removes the burden from the container management system while maintaining centralized control through the controller's database of testing parameters for each compute group.
4Ease of operation
If testing parameters are hardcoded in the container management system, then ease of operation is improved, but adaptability to different services is reduced
Solution Approach 1:
The system transitions from static hardcoded parameters to dynamic configurable parameters stored in a database. The controller can adjust testing parameters for different compute groups based on service requirements, maintaining ease of operation through centralized management while achieving adaptability through database-driven configuration.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Method for testing the resilience of a distributed network. The distributed network comprising one or more services, each service associated with compute groups, each compute group comprising active compute instances. The method comprising: for a service from the plurality of services: retrieving test parameters, the test parameters indicating at least the schedule for performing a resilience test on the service, unique identifiers of compute groups registered for resilience testing, and the probability of terminating a compute instance; determining whether to terminate a compute instance based on the probability of terminating the compute instance; in response to determining to terminate the compute instance: randomly selecting a compute group from the compute groups registered for resilience testing; receiving a list of active compute instances for the selected group; randomly selecting an active compute instance from the list of compute instances for terminating; and causing the selected compute instance to terminate.