Automated Fault Injection via Cloud Control Plane
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current chaos engineering approaches for testing the resilience of computer systems are limited, especially when third-party agents are used, as they often restrict the types of failures that can be tested due to security mechanisms and resource constraints, leading to inaccurate results and performance degradation.
Innovation Solution
An automated testing framework within a cloud provider network that allows for direct injection of faults into virtual machines, enabling comprehensive testing without the limitations of third-party agents, by utilizing the control plane to manipulate resources and simulate various failure scenarios.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If third-party agents are used for chaos engineering testing, then security mechanisms are maintained, but the types of failures that can be tested are restricted and results become inaccurate
Solution Approach 1:
The patent introduces a control plane as an intermediary between the testing framework and the cloud infrastructure. This control plane operates with elevated privileges to inject faults directly into the system under test, bypassing the limitations of third-party agents while maintaining security through controlled access. The control plane mediates between the testing requirements and system constraints, enabling comprehensive fault injection without compromising security.
Solution Approach 2:
Instead of using third-party agents with limited permissions to test system resilience, the patent inverts the approach by using the cloud provider's own control plane with full system access to perform the testing. This inversion allows the testing framework to directly manipulate system resources and simulate failures that would be inaccessible to external agents, thereby achieving both security and comprehensive test coverage.
2Productivity
If third-party agents are deployed in virtual machines, then testing can be performed, but system resources are consumed and performance degradation occurs
Solution Approach 1:
The patent implements self-service by having the cloud provider's control plane perform the fault injection and testing operations directly, without requiring external third-party agents. The control plane uses its own built-in capabilities to manipulate system resources and simulate failures, eliminating the need for additional agent software that would consume virtual machine resources. This self-service approach maintains high testing efficiency while avoiding the performance degradation caused by resource-constrained agents.
3Adaptability or versatility
If direct fault injection into virtual machines is implemented, then comprehensive testing without agent limitations is achieved, but system control complexity increases
Solution Approach 1:
The control plane is designed with multi-functionality to handle diverse fault injection scenarios through a unified interface. It can simulate various failure types (hardware failures, software errors, network issues, resource exhaustion) using the same underlying mechanism of direct system access. This universality allows comprehensive testing coverage without proportionally increasing control plane complexity, as the same control structures handle multiple failure modes.
Data Source
AI summary
Disclosed are various embodiments for automated testing of systems and applications. A computing device can receive a command to execute a test, the test comprising at least one fault instruction and an alarm, the alarm being specified as an evaluation criterion for the test. The computing device can then send the fault instruction to a fault injection service. Next, the computing device can receive a message from a monitoring service, the message indicating that the alarm has been triggered subsequent to sending the fault instruction to the fault injection service. Subsequently, the computing device can determine that the test has failed based at least in part on the alarm being triggered. Finally, the computing device can send a failure notification to a predefined recipient in response to detecting that the alarm has been triggered.