Chaos Level Engine for Controlled Production Testing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing chaos engineering techniques struggle to introduce appropriate failures in a computing system's production environment without causing significant disruptions, making it difficult to assess the system's resilience and robustness effectively.
Innovation Solution
The introduction of a chaos scale that augments a chaos network controller, allowing users to inject failures based on network topology, measuring the degree of failure injections with predictable outcomes, and automatically generating and executing chaos experiments.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If chaos is introduced into a production environment to test robustness, then the system's resilience understanding is improved, but the production environment may become inoperable or experience significant disruptions
Solution Approach 1:
The patent applies preliminary action by performing chaos experiments in a cloned production environment before deploying changes to the actual production system. The system creates a replica of the production environment, introduces chaos conditions to the clone, and validates resilience without risking the operational production environment. This resolves the contradiction by enabling resilience testing while preventing harmful disruptions to the actual production system.
Solution Approach 2:
The patent uses a cloned production environment as an intermediary between the tester and the actual production system. This intermediary allows chaos conditions to be introduced and observed without directly affecting the production environment. The clone serves as a safe mediator that enables resilience assessment while isolating the production system from harmful disruptions.
2Adaptability or versatility
If hard failures are injected into the computing system to test robustness, then the failure scenario coverage is improved, but the impact on the production environment becomes significant
Solution Approach 1:
The patent applies preliminary action by injecting hard failures into a cloned production environment rather than the actual production system. This allows comprehensive failure scenario coverage to be achieved in the clone while preventing significant impact on the operational production environment. The cloned environment serves as a safe testing ground for extreme failure conditions.
3Ease of operation
If simple statistics-defined chaos testing is used, then the testing simplicity is maintained, but the testing sufficiency becomes inadequate for dynamic networking conditions
Solution Approach 1:
The patent applies parameter changes by dynamically adjusting chaos injection parameters based on the cloned production environment's response and characteristics. Rather than using fixed simple statistics, the system modifies chaos parameters (such as failure probability, intensity, and type) based on observed system behavior, thereby achieving adequate testing sufficiency for dynamic networking conditions while maintaining operational simplicity through automated parameter adjustment.
Data Source
AI summary
A method includes receiving, at a chaos level engine, initial input parameters. The method may further include, with the chaos level engine, determining scaled input parameters based on the initial input parameters. The scaled input parameters define how the initial input parameters effect a computing environment to be tested. The method may further include, with the chaos level engine determining a chaos level for performing a chaos experiment on the computing environment based on the scaled input parameters and sending the chaos level to the computing environment for the chaos experiment. The method may further include, with the chaos level engine, receiving, from the computing environment, feedback defining an impact caused by the chaos experiment created at the computing environment and an intended level of chaos.