Chaos Fault Scenario Generation Using Telemetry Feedback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current chaos engineering systems require manual configuration of experiments, which is time-consuming and limits the scope and breadth of simulated failures, making it difficult to effectively test a computing system's resilience.
Innovation Solution
Automated fault scenario generation using a chaos engine that randomly creates disruptions and failures based on a service's configuration, iteratively applies these scenarios, and analyzes telemetry data to identify vulnerabilities.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If manual configuration of chaos experiments is used, then the scope and breadth of simulated failures are limited, but the time required for service validation increases
Solution Approach 1:
The system dynamically generates fault scenarios based on service configuration and telemetry data, transitioning from static manual experiments to adaptive automated experiments that evolve with the service
Solution Approach 2:
The system performs self-validation by automatically generating and executing fault scenarios based on its own telemetry data and service configuration, eliminating the need for external manual intervention
2Reliability
If manual creation of chaos experiments is performed, then experiments can be customized, but the process is time-consuming and repetitive
Solution Approach 1:
The system collects and analyzes telemetry data in advance to identify vulnerabilities before formal validation, preparing fault scenarios proactively rather than reactively
Solution Approach 2:
The system uses telemetry data feedback to continuously improve and refine fault scenario generation, creating a closed-loop validation process that learns from each experiment
3Adaptability or versatility
If each chaos experiment is manually crafted, then the experiments can be precisely controlled, but the scope is limited to foreseeable failures
Solution Approach 1:
The system creates a universal fault scenario generation mechanism that handles multiple types of failures across different service components through a single automated process
Solution Approach 2:
The system varies fault scenario parameters dynamically based on telemetry data analysis, adjusting the nature and intensity of failures to match actual service vulnerabilities
Data Source
AI summary
Aspects of the disclosure include methods and systems for performing automated fault scenario generation for chaos engineering. Aspects include obtaining a configuration of a service under test, obtaining a first plurality of fault scenarios, and applying each of the first plurality of fault scenarios to the service under test. Aspects also include recording telemetry data regarding an operation of the service under test under each of the fault scenarios, selecting, based on the telemetry data, a first fault scenario from the fault scenarios, and generating a second plurality of fault scenarios. Aspects further include applying each of the second plurality of fault scenarios to the service under test, recording telemetry data regarding the operation of the service under test under each of the second plurality of fault scenarios, and identifying a vulnerability of the service under test based on the recorded telemetry data.