Fault Injection Engine for Cloud Platform Resilience Testing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Developers building cloud applications on cloud platforms lack the ability to evaluate the resilience of their applications to infrastructure failures before production, as they have no control over the cloud platform infrastructure, leading to potential unforeseen issues during deployment.
Innovation Solution
A cloud platform system that includes a fault condition injection engine to generate specific fault conditions on selected nodes, allowing users to simulate various types of failures in a controlled sandbox environment, enabling developers to test the resilience of their applications through user-defined experimentation parameters.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If developers use cloud platform infrastructure without control, then they can deploy applications quickly, but they cannot evaluate resiliency to infrastructure failures before production
Solution Approach 1:
The system performs preliminary fault condition injections into the sandbox environment before production deployment. By simulating infrastructure failures in advance within the sandbox, developers can evaluate application resiliency and make necessary adjustments before deploying to production, thus resolving the contradiction between quick deployment and reliability verification
Solution Approach 2:
A sandbox environment is introduced as an intermediary between development and production. This sandbox allows controlled fault injections and resiliency testing without affecting the actual production infrastructure, enabling developers to verify application robustness while maintaining rapid deployment capabilities
2Reliability
If developers are given control over cloud platform infrastructure, then they can test fault conditions, but it increases system complexity and resource requirements
Solution Approach 1:
The system extracts fault testing capabilities from the production infrastructure and relocates them to a separate sandbox environment. This extraction allows fault condition injections without compromising production systems, reducing the complexity burden on the main cloud platform while maintaining comprehensive testing capabilities
Solution Approach 2:
The infrastructure is segmented into production environments and sandbox environments with distinct purposes. The sandbox handles all fault testing and resiliency evaluation, while production focuses on application deployment. This segmentation reduces overall system complexity by isolating testing operations from critical production systems
3Measurement precision
If fault conditions are injected into production infrastructure, then real-world scenarios can be tested, but it risks causing actual service disruptions
Solution Approach 1:
The sandbox serves as an intermediary that replicates production infrastructure characteristics without being the actual production system. Fault conditions are injected into this intermediary sandbox environment, providing accurate measurement of application resiliency while completely isolating production services from harmful fault injection effects
Solution Approach 2:
A sandbox environment is created as a copy of the production infrastructure topology and configuration. This copy allows realistic fault scenario testing with high measurement precision while ensuring that any fault effects are confined to the sandbox copy and cannot affect the actual production services
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A computer system is provided that includes a cloud platform that includes a plurality of nodes. Each node includes a processor configured to run virtual machines. The cloud platform includes a fault condition injection engine configured to generate fault conditions on selected nodes of the plurality of nodes. The computer system further includes a user interface system configured to receive user input of fault condition experimentation parameters from a user for a target virtual machine associated with the user. The cloud platform allocates a set of nodes of the plurality of nodes for a controlled sandbox environment configured to run the target virtual machine of the user. The fault condition injection engine generates fault conditions on the allocated set of nodes based on the fault condition experimentation parameters.