Fault Injection for FMS SaaS Platforms
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional software testing approaches are insufficient for identifying potential failures in distributed and cloud software systems, which are critical for ensuring high availability and reliability due to factors like geographical distribution, security, and human errors.
Innovation Solution
A method and system for fault injection that observes platform behavior, performs fault injection based on user context and service level criticality, and analyzes responses to identify and mitigate potential failures without impacting end customers, including randomized scaling, message modification, and infrastructure removal.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional software testing approaches are used, then the testing process is simple and familiar, but they are insufficient for identifying potential failures in distributed and cloud software systems
Solution Approach 1:
The system performs preliminary actions by observing platform behavior and building a hypothesis about steady state behavior before fault injection. This preparation phase enables the system to detect deviations caused by faults more effectively, improving failure identification capability without requiring complex testing procedures during actual operation.
Solution Approach 2:
The system implements feedback mechanisms by continuously observing platform behavior, comparing it against the steady state hypothesis, and using this information to detect faults. The feedback loop allows the system to adapt to changing conditions and improve its ability to identify potential failures in distributed and cloud environments.
2Reliability
If fault injection is performed to assess platform resilience, then potential failures can be identified, but it may impact end customers and service availability
Solution Approach 1:
The system applies partial fault injection by introducing disturbances that are sufficient to detect failures but not excessive enough to cause significant service degradation. By carefully controlling the fault injection intensity and scope, the system can identify potential failures while minimizing impact on end customers and maintaining service availability.
Solution Approach 2:
The system uses an intermediary approach by first establishing a steady state hypothesis through observation, then using this hypothesis as a reference to detect faults. This intermediary reference frame allows for sensitive fault detection without requiring direct manipulation of production systems, thereby reducing impact on end customers.
3Adaptability or versatility
If multiple fault types are injected to comprehensively test the platform, then coverage is improved, but the complexity of managing and analyzing results increases
Solution Approach 1:
The system implements a universal fault detection framework that can handle multiple fault types through a common steady state hypothesis mechanism. By using a unified approach that works across different fault scenarios, the system achieves comprehensive coverage without proportionally increasing management complexity. The same observation and comparison methodology applies regardless of the specific fault type being tested.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Disclosed are methods, systems, and non-transitory computer-readable medium for fault injection and ensuring failsafe FMS SaaS platforms. For instance, the method may include observing a behavior of a platform; building a hypothesis about a steady state behavior based on the observing the behavior of the platform; determining whether a request for fault injection has been received; in response to receiving the request for the fault injection, performing the fault injection; observing a response of the platform to the fault injection; determining whether the fault injection is at an end; and in response to determining that the fault injection is at the end, collect and analyze data of observed response(s).