Fault Injection System for Distributed Resiliency Assessment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for testing the resiliency of distributed systems are time-consuming and may not comprehensively detect failures, especially when the system configuration changes, as they often rely on manual generation of failure scenarios or varying machine percentages, which can miss intensity-dependent failures.
Innovation Solution
A fault injection system that automatically generates fault profiles by selecting possible values for dimensions such as fault type, number of machines, and duration, and injects faults into the system to assess resiliency, using techniques like random, linearly increasing, or exponentially increasing fault profiles to identify potential failures.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual generation of failure scenarios is used to test distributed system resiliency, then testing can be performed with real user data, but the process becomes very time-consuming and testing may be less than comprehensive
Solution Approach 1:
The system performs preliminary actions by automatically generating comprehensive fault profiles and injection scenarios before actual resiliency testing begins. The fault injection system pre-defines multiple fault profiles with varying dimensions (fault type, intensity, duration, target components) so that when testing is needed, extensive test coverage is already prepared, eliminating the time-consuming manual scenario generation while maintaining comprehensive testing of system resiliency
Solution Approach 2:
The fault injection system enables self-service by automatically generating fault profiles and injection scenarios without requiring manual intervention. The system uses algorithms to autonomously create diverse fault scenarios, select target components, determine injection parameters, and execute tests, thereby eliminating the time-consuming manual process while ensuring comprehensive test coverage through systematic exploration of fault space
2Reliability
If failure scenarios are tested on random percentages of machines, then system resiliency can be verified, but intensity-dependent failures may not be detected
Solution Approach 1:
The system applies parameter changes by systematically varying fault injection parameters across multiple dimensions including fault intensity (from mild to severe), duration (short to long), target component selection, and affected machine percentage. This multi-dimensional parameter exploration enables detection of intensity-dependent failures that would be missed by simple random percentage testing, while maintaining efficient automated execution throughout the test space
Solution Approach 2:
The fault injection system adds another dimension to testing by introducing structured fault profile dimensions such as fault intensity levels, duration categories, and component-specific parameters. Instead of merely varying machine percentages randomly, the system explores additional dimensions of fault characteristics, enabling comprehensive detection of intensity-dependent and component-specific failures while maintaining systematic and efficient test execution
3Measurement precision
If comprehensive fault testing is performed manually, then all failure scenarios can be detected, but the process becomes extremely time-consuming
Solution Approach 1:
The system replaces the mechanical manual process of fault scenario generation and execution with an automated computational system. Algorithms automatically generate comprehensive fault profiles, select target components, determine injection parameters, and execute tests across the distributed system. This substitution maintains comprehensive failure detection capability while dramatically improving testing efficiency and productivity through automated execution of extensive test scenarios
Solution Approach 2:
The automated system systematically explores the fault parameter space by changing parameters such as fault intensity, duration, target component type, and affected machine percentage across multiple structured dimensions. This parameter-driven approach ensures comprehensive coverage of potential failure scenarios while maintaining efficient automated execution, achieving both high measurement precision in failure detection and high productivity in test execution
Data Source
AI summary
A method and system for assessing resiliency of a system is provided. A fault injection system may, for each of a plurality of dimensions of a fault profile, access an indication of possible values for the dimension, which may be specified by a user. The fault injection system may, for each of a plurality of fault profiles, automatically create the fault profile by, for each of the plurality of dimensions, selecting by the computing system a possible value for that dimension. For at least some of the fault profiles, the fault injection system injects a fault based on the fault profile into the system and determines whether a failure was detected while the fault was injected.