Automated Service Dependency Mapping for Cloud Fault Injection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for testing the availability of cloud-based services are inefficient due to the complexity of distributed systems and human error in manually creating fault models, which often miss critical failure scenarios and are costly to maintain, especially in rapidly changing environments.
Innovation Solution
A system that automatically creates and updates a service directory of dependencies, allowing for targeted and precise fault injection with monitored impacts, using a limited set of faults such as transport layer faults, memory pressure, and processor pressure, to identify failure points in a guided manner.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual fault models are created for testing service availability, then the testing process can be performed, but the approach becomes prohibitively expensive and time-consuming when functionality, architecture, and dependencies change frequently
Solution Approach 1:
The system performs preliminary actions by automatically discovering and mapping service dependencies before fault injection testing. The dependency mapping service continuously monitors service interactions and builds an up-to-date dependency graph, so when fault injection is needed, the system already has the current service topology ready, eliminating the need to manually recreate fault models for each testing scenario.
Solution Approach 2:
The system creates a virtual copy of the service dependency topology through automated discovery and mapping. Instead of manually modeling each dependency relationship, the system observes actual service interactions and generates a replicated dependency map that can be used for fault injection testing, allowing rapid adaptation to architecture changes without manual intervention.
2Reliability
If manual fault models are created, then testing can be performed, but human error and lack of knowledge of dependencies result in important failures being missed
Solution Approach 1:
The system implements self-service by having services automatically publish their own dependency information and having the dependency mapping service autonomously discover and map service interactions. This eliminates human error in creating fault models, as the system automatically generates accurate dependency maps based on actual service behavior, ensuring all critical failure scenarios are captured.
Solution Approach 2:
The system uses feedback mechanisms where services publish their dependency metadata and the dependency mapping service continuously monitors and updates the dependency graph based on actual service interactions. This feedback loop ensures the fault models remain synchronized with the current system state, capturing emerging dependencies and preventing missed failure scenarios.
3Ease of manufacture
If independently created fault models for different components are used, then testing can proceed, but new dependencies between services are not detected and many failure scenarios are missed
Solution Approach 1:
The system implements a universal dependency mapping service that handles all service components across the entire distributed system. Instead of creating separate fault models for each component, this single service discovers and maps dependencies across all services uniformly, providing a comprehensive view that adapts to new dependencies between any services automatically.
Solution Approach 2:
The dependency mapping service is dynamic and continuously updates the dependency graph as services are added, removed, or modified. When new services are deployed or dependencies change, the system automatically discovers these changes and updates the fault models accordingly, ensuring the system always reflects the current service topology without manual intervention.
4Reliability
If brute force fault injection methods are used, then machine failure can be tested, but transient and degraded faults cannot be evaluated
Solution Approach 1:
The system applies local quality by injecting specific types of faults at specific locations in the dependency graph based on the test objectives. Instead of uniform brute force injection, the fault injection service selects targeted dependencies and applies appropriate fault types (transport layer faults, memory pressure, processor pressure) to specific services, enabling precise evaluation of transient and degraded fault scenarios.
Solution Approach 2:
The system changes fault parameters by injecting varying degrees and types of faults rather than simple binary failure states. The fault injection service can apply transient faults, degraded performance conditions, and resource pressure scenarios with different intensity levels, allowing comprehensive evaluation of system resilience across multiple fault severity levels.
Data Source
Figure 1
Figure 2~3
Figure 4~5
AI summary
Systems and methods disclosed herein are directed to creating a service directory of dependencies for services running on a system, wherein instances of a first service are dependent upon instances of a second service. The directory of dependencies comprises metadata associated with connections between the services. The system injects faults targeting all levels of the dependencies. The system is monitored to detect failures created by the faults. The injected faults are selected from transport layer faults, memory pressure, processor pressure, storage pressure, virtual machine restart, and virtual machine shut down. A domain name service is monitored to identify names that are resolved for the services. The service directory is then updated continuously with additional dependencies using information about the resolved names. The faults may be injected in a guided manner, wherein the scope of the faults is increased in steps over time to identify a failure point in the system.