Session-Level Fault Injection for Resilient Testing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Traditional chaos engineering methods in multi-technology environments face challenges such as user impact, operational complexity, and organizational barriers during resiliency testing, particularly in restricting blast radii and managing simultaneous tests, which can lead to system unavailability and increased vulnerabilities.
Innovation Solution
A system and method for resiliency testing at a session level, where faults are injected deterministically and surgically within a specific session, allowing for controlled impairment of services or endpoints with restricted blast radii, enabling testing without impacting other users and allowing for parallel testing without interference.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional chaos engineering methods are used to test resiliency, then system failures can be identified and tested, but user impact increases and availability decreases
Solution Approach 1:
The patent segments the user base into distinct cohorts (test group and control group) and applies faults only to specific segments rather than the entire system. This allows resiliency testing to be conducted on a subset of users while maintaining normal service availability for other users, thus reducing overall user impact while still achieving reliable testing results.
Solution Approach 2:
The patent implements local quality by applying fault injection at specific locations in the system architecture (e.g., service boundaries, data centers) rather than system-wide. This localized approach allows testing of specific components without degrading the quality of service for the entire user base, resolving the contradiction between testing effectiveness and user impact.
2Reliability
If faults are injected to test resiliency, then system response to failures can be evaluated, but system availability decreases during testing
Solution Approach 1:
The patent applies partial action by injecting faults only to the extent necessary for testing purposes, rather than causing complete system failures. Faults are injected at controlled levels (e.g., partial outages, degraded performance) that are sufficient to evaluate resiliency mechanisms while maintaining basic system availability and functionality during the testing period.
Solution Approach 2:
The patent implements preliminary action by establishing control groups and monitoring mechanisms before faults are injected. This allows the system to be prepared with baseline metrics and expected behavior definitions, enabling resiliency evaluation without causing unnecessary disruption to system availability during the testing process.
3Productivity
If multiple tests are run simultaneously, then testing productivity increases, but operational complexity increases
Solution Approach 1:
The patent segments the testing infrastructure into independent test orchestrators that can operate simultaneously without interference. Each test is assigned to a dedicated orchestrator that manages its own fault injection and monitoring, allowing multiple tests to run in parallel while keeping operational complexity manageable through clear separation of responsibilities.
Solution Approach 2:
The patent introduces test orchestrators as intermediary components that mediate between multiple simultaneous tests and the system under test. These orchestrators handle the complexity of coordinating fault injection, monitoring, and analysis for multiple tests, enabling increased testing productivity while isolating the complexity management function from the core testing process.
4Object-affected harmful factors
If blast radius is restricted to specific groups, then user impact is reduced, but test coverage is limited
Solution Approach 1:
The patent segments the user population into multiple cohorts that can be independently selected for testing. This allows the system to restrict blast radius to specific groups for any given test while maintaining the ability to cover different user segments across multiple tests, thus balancing user impact reduction with comprehensive test coverage through systematic cohort management.
Data Source
AI summary
Provided is a system and method for resiliency testing at a session level. A fault injection request is received from a client device associated with a user, via a user interface. The fault injection request defines an active impairment of one of a service or an endpoint with a blast radius restricted to a scope of the received fault injection request. The scope of the received fault injection request corresponds to a first session in a production environment or a lower environment. A client request is received from the client device within the blast radius of the active impairment. Based on the fault injection request and a persistence within the blast radius, the one of the service or the endpoint is impaired in the first session. A response, received from an entity associated with the one of the service or the endpoint, is transmitted to the client device.