Canary Cluster Chaos Experiments Using Minimum Production Traffic
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Chaos engineering experiments in production environments affect user experience and lack sufficient observability in testing environments, making it difficult to accurately assess disaster tolerance and system stability.
Innovation Solution
Conduct chaos engineering experiments in a canary cluster within a production environment by adjusting service request traffic to the minimum required for observability, using a canary cluster formed by servers for service upgrades, to minimize user impact and ensure accurate disaster tolerance assessment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If chaos engineering experiment is executed in production environment, then disaster tolerance capability can be realistically and accurately detected, but user experience is seriously affected
Solution Approach 1:
The system segments the service cluster into a canary cluster and a non-canary cluster. The canary cluster is specifically designated for chaos engineering experiments, while the non-canary cluster continues to serve users normally. This segmentation allows experiments to be conducted in isolation, achieving accurate disaster tolerance detection without affecting overall user experience.
Solution Approach 2:
The canary cluster acts as an intermediary between the production environment and the experiment. By routing a portion of service requests to the canary cluster, the system creates a buffer zone where experiments can be performed with controlled traffic, preventing direct impact on user experience while maintaining measurement validity.
2Object-affected harmful factors
If chaos engineering experiment is executed in testing environment, then user experience is not affected, but observability and coverage are insufficient
Solution Approach 1:
The canary cluster serves as an intermediary environment that bridges the gap between testing and production environments. It provides real production traffic and system conditions for high observability, while being isolated enough to protect user experience. The canary cluster receives a controlled portion of production traffic, enabling realistic experiment observation without full production exposure.
3Measurement precision
If service request traffic for canary cluster is increased to meet observability requirements, then experiment coverage is improved, but user experience impact increases
Solution Approach 1:
The system applies partial action by routing only a portion (e.g., 5%-20%) of production service requests to the canary cluster. This partial traffic allocation provides sufficient observability and experiment coverage while limiting the impact on user experience. The exact proportion is optimized to meet minimum experimental traffic requirements without excessive user impact.
Data Source
AI summary
A chaos engineering experimentation method for a system, a device and a storage medium. Minimum experimental traffic of a service request required by a chaos engineering experiment is determined as target service request traffic; service request traffic is adjusted for a canary cluster in a to-be-tested server cluster according to the target service request traffic, where the canary cluster is a cluster formed by servers in the to-be-tested server cluster, the servers being used for canary release when a service is upgraded; an experimental task of the chaos engineering experiment is executed in the canary cluster, and an experimental result is outputted. The chaos engineering experiment is conducted on the canary cluster, and the service request traffic of the canary cluster is adjusted to the minimum experimental traffic of the service request required by the chaos engineering experiment.