Chaos Testing Prioritization via Organizational Distance in Microservices

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

In microservices architecture, the complexity of tracking and monitoring hundreds of microservices developed by different teams leads to increased challenges in cooperation and miscommunication, particularly due to large organizational distances between workers, making it difficult to diagnose and repair failures efficiently, which affects the system's availability and reliability.

Innovation Solution

Implementing chaos testing with a service mesh and organizational chart analysis to prioritize failure introduction based on organizational distance, using automated detection of code ownership and API dependencies to identify fragile links and reduce mean time to repair.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of manufacture

If microservices are decomposed into smaller, independently developed services to improve modularity and development efficiency, then the application becomes easier to understand, develop, and test, but the complexity of tracking and monitoring hundreds of microservices increases significantly

Engineering Contradiction:

Improveease of developmentVSAvoidsystem complexity

Core Design Contradiction:

Ease of manufactureVSDevice complexity

Solution Approach 1:

The patent introduces an intermediary system that automatically detects code ownership and API dependencies between microservices. This intermediary layer analyzes git repository data, API specifications, and service meshes to build an organizational model that maps code ownership to organizational structure, thereby reducing the complexity of tracking microservice relationships without sacrificing development independence.

Inventive Principle:

Principle #24Intermediary (Mediator)

2Productivity

If multiple autonomous teams develop and deploy microservices independently to parallelize development, then development speed increases, but communication and cooperation difficulties arise due to large organizational distances

Engineering Contradiction:

Improvedevelopment speedVSAvoidcooperation ease

Core Design Contradiction:

ProductivityVSEase of operation

Solution Approach 1:

The patent adds a new dimension to microservice analysis by incorporating organizational chart data and code ownership information. Instead of only analyzing technical dependencies, the system creates a multi-dimensional model that includes organizational distance metrics, enabling teams to understand both technical and organizational relationships between services.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Solution Approach 2:

The system provides feedback by automatically detecting and reporting code ownership relationships and API dependencies between microservices. This feedback mechanism helps identify fragile links caused by large organizational distances, enabling teams to improve communication and cooperation by understanding the organizational context of their service interactions.

Inventive Principle:

Principle #23Feedback

3Reliability

If chaos testing is applied to all microservice connections to ensure system reliability, then system reliability improves, but the time and resources required for testing increase significantly

Engineering Contradiction:

Improvesystem reliabilityVSAvoidtesting time

Core Design Contradiction:

ReliabilityVSLoss of time

Solution Approach 1:

The patent applies local quality by prioritizing chaos testing based on identified fragile links. Instead of uniformly testing all microservice connections, the system focuses testing resources on specific connections that have large organizational distances and high dependency criticality. This targeted approach maintains system reliability while significantly reducing the time and resources required for comprehensive testing.

Inventive Principle:

Principle #3Local quality

4Reliability

If comprehensive monitoring and tracking of all microservices is implemented to improve system reliability, then system reliability improves, but the complexity of diagnosing and repairing failures increases

Engineering Contradiction:

Improvesystem reliabilityVSAvoidfailure diagnosis ease

Core Design Contradiction:

ReliabilityVSEase of repair

Solution Approach 1:

The patent performs preliminary action by automatically building an organizational model that maps code ownership, API dependencies, and service relationships before failures occur. This pre-computed model includes fragility scores and organizational distance metrics that are ready to guide failure diagnosis and repair efforts, reducing the complexity of troubleshooting when failures do occur.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS11765043B2Data driven chaos engineering based on service mesh and organizational chart

Publication Date: 2023.09.19 DELL PROD LP

AI summary

An organizational graph and a microservice graph can be generated. For two neighboring microservices, a combined organizational distance can be determined based on a distance for respective workers of each microservice in the organizational graph. The combined organizational distance can be used to determine a priority for a connection between the two microservices, and chaos testing can be performed on that connection based on its priority.