Dynamic Diagnostics Graph for Cloud System Recovery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
As cloud systems become increasingly complex, diagnosing and recovering from incidents efficiently becomes a significant challenge due to the growing number of components and the need for rapid response as mandated by Service Level Agreements (SLAs).
Innovation Solution
A dynamic diagnostics graph is generated to visualize the hierarchy of diagnostic jobs, allowing for dynamic pruning and parallel execution across different nodes, coupled with a machine learning-based approach using a neural network trained from historical data to provide recommended actions for recovery.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional diagnostic methods are used on complex cloud systems, then comprehensive diagnosis can be achieved, but the diagnostic time and complexity increase significantly
Solution Approach 1:
The diagnostic process is segmented into a hierarchical graph structure with parent-child job relationships. The system divides comprehensive diagnosis into multiple independent diagnostic jobs that can be executed in parallel across different nodes, each job focusing on specific aspects of the cloud system. This segmentation maintains diagnostic completeness while reducing overall diagnostic time through parallel execution.
Solution Approach 2:
The diagnostic graph is dynamically adjusted based on job outcomes. When a parent job fails or is skipped, child jobs dependent on it are automatically skipped, dynamically pruning the diagnostic graph. This dynamic adaptation allows the system to maintain comprehensive diagnosis where needed while avoiding unnecessary diagnostic steps, thereby reducing diagnostic time without sacrificing completeness.
2Measurement precision
If all diagnostic jobs are executed to ensure complete system coverage, then diagnostic accuracy is maintained, but the time and resources consumed increase
Solution Approach 1:
The system executes only the necessary subset of diagnostic jobs based on dynamic pruning. When parent jobs fail or are skipped, child jobs are automatically skipped, avoiding excessive execution of unnecessary diagnostic steps. This partial action approach maintains diagnostic accuracy for critical paths while improving efficiency by eliminating redundant diagnostic operations.
Solution Approach 2:
The system performs preliminary diagnostic jobs that can determine the outcome of subsequent jobs. By executing parent jobs first and using their results to prune the diagnostic graph, the system avoids unnecessary child job executions. This preliminary action maintains diagnostic accuracy by ensuring critical diagnostics are performed while improving efficiency by preventing redundant work.
3Ease of operation
If diagnostic jobs are executed sequentially to maintain order and dependencies, then job dependency management is simplified, but the overall diagnostic process becomes slower
Solution Approach 1:
The diagnostic process is segmented into independent jobs with explicit dependency relationships defined in the graph structure. Jobs are organized in a hierarchical parent-child structure where dependencies are clearly defined. This segmentation allows the system to execute independent jobs in parallel across different nodes while maintaining proper dependency management through the graph structure, thereby improving diagnostic speed without complicating dependency management.
Solution Approach 2:
The diagnostic graph serves as an intermediary structure that manages job dependencies. The graph explicitly defines parent-child relationships and dependency rules, allowing the system to automatically determine execution order and parallelization opportunities. This intermediary structure simplifies dependency management by centralizing dependency logic in the graph while enabling parallel execution of independent job subsets.
Data Source
AI summary
Embodiments relate to diagnosis and recovery of cloud based systems. From an incident ticket, a dynamic diagnostics graph is generated visualizing a hierarchy (ancestor, child) of diagnostic jobs investigating the functioning cloud system. By indicating and checking job statuses, child jobs dependent on a skipped or failed job can be skipped according to a dynamic pruning technique—thereby trimming an entire branch. And, by running separate groups of diagnostic jobs in parallel across different nodes, the diagnostic process can be finished rapidly and efficiently. A diagnostic report includes the dynamic diagnostics graph. For system recovery, the dynamic diagnostic graph is analyzed to automatically provide one or more appropriate Recommended Actions (RAs) resolving cloud system problem(s) revealed by diagnostic efforts. Those appropriate RAs may be provided by performing machine learning (e.g., referencing a neural network) with a model trained from historical cloud diagnostic and recovery activity.