Incident Investigation Workflow for Automated Root Cause Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing incident investigation systems are manually implemented and domain-based, lacking awareness of application and platform layers, leading to inaccurate responses and inefficient incident resolution.
Innovation Solution
An automated incident investigation system that performs end-to-end analysis across application, managed infrastructure, and platform layers, utilizing anomaly detection, troubleshooting, and large language models to provide comprehensive incident summaries and root cause analysis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual incident investigation processes are used, then system complexity is reduced, but investigation accuracy and speed deteriorate
Solution Approach 1:
The incident investigation process is segmented into distinct phases: data collection from multiple sources (logs, metrics, traces), anomaly detection, root cause analysis, and impact assessment. Each phase is handled by specialized modules that work independently but integrate results, making the complex overall process manageable and accurate.
Solution Approach 2:
An intermediary system layer is introduced between incident detection and resolution, which automatically collects data, identifies anomalies, determines root causes, and generates recommendations. This intermediary layer handles the complexity of multi-source data integration and analysis, freeing operators from manual investigation while maintaining high accuracy.
2Productivity
If automated multi-layer analysis is implemented, then incident resolution speed is improved, but system complexity increases
Solution Approach 1:
The system implements a universal incident investigation platform that handles multiple types of incidents across different layers (application, infrastructure, platform) using the same core capabilities. The anomaly detection and root cause analysis mechanisms work across diverse data types, eliminating the need for separate specialized tools for each layer and improving resolution speed without proportionally increasing complexity.
Solution Approach 2:
The system performs preliminary data collection and anomaly detection automatically when an incident is detected, before operators need to intervene. By pre-processing data from multiple sources and identifying potential root causes in advance, the system reduces the time operators need to spend on investigation and accelerates the overall resolution speed.
3Loss of information
If domain-based manual investigation is used, then ease of operation is maintained, but information accuracy deteriorates
Solution Approach 1:
The system continuously collects feedback from multiple data sources (logs, metrics, traces, anomalies) and uses this feedback to refine the root cause analysis and generate more accurate recommendations. The feedback loop ensures that information accuracy is maintained by constantly validating findings against actual system behavior, while the automated nature of feedback collection keeps operations simple.
Solution Approach 2:
The system merges data from multiple previously separate domains (application layer, infrastructure layer, platform layer) into a unified incident view. By combining logs, metrics, traces, and anomaly data into a single coordinated analysis, the system eliminates information silos and provides comprehensive accurate information without requiring operators to manually query each domain separately.
Data Source
AI summary
Systems and methods are provided for automated incident investigation. Anomaly detection is used to identify anomalies in incident data (e.g., alerts, changes, metrics, logs, and/or system health), and the identified anomalies are converted into facts (or textual prompt inputs for a large language model (“LLM”)). A troubleshooting or diagnostic system is run on the anomalies to provide additional facts to identify a root cause of an incident. The facts from the diagnostics, the facts from the anomaly detections are entered into a consolidated explainer that generates a summary of what happened, what is a likely cause, and what to do next to resolve the issue. In examples, anomaly enrichment data including a time correlation result, a weighted list of abnormal transaction patterns, a list of abnormal trace patterns, a list of exception patterns, a difference pattern, and/or region data are input as further facts to enhance the incident investigation process.