AI Incident Postmortem Generation From Multi-Source Event Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing incident report generation processes are time-consuming, cumbersome, and prone to human error, relying heavily on manual documentation by site reliability engineers (SREs) and consuming significant CPU cycles and network resources.
Innovation Solution
Employing generative artificial intelligence (AI) to automatically generate incident postmortems by collecting data from various systems and using engineered prompts to create a draft postmortem, which can be reviewed and modified by responders, ensuring accuracy and completeness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual incident report generation is used, then the report can be customized and reviewed by SREs, but the process is time-consuming and prone to human error
Solution Approach 1:
The system automatically generates incident reports by having the incident response system itself compile and document the incident details, timeline, and resolution steps without requiring manual SRE documentation. The system serves itself by extracting data from its own operational logs and incident records.
Solution Approach 2:
The system continuously collects and structures incident data, timeline events, and resolution steps during the incident response process itself, so that when report generation is needed, all necessary information is already prepared and organized, eliminating the need for post-incident manual documentation.
2Loss of information
If manual documentation is used, then detailed incident information can be captured, but significant CPU cycles and network resources are consumed
Solution Approach 1:
The incident response system automatically documents and structures incident data using its own existing logging and monitoring infrastructure, eliminating the need for separate manual documentation processes that would consume additional computational resources.
Solution Approach 2:
The system combines incident data collection, timeline generation, and report compilation into a single automated process that leverages existing system resources, rather than using separate manual processes that would duplicate effort and consume additional CPU and network resources.
3Loss of information
If SREs manually aggregate notes and documents, then comprehensive incident coverage is achieved, but the process is cumbersome and error-prone
Solution Approach 1:
The system automatically aggregates and structures incident data from its own operational logs, monitoring systems, and incident records, eliminating the need for SREs to manually collect and organize information from multiple sources.
Solution Approach 2:
The system continuously pre-processes and structures incident data during the response process, organizing timeline events, affected services, and resolution steps in advance, so that comprehensive documentation is generated automatically without requiring SRE effort.
4Productivity
If automated report generation is implemented, then time and human effort are reduced, but the report may lack customization and require review
Solution Approach 1:
The system generates incident reports dynamically by querying its operational data and incident logs in real-time, allowing the report content to be automatically customized based on the specific incident details, affected services, and resolution steps without requiring manual intervention.
Data Source
AI summary
A computer-implemented method executed using one or more processors of an incident management system, the computer-implemented method comprising accessing one or more data sets of information associated with an incident event corresponding to an incident associated with a computer system; generating a prompt based on the one or more data sets of information, wherein generating the prompt comprises generating a plurality of sub-prompts to be provided to a machine-learning model for generating a report of the incident event in accordance with a predetermined criteria; inputting the prompt into a machine-learning model that has been trained to generate a report of the incident event based on the prompt; outputting, by the machine-learning model, the report of the incident event, wherein the report comprises an analysis of the incident event; transmitting the report to one or more computing devices associated with the computer system.