LLM Postmortem Generation With Incident Ticket Verification

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing postmortem generation techniques, such as PTL 1, lack sufficient information for incident response processes and may generate invalid postmortems by correlating irrelevant incidents, leading to unclear answers in the postmortem analysis.

Innovation Solution

A generation support system that utilizes a language model to probabilistically predict answer sentences for check items in a postmortem, ensuring clear indication of the presence of answers through a series of prompts and ticket searches to generate a comprehensive postmortem.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If incident data is aggregated by keyword matching to generate postmortem automatically, then postmortem generation efficiency is improved, but the accuracy and reliability of the generated postmortem deteriorates due to insufficient information and potential correlation of irrelevant incidents

Engineering Contradiction:

Improvepostmortem generation efficiencyVSAvoidpostmortem accuracy

Core Design Contradiction:

ProductivityVSReliability

Solution Approach 1:

The patent segments the postmortem generation process into multiple distinct stages: incident collection, correlation analysis, answer sentence generation, and completeness verification. Each stage handles specific tasks independently, allowing for more precise control over the generation process and reducing the risk of generating inaccurate content while maintaining efficiency.

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The patent implements feedback mechanisms where the system verifies whether generated answer sentences meet the requirements of check items, and iteratively refines the generation process. The verification step provides feedback on the quality of generated content, enabling continuous improvement of postmortem accuracy without sacrificing generation speed.

Inventive Principle:

Principle #23Feedback

2Extent of automation

If keyword matching is used to correlate incidents, then automated incident correlation is achieved, but the precision of correlation deteriorates due to potential matching of irrelevant incidents

Engineering Contradiction:

Improveautomated incident correlationVSAvoidcorrelation accuracy

Core Design Contradiction:

Extent of automationVSMeasurement precision

Solution Approach 1:

The patent introduces an intermediary verification mechanism that acts as a mediator between automated keyword matching and final incident correlation. This intermediary layer checks whether correlated incidents truly satisfy the correlation requirements by verifying answer sentence generation, thereby improving correlation precision while maintaining automation.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent replaces simple mechanical keyword matching with a more sophisticated system that uses language models to generate and verify answer sentences. This substitution transforms the correlation process from a purely mechanical operation to an intelligent process that can distinguish between relevant and irrelevant incidents, significantly improving correlation accuracy.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Loss of information

If comprehensive check items are required for postmortem, then postmortem completeness is improved, but the complexity of verification process increases

Engineering Contradiction:

Improvepostmortem completenessVSAvoidverification process complexity

Core Design Contradiction:

Loss of informationVSDevice complexity

Solution Approach 1:

The patent creates a universal verification framework that can handle multiple check items through a single integrated process. The language model performs multiple functions simultaneously: generating answer sentences, verifying their completeness against check items, and ensuring consistency across different incident data. This multi-functionality reduces verification complexity while maintaining comprehensive coverage.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Data Source

PatentUS20250363298A1Generation support system, generation support method, and generation support program

Publication Date: 2025.11.27 HITACHI LTD

AI summary

When a natural sentence that is text data of a natural language is inputted, the generation support system can access a language model that interprets the natural sentence and that probabilistically predicts an answer sentence to the natural sentence. The processor outputs a generation-source ticket of a natural sentence related to an incident, a check item group to be checked for the incident, and a first prompt that is the natural sentence requesting that an answer sentence about each check item of the check item group be generated with reference to the generation-source ticket, to the language model, causes the language model to acquire a postmortem including a result of a first trial of generation of an answer sentence about each check item in the first prompt, as a result of output of the generation-source ticket, the check item group, and the first prompt, and outputs the postmortem.