Incident Duplicate Detection Using Summaries and Contextual Embeddings
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional methods for detecting duplicate incidents in issue tracking systems, such as software error tracking, are inefficient and resource-intensive due to reliance on static rules that fail to consider semantic context and are prone to the curse of dimensionality.
Innovation Solution
A hybrid deep learning approach using a transformer-based summarizer model to generate summaries and contextual embeddings, followed by a transformer-based transformation model to compute similarity scores, identifies duplicate incidents by comparing embeddings.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If machine learning models are used to generate summaries and embeddings for incident detection, then duplicate detection accuracy is improved, but computational resource requirements increase
Solution Approach 1:
The incident detection process is divided into multiple stages: initial filtering using lightweight rules, followed by selective application of machine learning models only to incidents that pass the initial filter. This segmentation allows the system to maintain high accuracy where needed while reducing overall computational resource consumption.
Solution Approach 2:
The system dynamically adjusts the complexity of analysis applied to different incidents based on their characteristics. By changing the parameter of analysis depth - applying simple rule-based filtering to obvious cases and reserving complex ML model analysis for ambiguous cases - the system optimizes the balance between detection accuracy and resource utilization.
2Reliability
If comprehensive incident analysis is performed using machine learning, then detection reliability is improved, but processing time increases
Solution Approach 1:
The system performs preliminary analysis using fast rule-based filtering and heuristic methods before applying comprehensive machine learning analysis. This preliminary action identifies and resolves obvious duplicate incidents quickly, reserving the more time-consuming ML analysis for cases where it is truly needed, thus improving overall processing efficiency while maintaining reliability.
Solution Approach 2:
The system applies partial analysis to most incidents (using lightweight filtering) and reserves excessive/comprehensive analysis (full ML processing) only for a subset of incidents that require higher confidence levels. This selective application of analysis depth maintains detection reliability for critical cases while reducing average processing time across all incidents.
Data Source
AI summary
Methods, apparatus, and processor-readable storage media for detecting duplicate incidents using machine learning techniques are provided herein. An example method includes obtaining information associated with tracking a first incident in an incident database, generating a summary of the first incident by processing at least a portion of the information using at least one first machine learning model, and generating an embedding of the first incident by processing the generated summary using at least one second machine learning model. The method also includes computing a set of similarity scores for the first incident, determining whether the first incident is a duplicate of at least one of a plurality of additional incidents in the incident database based on the set of similarity scores, and initiating an update to one or more data records in the incident database based at least in part on a result of the determining.