Conflict Detection for Modular Intent Classifiers

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Existing cognitive question answering (QA) systems face instability and unpredictability when combining independently trained intent classifiers due to overlapping intent definitions, leading to conflicts and unpredictable outcomes, as there are no effective methods for evaluating and resolving these overlaps automatically.

Innovation Solution

A system and method for detecting and resolving conflicts between independently trained intent classifiers by using conflict detection and resolution engines, which evaluate overlapping intent definitions, provide alerts, and offer automated recommendations for revising training data or redefining intents to prevent unstable decisions.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Adaptability or versatility

If multiple independently trained intent classifiers are combined to handle different application domains, then the system's versatility and coverage are improved, but conflicts and instability arise due to overlapping intent definitions

Engineering Contradiction:
Improvedomain coverageVSAvoiddecision stability
Core Design Contradiction:
Adaptability or versatilityVSReliability

Solution Approach 1:

The patent applies preliminary action by performing conflict detection and resolution analysis before deploying multiple intent classifiers into production. The system evaluates potential overlaps between classifiers using test data and confusion matrices, identifying conflicting intent pairs in advance. This allows developers to resolve conflicts through retraining or intent redefinition before the classifiers are combined, preventing instability from occurring during actual operation rather than reacting to it afterward.

Inventive Principle:
Principle #10Preliminary action

2Measurement precision

If manual trial-and-error processes are used to evaluate and resolve classifier conflicts, then customization and precision are improved, but time consumption and operational complexity increase significantly

Engineering Contradiction:
Improveconflict detection accuracyVSAvoidevaluation time
Core Design Contradiction:
Measurement precisionVSLoss of time

Solution Approach 1:

The patent implements self-service by enabling the system to automatically evaluate intent classifier conflicts without requiring manual intervention. The conflict detection mechanism automatically feeds test data through multiple classifiers, collects their predictions, builds confusion matrices, and identifies conflicting intent pairs autonomously. Resolution recommendations are generated automatically based on the detected conflicts, eliminating the need for developers to manually test each classifier combination and significantly reducing the time required for conflict resolution while maintaining high precision through systematic evaluation.

Inventive Principle:
Principle #25Self-service

3Measurement precision

If confidence score calibration is improved to make scores more reliable, then the accuracy of individual classifier predictions is improved, but conflicts between classifiers with overlapping intents remain unstable and unpredictable

Engineering Contradiction:
Improveconfidence score reliabilityVSAvoidclassifier competition stability
Core Design Contradiction:
Measurement precisionVSReliability

Solution Approach 1:

The patent introduces an intermediary conflict detection and resolution layer between the individual intent classifiers and the final decision-making process. This intermediary mechanism uses confusion matrices to identify which intent pairs are likely to conflict when classifiers compete. By detecting these conflicts in advance and providing resolution recommendations (such as retraining classifiers or redefining intents), the system prevents unstable competition from occurring, allowing confidence scores to remain reliable indicators of individual classifier performance without being undermined by inter-classifier conflicts.

Inventive Principle:
Principle #24Intermediary (Mediator)

Data Source

PatentUS11455981B2Method, apparatus, and system for conflict detection and resolution for competing intent classifiers in modular conversation system
Publication Date: 2022.09.27 INTERNATIONAL BUSINESS MACHINE CORPORATION
  • US11455981B2 patent drawing
  • US11455981B2 patent drawing
  • US11455981B2 patent drawing

AI summary

A method, apparatus, and system are provided for resolving conflicts between training data conflicts by retrieving independent training data sets, each comprising a plurality of intents and end-user utterances for use in training one or more classifiers to recognize a corresponding intent from one or more of the end-user utterances, providing a first test end-user utterance associated with a first intent from the first independent training data set to the one or more classifiers to select an output intent generated by the one or more classifiers; identifying a first conflict when the first intent does not match the output intent, and automatically generating, by the system, one or more conflict resolution recommendations for display and selection by an end user to resolve the first conflict.