Chatbot Regression Testing via Levenshtein Distance Analysis

Resolve Bottlenecks,
Find Innovative Solutions
Generate Solutions

Solution Overview

Problem

Manual regression testing for chatbots is time-consuming and often not done frequently due to resource constraints, leading to response degradation in live environments.

Innovation Solution

A method that uses a computing device to receive and standardize input test files containing data on chatbot interactions, generates predicted intents and responses using an AI-based chatbot, and calculates Levenshtein distances to determine similarity between expected and predicted results, facilitating automated regression testing.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If manual regression testing is performed on chatbots, then testing accuracy can be maintained, but time consumption and resource requirements increase significantly

Engineering Contradiction:
Improvetesting accuracyVSAvoidtime consumption
Core Design Contradiction:
ReliabilityVSLoss of time

Solution Approach 1:

The patent replaces manual mechanical testing processes with an automated system that uses Levenshtein distance calculations and machine learning models to perform regression testing. The system automatically compares predicted responses with expected responses using algorithmic string similarity metrics, eliminating the need for manual human review while maintaining or improving testing accuracy.

Inventive Principle:
Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent introduces an intermediary automated testing system that acts as a mediator between the chatbot system and the testing framework. This intermediary layer uses pre-trained language models and similarity calculation algorithms to bridge the gap between automated testing requirements and nuanced natural language evaluation, enabling accurate automated regression testing without direct human intervention.

Inventive Principle:
Principle #24Intermediary (Mediator)

2Reliability

If regression testing frequency is increased to maintain performance, then response quality improves, but resource commitment becomes unsustainable

Engineering Contradiction:
Improveresponse qualityVSAvoidtesting efficiency
Core Design Contradiction:
ReliabilityVSProductivity

Solution Approach 1:

The patent replaces resource-intensive manual regression testing with an automated computational system that can execute tests rapidly without human fatigue or additional resource requirements. The system uses efficient string similarity algorithms and pre-trained models that can process numerous test cases automatically, enabling frequent testing while maintaining or improving productivity.

Inventive Principle:
Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent changes the fundamental parameters of regression testing by introducing Levenshtein distance calculations and confidence threshold mechanisms. These parameter changes enable the system to automatically determine pass/fail criteria based on similarity metrics rather than requiring exhaustive manual verification, thereby increasing testing efficiency and enabling more frequent execution.

Inventive Principle:
Principle #35Parameter changes

3Measurement precision

If non-binary text matching is used for regression testing, then accuracy of results improves, but computational complexity increases

Engineering Contradiction:
Improvetext matching accuracyVSAvoidcomputational complexity
Core Design Contradiction:
Measurement precisionVSDevice complexity

Solution Approach 1:

The patent replaces simple binary text matching (exact match or no match) with Levenshtein distance-based similarity calculations. This substitution provides nuanced accuracy by measuring the minimum number of single-character edits needed to transform one string into another, enabling the system to identify semantically similar responses even with minor variations in wording or formatting.

Inventive Principle:
Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent applies partial action by introducing confidence thresholds that allow the system to accept responses with a certain degree of similarity without requiring perfect matches. This approach balances computational complexity with accuracy by setting appropriate thresholds that filter out clearly incorrect responses while accepting sufficiently similar ones, avoiding unnecessary computational overhead for marginal cases.

Inventive Principle:
Principle #16Partial or excessive action

Data Source

PatentUS20250193134A1Conversational Artificial Intelligence Regression Ensemble
Publication Date: 2025.06.12 ELEVANCE HEALTH INC
  • US20250193134A1 patent drawing
  • US20250193134A1 patent drawing
  • US20250193134A1 patent drawing

AI summary

A method of improving chatbot accuracy includes receiving an input test file containing chatbot interactions data including utterances, expected intents, and expected responses. For each interaction, the method may generate predicted intents and responses using an AI-based chatbot trained on previous interaction data. The method may determine similarity between predicted and expected responses/intents by calculating Levenshtein distances and comparing to predetermined thresholds. Failed interaction keywords may be extracted using natural language processing to identify failing topics. Machine learning decision tree classification may analyze patterns in failed interactions to generate resolution suggestions. The method may generate an interactive dashboard displaying historical performance trends, domain-specific accuracy metrics, visualizations of frequently keywords, and confidence distributions for failed interactions. This automated regression testing approach enables efficient identification and resolution of chatbot performance issues while maintaining response quality.