Chatbot Regression Testing via Levenshtein Distance Analysis
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Manual regression testing for chatbots is time-consuming and often not done frequently due to resource constraints, leading to response degradation in live environments.
Innovation Solution
A method that uses a computing device to receive and standardize input test files containing data on chatbot interactions, generates predicted intents and responses using an AI-based chatbot, and calculates Levenshtein distances to determine similarity between expected and predicted results, facilitating automated regression testing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If manual regression testing is performed on chatbots, then testing accuracy can be maintained, but time consumption and resource requirements increase significantly
Solution Approach 1:
The patent replaces manual mechanical testing processes with an automated system that uses Levenshtein distance calculations and machine learning models to perform regression testing. The system automatically compares predicted responses with expected responses using algorithmic string similarity metrics, eliminating the need for manual human review while maintaining or improving testing accuracy.
Solution Approach 2:
The patent introduces an intermediary automated testing system that acts as a mediator between the chatbot system and the testing framework. This intermediary layer uses pre-trained language models and similarity calculation algorithms to bridge the gap between automated testing requirements and nuanced natural language evaluation, enabling accurate automated regression testing without direct human intervention.
2Reliability
If regression testing frequency is increased to maintain performance, then response quality improves, but resource commitment becomes unsustainable
Solution Approach 1:
The patent replaces resource-intensive manual regression testing with an automated computational system that can execute tests rapidly without human fatigue or additional resource requirements. The system uses efficient string similarity algorithms and pre-trained models that can process numerous test cases automatically, enabling frequent testing while maintaining or improving productivity.
Solution Approach 2:
The patent changes the fundamental parameters of regression testing by introducing Levenshtein distance calculations and confidence threshold mechanisms. These parameter changes enable the system to automatically determine pass/fail criteria based on similarity metrics rather than requiring exhaustive manual verification, thereby increasing testing efficiency and enabling more frequent execution.
3Measurement precision
If non-binary text matching is used for regression testing, then accuracy of results improves, but computational complexity increases
Solution Approach 1:
The patent replaces simple binary text matching (exact match or no match) with Levenshtein distance-based similarity calculations. This substitution provides nuanced accuracy by measuring the minimum number of single-character edits needed to transform one string into another, enabling the system to identify semantically similar responses even with minor variations in wording or formatting.
Solution Approach 2:
The patent applies partial action by introducing confidence thresholds that allow the system to accept responses with a certain degree of similarity without requiring perfect matches. This approach balances computational complexity with accuracy by setting appropriate thresholds that filter out clearly incorrect responses while accepting sufficiently similar ones, avoiding unnecessary computational overhead for marginal cases.
Data Source
AI summary
A method of improving chatbot accuracy includes receiving an input test file containing chatbot interactions data including utterances, expected intents, and expected responses. For each interaction, the method may generate predicted intents and responses using an AI-based chatbot trained on previous interaction data. The method may determine similarity between predicted and expected responses/intents by calculating Levenshtein distances and comparing to predetermined thresholds. Failed interaction keywords may be extracted using natural language processing to identify failing topics. Machine learning decision tree classification may analyze patterns in failed interactions to generate resolution suggestions. The method may generate an interactive dashboard displaying historical performance trends, domain-specific accuracy metrics, visualizations of frequently keywords, and confidence distributions for failed interactions. This automated regression testing approach enables efficient identification and resolution of chatbot performance issues while maintaining response quality.


