Digital Assistant Action Validation for Conversational Flow Testing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Complex digital assistant-based applications face challenges in identifying and rectifying errors or malfunctions, particularly in executing actions triggered by unstructured voice-based inputs, due to the vast number of possible conversational flows and interactions, making it difficult to efficiently and reliably test all scenarios.
Innovation Solution
A data processing system simulates device operation, generates voice-based inputs, and automates user interactions to test applications, identifying errors and modifying them accordingly, using a scalable platform that includes a Monte Carlo simulation, historical user conversations, and a classifier to validate actions before launch.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If comprehensive testing of all conversational flows is performed, then application reliability is improved, but testing time and complexity increase significantly
Solution Approach 1:
The system performs preliminary testing of conversational flows and actions before application launch. Test results, including identified errors and validation outcomes, are obtained in advance through automated simulation of user interactions, allowing issues to be resolved before the application goes live.
Solution Approach 2:
The system creates virtual copies of user interactions by generating simulated voice-based inputs and conversational flows. These synthetic test cases replicate real user scenarios without requiring actual user participation, enabling comprehensive testing while saving time.
2Reliability
If comprehensive testing of all conversational flows is performed, then application reliability is improved, but device complexity increases
Solution Approach 1:
The testing system is self-configuring and automatically generates test cases without requiring manual setup. The system autonomously obtains application information, identifies conversational flows, generates simulated user inputs, and validates actions, reducing the complexity burden on external testing infrastructure.
Solution Approach 2:
The testing system performs multiple functions through a unified platform: it parses application manifests, generates synthetic voice inputs, simulates conversational flows, validates actions, and produces test reports. This multi-functional approach consolidates complexity into a single versatile system rather than requiring separate specialized tools for each testing task.
3Measurement precision
If manual testing of actions is performed, then testing precision is improved, but productivity decreases
Solution Approach 1:
The system replaces manual mechanical testing processes with automated computational methods. Synthetic voice-based inputs are programmatically generated and processed through the application's conversational interface, with automated validation of actions replacing manual observation and reporting, thereby maintaining precision while dramatically increasing productivity.
4Reliability
If testing covers all possible conversational flows, then coverage is improved, but ease of operation worsens
Solution Approach 1:
The testing system automatically obtains application information from manifests, independently identifies conversational flows and actions, and generates appropriate test cases without requiring operator intervention. This self-service capability maintains comprehensive coverage while preserving ease of operation.
Solution Approach 2:
The system generates a large number of synthetic test variations that exceed minimum requirements, testing conversational flows with varied inputs and conditions. This excessive action approach ensures comprehensive coverage while the automation maintains ease of operation by handling the volume of test cases without operator burden.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Validating actions in a digital assistant-based application is provided. The system identifies an application with a conversational interface. The system selects an action from an action repository and generates, via a natural language processor, a trigger phrase for input into the application. The system executes the application to process the trigger phrase to identify an action of the application. The system identifies a parameter used by the application to execute the action, and generates, based on the parameter and via execution of the conversational interface of the application, a first query responsive to the trigger phrase. The system generates a first response to the first query for input into the application. The system determines, based on execution of the application to process the first response, a state of the application. The system evaluates the state to determine an error code and provide a notification based on the error code.