Full-Duplex Speech System Testing via Mixed Corpus Evaluation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speech interaction systems lack comprehensive testing methods and evaluation indicators, particularly for full-duplex systems, where independent module testing is insufficient to assess the system's overall performance.
Innovation Solution
A method and apparatus for testing full-duplex speech interaction systems involve creating a scene mixed corpus set by combining valid and invalid corpora, playing these corpora to the system, and analyzing the resulting logs to determine rejection and false response rates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If modules are tested independently in half-duplex systems, then individual module performance can be evaluated, but comprehensive system performance and interaction characteristics cannot be assessed
Solution Approach 1:
The testing method segments the evaluation into two distinct rates: rejection rate (for invalid requests) and false response rate (for valid requests). This segmentation allows precise measurement of different system characteristics while maintaining a unified testing framework, resolving the contradiction between comprehensive evaluation and method complexity.
Solution Approach 2:
The dual-rate evaluation system serves multiple functions: it assesses both the system's ability to reject invalid requests and its ability to respond to valid requests within time constraints. This multi-functional approach enables comprehensive system performance evaluation without requiring separate testing methodologies for different interaction modes.
2Productivity
If all speech requests are responded to in half-duplex systems, then interaction completeness is achieved, but resource waste occurs in full-duplex systems where not all requests are valid
Solution Approach 1:
The rejection rate metric provides feedback on the system's ability to identify and reject invalid requests. By monitoring this rate, the system can optimize its validation mechanisms to improve efficiency while minimizing unnecessary responses, thereby reducing energy consumption without compromising productivity.
Solution Approach 2:
The testing method introduces two distinct parameter thresholds: one for rejection rate and another for false response rate. By adjusting these parameters based on system requirements, the system can optimize the balance between responding to all requests versus conserving resources, enabling efficient operation in full-duplex environments.
3Reliability
If independent module testing is used, then testing simplicity is maintained, but system-level interaction characteristics and end-to-end performance cannot be measured
Solution Approach 1:
The method merges the evaluation of multiple system characteristics into a unified testing framework that measures both rejection behavior and false response behavior simultaneously. This combination approach enables system-level reliability assessment while maintaining operational simplicity through a single integrated testing process rather than multiple separate tests.
Data Source
AI summary
Disclosed are a method and apparatus for testing a full-duplex speech interaction system. The method includes: determining a scene mixed corpus set by mixing a valid corpus set related to a test scene with an invalid corpus set unrelated to the test scene; playing each corpus audio in the scene mixed corpus set to a speech interaction device under test equipped with the full-duplex speech interaction system; acquiring a work log of the speech interaction device under test, the work log including at least a first log and a second log; and obtaining number of false responses by counting number of log entries which have false response result in the second log, and determining a false response rate based on the number of false responses and a total number of corpus audios played. End-to-end testing of the full-duplex speech interaction system is realized.