LLM Continuous Conversation Check for Short and Noisy Utterances
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Large language model voice assistants face challenges in accurately detecting continuous conversations due to issues such as low similarity scores for short utterances, misidentification when users speak to others during microphone open periods, and poor performance in high background noise scenarios.
Innovation Solution
Implementing a speaker consistency check module for verification and acoustic consistency check module for audio energy and signal-to-noise ratio estimation, followed by a result fusion module to determine if subsequent user utterances are intended for the AI assistant.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If speaker verification is used to identify continuous conversation, then speaker identity can be verified, but short utterances generate low similarity scores leading to wrong predictions
Solution Approach 1:
The patent combines speaker verification with acoustic consistency checking (audio energy, SNR estimation) to create a multi-dimensional detection system. This merging compensates for the weakness of speaker verification alone on short utterances by adding acoustic features that remain reliable even when speech content is minimal.
Solution Approach 2:
The system changes from relying solely on speaker similarity scores to using multiple parameters including audio energy, signal-to-noise ratio, and spoken speed. This parameter diversification allows the system to accurately detect continuous conversation even when individual parameters like similarity scores are unreliable for short utterances.
2Adaptability or versatility
If the microphone remains open during response to capture follow-up utterances, then continuous conversation can be detected, but the user may speak to other people causing misidentification
Solution Approach 1:
The system uses acoustic consistency checks to provide feedback on whether the captured utterance is likely directed at the assistant. By analyzing audio energy patterns and SNR, the system can determine if the user is speaking towards the device, providing feedback that corrects misidentification errors.
Solution Approach 2:
The acoustic consistency check acts as an intermediary between the microphone input and the speech processing pipeline. It filters utterances by evaluating acoustic properties before they reach the speaker verification and NLP modules, preventing misidentified speech from being processed.
3Device complexity
If audio energy comparison is used to detect continuous conversation, then simple detection is possible, but performance deteriorates in high background noise scenarios
Solution Approach 1:
The system transitions from one-dimensional audio energy comparison to multi-dimensional acoustic analysis by adding signal-to-noise ratio estimation and spoken speed checking. These additional dimensions allow the system to distinguish between background noise and actual user speech, maintaining reliability in noisy environments.
Data Source
AI summary
In addition to speaker verification, an LLM continuous conversation check employs a spoken speed check using a phonemes-based spoken speed calculation, and acoustic energy check, and a signal-to-noise estimation to determine whether first and second audio inputs include utterances forming a continuous conversation by the user with the LLM. Results from the various components of the continuous conversation check are fused based on automatically assigned weights in making the determination. Continuous conversation detection for LLMs is therefore more robust, particularly for a very short second utterance. Optionally a distance to microphone check or a semantic consistency check may also be employed.