Multimodal Speech Endpoint Detection Using Face Cues
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing speech detection methods struggle with inaccurate detection of speech end points due to interference from background noise and pauses, leading to delayed or premature detection, which affects interaction efficiency and accuracy.
Innovation Solution
A multi-modal speech end point detection method that incorporates facial recognition to predict user intentions by analyzing face images, in addition to acoustic features, to accurately determine speech end points, reducing interference from noise and pauses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional audio-only speech detection method is used, then device complexity is low, but speech end point detection accuracy deteriorates due to interference from background noise and pauses
Solution Approach 1:
The patent combines audio signal analysis with face image analysis into a unified speech detection system. The audio processing unit processes acoustic features while the image processing unit processes visual features from face images, and both results are integrated to determine speech end points, thereby improving detection accuracy through multi-modal fusion
Solution Approach 2:
The patent transitions from single-modal audio detection to multi-modal detection by adding the visual dimension of face images. This dimensional expansion allows the system to capture user speaking states through both acoustic and visual channels, resolving the accuracy limitation of audio-only methods
2Loss of time
If trailing silence duration threshold is lowered to detect speech end points earlier, then detection delay is reduced, but premature detection increases due to user pauses during speech
Solution Approach 1:
The system continuously monitors both audio signals and face images in real-time, providing feedback to the speech end point determination unit. By analyzing the user's facial state during pauses, the system can distinguish between intentional pauses (where the face image shows the user still speaking) and actual speech endings, thereby avoiding premature detection while maintaining low detection delay
3Reliability
If trailing silence duration threshold is raised to avoid premature detection, then detection accuracy is improved, but detection delay increases causing missed speech end points
Solution Approach 1:
The system performs preliminary analysis of face images during the speech process to predict user speaking intentions. By examining facial expressions and mouth movements in advance, the system can anticipate whether a pause is temporary or marks the end of speech, enabling earlier and more accurate detection without increasing delay
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
This application provides a speech detection method, a prediction model training method, an apparatus, a device, and a medium, and belongs to the field of speech interaction technologies. This application provides a multi-modal speech end point detection method. Recognition is performed on a photographed face image by using a model, to predict whether a user intends to continue speaking, and determine whether a collected audio signal is a speech end point with reference to a prediction result. Because detection is performed not only based on an acoustic feature but also based on a visual-modal feature such as a face image, even if background noise is strong or the user pauses during speech, whether a speech signal is the speech end point can still be accurately determined by using the face image. Therefore, interference caused by background noise and a pause in speech is avoided, to avoid a problem, of delayed or premature detection of an ended state of speech interaction, resulting from interference caused by background noise and a pause in speech. This improves accuracy of speech end point detection.