Isolated Speaker Signal Selection Using Audio-Video Text Matching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing digital humans struggle to engage users effectively due to reduced physical interactions, making it difficult to assess user interests and maintain dialogue engagement.
Innovation Solution
A method is employed to select an isolated speaker signal by comparing text from audio and video streams using speech-to-text and lip motion-to-text conversion models, iteratively adjusting steering vectors, and orienting the digital human towards the speaker based on coordinates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If digital humans interact with users through reduced physical interactions, then user engagement is reduced, but maintaining dialogue engagement becomes more difficult
Solution Approach 1:
The patent uses audio and video streams as intermediary signals to capture speaker information. By processing these streams through speech-to-text and lip motion-to-text conversion, the system creates a mediator mechanism that identifies speaker coordinates without requiring direct physical interaction, thus maintaining dialogue engagement while reducing physical contact
Solution Approach 2:
The patent replaces mechanical physical interaction with computational analysis. Instead of relying on physical presence or traditional audio processing, the system substitutes mechanical systems with computational models (speech-to-text conversion, lip motion analysis) that process audio and video signals to identify speaker locations and maintain engagement
2Measurement precision
If multiple audio signals are processed to identify speaker of interest, then speaker identification accuracy is improved, but processing complexity increases
Solution Approach 1:
The patent segments the audio processing task into distinct components: speech-to-text conversion for audio signals and lip motion-to-text conversion for video signals. By dividing the complex processing into separate specialized modules, the system improves speaker identification accuracy while managing processing complexity through functional segmentation
Solution Approach 2:
The patent adds a visual dimension to audio processing by incorporating lip motion analysis from video streams. This dimensional addition provides another source of speaker identification information, improving accuracy through multi-modal processing while distributing the computational load across different data types
Data Source
AI summary
Techniques are provided for selecting an isolated speaker signal by comparing text obtained from audio and video streams. One method comprises transforming audio signals from at least one speaker to first sets of predicted spoken words using a speech-to-text conversion model; transforming a video signal to a second set of predicted spoken words using a lip motion-to-text conversion model, wherein the second set of predicted spoken words is based on an analysis of an image associated with a respective speaker; iteratively adjusting a steering vector associated with the audio signals to compare the first sets of predicted spoken words with the second set of predicted spoken words; and selecting an isolated speaker audio signal associated with a particular one of the first sets of predicted spoken words and the second set of predicted spoken words, wherein the selection is based on a result of the comparison.