Voice Assistant Active Listening With Non-Visual Feedback
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing assistant systems face challenges in increasing operating capacity while maintaining active listening capabilities, especially when the client system is locked, and providing non-interruptive feedback to users to avoid confusion about issuing voice commands.
Innovation Solution
The solution involves providing continuous non-visual feedback through subtle and non-interruptive background sounds, allowing the assistant system to access increased computing capacity without unlocking the client system, and using a hybrid architecture that leverages both client-side and server-side processes for efficient resource utilization and privacy protection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional NLP-based approaches are used for sentiment analysis, then text processing capability is provided, but contextual understanding and emotional nuance detection are insufficient
Solution Approach 1:
The system segments audio processing and text processing into separate specialized modules, with audio data processed by audio encoders and text data processed by text encoders, then combines their outputs. This segmentation allows each module to be optimized for its specific modality while maintaining overall system functionality.
Solution Approach 2:
The patent introduces audio embeddings as an intermediary representation that captures emotional context from audio signals. These embeddings serve as a bridge between the audio input and the text analysis components, enabling the system to incorporate non-verbal emotional cues into the sentiment analysis process.
2Loss of information
If audio and text data are processed separately, then processing efficiency is maintained, but emotional context integration is lost
Solution Approach 1:
The system merges audio embeddings and text embeddings into a unified combined embedding representation. This combination occurs at the feature level, allowing the model to leverage both audio-based emotional cues and text-based semantic information simultaneously for more accurate sentiment and emotion detection.
Solution Approach 2:
The dual-encoder architecture is designed to handle multiple types of input data (audio and text) through a unified processing framework. The same transformer-based architecture processes both modalities, demonstrating multi-functionality while maintaining processing efficiency through parallel operation of the separate encoders.
3Measurement precision
If comprehensive emotional analysis is implemented, then detection accuracy is improved, but computational resources and processing time increase
Solution Approach 1:
The system performs preliminary processing by extracting audio embeddings and text embeddings separately through specialized encoders before combining them. This preliminary separation allows for optimized processing of each modality independently, reducing overall computation time while maintaining accuracy.
Solution Approach 2:
The patent utilizes parameter changes in the embedding space to represent different emotional states. By transforming raw audio and text inputs into standardized embedding representations with specific dimensionalities and normalizations, the system enables efficient comparison and analysis across different input types while maintaining detection accuracy.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
In one embodiment, a method includes receiving a first user input comprising a wake word associated with an assistant xbot from a first client system, setting the assistant xbot into a listening mode, wherein a continuous non-visual feedback is provided via the first client system while the assistant xbot is in the listening mode, receiving a second user input comprising a user utterance from the first client system while the assistant xbot is in the listening mode, determining the second user input has ended based on a completion of the user utterance, and setting the assistant xbot into an inactive mode, wherein the non-visual feedback is discontinued via the first client system while the assistant xbot is in the inactive mode.