Voice Input Classification Using Visual Audio Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Voice-control systems face challenges in determining when to stop listening to users, as they struggle to differentiate between user-directed inputs and deliberation among multiple people, leading to unnecessary actions during conversation periods.
Innovation Solution
The system classifies voice inputs as follow-on requests or deliberation based on visual and audio data, including head angle, number of people, and lip movement, to determine whether to provide responses, and can enter an active listen mode based on explicit instructions or user engagement, thereby avoiding responses during deliberation periods.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the system continuously monitors and responds to all voice inputs, then it ensures comprehensive user interaction, but it generates unnecessary responses during deliberation periods when users are not addressing the system
Solution Approach 1:
The system performs preliminary classification of voice inputs using audio and visual data before generating responses. By analyzing head angle, lip movement, and speaker identity in advance, the system determines whether a voice input is directed at the system or part of user deliberation, preventing unnecessary responses before they occur
Solution Approach 2:
The system uses visual feedback from cameras and audio feedback from microphones to continuously monitor user behavior. By analyzing head orientation, lip movement, and speaker characteristics, the system receives feedback about user engagement state and adjusts its response behavior accordingly, stopping responses during deliberation periods
2Measurement precision
If the system uses multiple sensors and analysis methods to detect user intent, then it improves classification accuracy, but it increases system complexity
Solution Approach 1:
The system segments the voice input detection task into distinct components: audio data analysis (speaker identification, voice characteristics) and visual data analysis (head angle, lip movement). Each component is processed separately by dedicated modules, making the complex system more manageable and maintainable while improving overall classification accuracy
Solution Approach 2:
The system merges audio and visual data streams to achieve more accurate classification. By combining information from microphones (audio) and cameras (visual) along with speaker identification, the system creates a comprehensive view of user intent that is more accurate than any single sensor could provide alone
Data Source
AI summary
Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for classifying voice inputs. The methods, systems, and apparatus include actions of providing an answer to a first voice input from a user and receiving visual or audio data corresponding to a second voice input. Further actions include classifying the second voice input as a follow on request to the first voice input or as deliberation on the answer, based on the visual data or the audio data. Additionally, the actions include determining whether to provide a response to the second voice input based on the classification of the second voice input.