Visual Targeting Alignment for Time-Sensitive Voice Commands
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing systems face delays and inaccuracies in processing voice commands due to the time-based nature of human speech and rapid changes in visual targeting, leading to misapplication of commands to unintended objects.
Innovation Solution
Implementing proactive, reactive, and real-time targeting analysis techniques to identify visual targets before, during, or after spoken utterances, utilizing gaze and speech recognition to align audio and visual data streams for accurate command association.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If voice commands are processed sequentially after speech recognition, then speech recognition accuracy is maintained, but command association with visual targets becomes inaccurate due to time delays
Solution Approach 1:
The system performs preliminary action by capturing visual targeting data (gaze direction, head orientation) in advance and storing it in a buffer before the voice command is fully processed. This allows the visual target to be pre-identified and associated with the upcoming voice command, eliminating the time delay problem where the target might have changed by the time processing completes.
Solution Approach 2:
The patent introduces an intermediary mechanism - a temporal alignment system that uses timestamps and buffered data to bridge the gap between visual targeting detection and voice command processing. This intermediary layer synchronizes the two data streams (visual and audio) in time, ensuring accurate association even when processing occurs sequentially.
2Speed
If visual targeting data is captured in real-time, then responsiveness to user focus changes is improved, but processing latency increases due to continuous data collection and alignment
Solution Approach 1:
The system implements periodic action by capturing visual targeting data at discrete time intervals and buffering it, rather than continuously processing it in real-time. This periodic capture approach maintains responsiveness to focus changes while reducing processing latency by batch-processing the captured data when needed, rather than requiring immediate continuous alignment.
Solution Approach 2:
Visual targeting data is captured and buffered in advance (preliminary action) before the voice command processing needs it. This allows the system to prepare the data ahead of time, reducing the processing latency when the voice command is detected, while still maintaining real-time responsiveness through continuous buffering.
3Reliability
If the system waits for complete speech recognition before identifying visual targets, then speech processing is thorough, but commands may be misapplied to unintended objects due to rapid user focus changes
Solution Approach 1:
The system performs preliminary action by capturing and buffering visual targeting data before the voice command is fully processed. This allows the visual target to be identified and locked in advance, ensuring that even if speech processing takes time, the visual target remains associated with the correct object and not shifted due to rapid user focus changes during the processing delay.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method performed by a computing system for directing a voice command to a function associated with a visual target includes receiving a set of time-variable sensor-based data streams, including an audio data stream and a targeting data stream. The targeting data stream is stored in a buffer as buffered targeting data. Presence of a spoken utterance is identified within the audio data stream and is associated with a temporal identifier corresponding in time to the set of sensor-based data streams. A voice command corresponding to the spoken utterance is identified. A visual targeting vector within the buffered targeting data and a visual target of that visual targeting vector is identified at a time corresponding to the temporal identifier. The voice command is directed to a function associated with the visual target to generate an output.