Head-Wearable Object Targeting Using Gaze-Guided AI Assistance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing eye tracking technologies in devices lacking displays have limited applications for AI-assisted interactions, necessitating improved methods for user interaction and context-based command execution.
Innovation Solution
A head-wearable device equipped with sensors, imaging devices, and AI programs that utilize eye-tracking and head-orientation data to initiate and perform context-based commands, enabling interaction without voice or hand gestures, by capturing and processing gaze and field-of-view data to identify objects of interest.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If eye tracking and AI processing are implemented in displayless head-wearable devices, then interaction efficiency and accessibility are improved, but device complexity and power consumption increase
Solution Approach 1:
The system segments the interaction process into distinct phases: eye tracking for object identification, AI processing for command interpretation, and selective FOV capture for detailed analysis. This segmentation allows each component to operate independently and efficiently, reducing overall system complexity while maintaining high interaction efficiency.
Solution Approach 2:
The system performs preliminary actions by continuously monitoring eye tracking data and pre-processing FOV images before AI command execution. This allows the system to be ready for immediate response when a user provides a command, reducing interaction latency and improving ease of operation without requiring complex real-time processing.
2Measurement precision
If continuous FOV image capture is performed, then object identification accuracy is improved, but power consumption increases
Solution Approach 1:
Instead of continuous capture, the system uses periodic action by capturing FOV images only when eye tracking detects significant gaze changes or when AI processing is triggered. This maintains high object identification accuracy for relevant objects while dramatically reducing power consumption by avoiding unnecessary continuous capture.
Solution Approach 2:
The system applies local quality by selectively capturing and processing only the portion of the FOV where the user's gaze is directed, rather than processing the entire field of view. This targeted approach maintains high identification accuracy for objects of interest while reducing overall processing power requirements.
3Speed
If AI processing is triggered by any data capture, then responsiveness is improved, but false triggers increase
Solution Approach 1:
The system implements feedback mechanisms by continuously monitoring eye tracking data and using this feedback to modulate the sensitivity of AI trigger detection. When eye tracking indicates stable gaze on an object, the system becomes more responsive to commands. When gaze is unstable or on non-relevant objects, the system reduces trigger sensitivity, thereby reducing false triggers while maintaining high responsiveness when needed.
Data Source
AI summary
System and method for using an artificial intelligence (AI) system of a head-wearable device to process image data using eye-tracking data are disclosed. An example method includes, in accordance with an indication that first data captured by the head-wearable device satisfies an AI assistant trigger condition, initiating the AI assistant and capturing, by the head-wearable device, second data and field-of-view (FOV) image data. The example method includes determining, by the AI assistant, a user query and contextual information based on the second data and the FOV image data. The example method includes detecting, based on the user query and the contextual information, a portion of the FOV image data including an object of interest; and performing a context-based command on the portion of the FOV image data including the object of interest. The context-based command is based on one or more of the contextual information and the user query.