XR Virtual Assistant Interaction Using Context and Eye Tracking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Extended reality headsets face limitations in user interaction methods, such as hand controllers and hand gestures, which can cause fatigue and restrict use, necessitating a more intuitive interface for tasks and information presentation.
Innovation Solution
A virtual assistant analyzes contextual clues from the extended reality environment, integrating customized digital information through eye-tracking, user gestures, and environmental sensors to recommend and initiate actions, using graph-based interactions and machine learning to enhance user interaction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If hand controllers or hand gestures are used for interaction in extended reality, then the interface provides control capability, but user fatigue increases and usability is restricted
Solution Approach 1:
The patent replaces mechanical hand controllers and physical hand gestures with an eye-tracking based interaction system. The eye tracker detects user gaze direction and eye movements to control the extended reality interface, eliminating the need for physical hand movements and controller manipulation, thereby reducing user fatigue while maintaining control capability.
2Loss of information
If conventional virtual assistants process voice or text utterances, then they can answer questions and provide information, but they cannot intuitively understand contextual clues from the extended reality environment
Solution Approach 1:
The patent merges multiple data sources including eye-tracking data, environmental sensor data, and user profile information into a unified contextual understanding model. This integrated approach allows the virtual assistant to comprehensively analyze the extended reality environment and user state, improving contextual information understanding while managing system complexity through coordinated processing.
Solution Approach 2:
The patent introduces a contextual clues analysis module as an intermediary between the raw sensor data and the virtual assistant's decision-making process. This module processes and interprets eye-tracking and environmental data to extract meaningful contextual information, which then informs the virtual assistant's interactions, thereby improving contextual understanding without overwhelming the core assistant system.
3Ease of operation
If extended reality headsets provide hand controllers for interaction, then users can control applications, but the field of view is limited and arms must be extended which causes fatigue
Solution Approach 1:
The patent replaces the mechanical hand controller system with an eye-tracking based control mechanism. Users control applications through their natural eye movements and gaze direction, which are detected by the eye tracker and translated into interaction commands. This eliminates the need to extend arms and hold controllers, reducing fatigue while maintaining full application control capability.
Data Source
AI summary
Techniques are for analyzing contextual clues from an extended reality environment and, based on the analysis of the contextual clues, intuitively superimposing and integrating customized digital information into the artificial reality environment via a virtual assistant to recommend and lead the user into suggested action. A method includes obtaining input data from a user, generating a graph of objects, attributes, and relationships between objects extracted from the input data, determining one or more interactions to be presented, initiated, or executed based on the graph and a profile associated with the user, determining virtual content data to be used for rendering virtual content based on the one or more interactions, and rendering the virtual content in an extended reality environment displayed to the user based on the virtual content data. The virtual content is used to present, initiate, or execute the one or more interactions for the user.