LLM Virtual Assistant for Hands-Free XR Interaction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Users in extended reality environments face limitations in interacting with applications, as traditional methods like hand controllers and gestures can cause fatigue and restrict use due to limited field of view.
Innovation Solution
A contextualized action recommendation virtual assistant system that utilizes a large language model to provide users with natural language recommendations tailored to their high-level goals, leveraging user context data from sensors and personal information to suggest actions that can be taken in the user's environment.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If hand controllers are used for interacting with extended reality applications, then users can control applications, but user fatigue increases and hands are occupied
Solution Approach 1:
The patent replaces mechanical hand controllers with a voice-activated virtual assistant system. Users can control extended reality applications through natural speech commands, eliminating the need for physical hand controllers and reducing hand fatigue while maintaining full interaction capability.
Solution Approach 2:
The virtual assistant acts as an intermediary between the user and the extended reality application. Instead of directly controlling applications with hands, users communicate goals to the virtual assistant, which then translates them into appropriate application actions, providing a hands-free interaction layer.
2Ease of operation
If hand gestures are used for interacting with extended reality applications, then users can control applications, but user fatigue increases and interaction is limited by field of view
Solution Approach 1:
The patent replaces mechanical hand gestures with acoustic voice commands. Users can control extended reality applications through speech, eliminating the need to extend arms and perform gestures, thereby reducing physical fatigue while overcoming field of view limitations.
Solution Approach 2:
The virtual assistant serves as an intermediary that translates user speech goals into application actions. This allows users to control applications without needing to be within the headset's field of view or perform physical gestures, expanding interaction accessibility.
3Adaptability or versatility
If traditional interaction methods are used in extended reality, then applications can be controlled, but the interface does not adapt to user goals and context
Solution Approach 1:
The virtual assistant system performs self-service by automatically understanding user goals, retrieving relevant context information from multiple sources, and generating appropriate application actions without requiring explicit user input for each step. The system serves itself by interpreting high-level goals and translating them into contextualized actions.
Solution Approach 2:
The system performs preliminary actions by proactively retrieving relevant context information about the user, their environment, and available applications before the user needs to interact. This pre-processing of information enables more intuitive and context-aware interactions when the user communicates their goals.
Data Source
AI summary
The present disclosure relates to using a large language model (LLM), provided with user context information and user high-level goal information to generate contextualized action recommendations that can help the user achieve the high-level goal(s). In one exemplary embodiment, a user system, and an AI action recommendation system that is associated with an LLM, are communicatively coupled and cooperatively implement a contextualized action recommendation virtual assistant. Input data comprising personal information data of the user that includes at least one high-level goal of the user, and user context data obtained from the user system can be collected and used to generate a prompt that is input to the LLM. The LLM can then generate a contextualized action recommendation for the user based on the prompt, and the contextualized action recommendation can be presented to the user via a virtual assistant user interface on a display of the user system.