Language Model Prompting from Predicted User Regions of Interest
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Digital humans lack the ability to predict user questions through observation, reducing engagement and accuracy in interactions due to reduced physical interactions and difficulty in assessing user interests remotely.
Innovation Solution
Utilizing computer vision and audio analysis to determine a predicted region of interest, generating prompts for a language model to improve user engagement and response accuracy by leveraging metadata and information retrieval systems.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If digital humans interact remotely without physical presence, then user accessibility is improved, but ability to predict user questions and assess user interests deteriorates
Solution Approach 1:
The patent introduces computer vision technology as an intermediary to capture and analyze user visual attention. Eye tracking devices and camera systems serve as mediators between the remote digital human and user, translating physical eye movements into digital signals that reveal user interests and predicted questions, thereby compensating for the loss of physical presence.
Solution Approach 2:
The patent replaces direct mechanical/physical interaction with optical and computational systems. Instead of relying on physical proximity to observe user behavior, the system uses optical tracking (cameras, eye trackers) combined with machine learning algorithms to detect and interpret user visual focus, substituting mechanical presence with sophisticated sensing and processing systems.
2Device complexity
If digital humans rely on traditional text-based interaction, then system complexity is reduced, but user engagement and response accuracy deteriorate
Solution Approach 1:
The patent applies preliminary action by continuously tracking and analyzing user visual attention in real-time before the user verbally expresses their interest. The system proactively processes eye movement data, identifies regions of interest, and prepares context-aware responses in advance, enabling more engaging and accurate interactions without significantly increasing system complexity.
Solution Approach 2:
The patent implements a feedback loop where the digital human system continuously monitors user eye movements, analyzes visual attention patterns, and adjusts its responses accordingly. This real-time feedback mechanism allows the system to adapt to user interests dynamically, improving engagement while maintaining manageable complexity through iterative refinement of the interaction model.
Data Source
AI summary
Techniques are provided for automatically generating language model prompts using predicted regions of interest. One method comprises obtaining a focus of a user interacting with an image and/or a virtual environment; determining a predicted region of interest of the user based on the focus, wherein the predicted region of interest has associated metadata indicating an item of interest, in the image and/or the virtual environment, to the user; providing a query to an information retrieval system that generates query results, wherein the query is based on the item of interest to the user obtained from the metadata; determining a prompt for a language model based on the query results; and providing a message, automatically generated by the language model based on the prompt, to the user. The predicted region of interest of the user may be obtained by tracking a movement of an eye of the user.