Gaze-Guided Digital Assistant for Moving Graphical Elements
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing digital assistants require explicit specification of graphical elements and target locations for moving or copying, which is inefficient and increases user input requirements, thereby affecting device operability and power consumption.
Innovation Solution
A digital assistant interprets natural language inputs including deictic references and user gaze to determine the intended graphical element and target location, allowing for efficient movement or copying without additional user input.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If explicit specification of graphical elements and target locations is required, then accuracy of operation is improved, but user input requirements increase and device operability deteriorates
Solution Approach 1:
The patent introduces gaze detection as an intermediary mechanism between the user and the digital assistant. The gaze detection system mediates the interaction by automatically identifying which graphical element the user intends to manipulate, eliminating the need for explicit specification while maintaining accurate element identification. This intermediary layer resolves the contradiction by providing implicit selection through eye tracking technology.
Solution Approach 2:
The patent replaces traditional mechanical input methods (such as clicking, typing, or pointing) with a gaze-based selection system. Instead of requiring physical interaction to specify graphical elements, the system uses optical detection of user gaze to automatically identify the intended element. This substitution reduces physical input requirements while maintaining operational precision.
2Measurement precision
If explicit specification of graphical elements and target locations is required, then accuracy of operation is improved, but number of user inputs increases and efficiency deteriorates
Solution Approach 1:
The system performs self-service by automatically identifying graphical elements and target locations through gaze detection and natural language processing. Instead of requiring the user to manually specify each parameter, the system autonomously determines the intended element and location based on the user's gaze direction and spoken commands. This self-service capability maintains accuracy while dramatically reducing the number of user inputs required.
Solution Approach 2:
The system performs preliminary action by pre-identifying potential graphical elements based on user gaze before the actual manipulation command is given. The gaze detection system continuously monitors and pre-selects elements in the user's line of sight, so when the user speaks a command like 'move this here,' the system already has the target element identified, enabling immediate execution without additional specification steps.
3Ease of operation
If natural language processing with deictic references is implemented, then ease of operation is improved, but complexity of system increases
Solution Approach 1:
The patent merges multiple subsystems into an integrated natural language processing framework. The gaze detection system, deictic reference interpretation, and graphical element identification are combined into a unified processing pipeline. This merging allows the system to handle complex natural language commands with deictic references like 'this,' 'that,' or 'it' while maintaining ease of operation, as the combined system automatically correlates gaze data with linguistic references to identify intended elements.
4Ease of operation
If gaze detection is used to identify deictic references, then user input requirements are reduced, but measurement precision requirements increase
Solution Approach 1:
The system implements feedback mechanisms to verify and refine gaze detection results. The natural language processing component provides contextual feedback by analyzing the user's spoken words and comparing them with detected gaze direction. This feedback loop allows the system to compensate for minor imprecisions in gaze detection by using linguistic context to confirm or adjust the identified graphical element, thereby maintaining high accuracy while keeping the interface simple.
Data Source
AI summary
An example process includes while displaying, on a display, a graphical element having a first display location, receiving a natural language input including a deictic reference to the graphical element; determining whether the natural language input corresponds to a user intent to move or copy the graphical element; in accordance with a determination that the natural language input corresponds to the user intent to move or copy the graphical element: determining, based on detected user gaze input, that the deictic reference refers to the graphical element; and in accordance with determining that the deictic reference refers to the graphical element, displaying the graphical element at a second display location different from the first display location.