Gaze-Assisted Search Query for Automatic Object Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional visual search systems require manual user input to select a region of interest when multiple objects are present, leading to inefficiencies and cluttered image processing.
Innovation Solution
A gaze-assisted search system using a headset with an eye tracking system and world-facing camera assembly to automatically determine a user's gaze location, capture images, and format them based on a region of interest, generating a query that can be processed by a multi-modal large language model to provide answers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual user input is required to select region of interest, then object identification accuracy is improved, but user operation efficiency deteriorates
Solution Approach 1:
The system automatically determines the user's region of interest by tracking eye gaze direction and analyzing facial orientation, eliminating the need for manual input. The controller processes images from the camera assembly, detects the user's gaze direction, and automatically identifies the object in the gazed-at region, allowing the system to serve itself by inferring user intent without explicit commands
Solution Approach 2:
The patent replaces manual mechanical selection (touchscreen interaction, button pressing) with an optical-based eye tracking system. The camera assembly captures images of the user's face and eyes, and the controller analyzes these images to determine gaze direction, substituting the mechanical selection process with an optical and computational approach that is both faster and more intuitive
2Loss of information
If all objects in the image are processed, then comprehensive information is obtained, but computational resource consumption increases
Solution Approach 1:
The system extracts only the relevant portion of the image by identifying the region of interest based on user gaze direction. The controller processes only the specific region containing the gazed-at object rather than analyzing the entire image, thereby extracting necessary information while discarding unnecessary data from other regions
Solution Approach 2:
The patent applies different processing quality to different regions of the image. The region of interest (where the user is gazing) receives full processing attention for accurate object identification, while other regions are either processed at lower quality or not processed at all, optimizing computational resource allocation based on local importance
3Adaptability or versatility
If multiple objects are present in the image, then scene complexity increases, but object selection difficulty increases
Solution Approach 1:
The user's eye gaze acts as an intermediary that bridges the scene and the object of interest. Instead of directly analyzing multiple objects to determine which one the user wants, the system uses the gaze direction as an intermediate indicator to automatically identify the target object, simplifying the selection process in complex scenes
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Enables efficient and accurate identification of objects in complex scenes without manual user input, enhancing user experience and reducing computational resources by focusing on relevant image regions.
Implementation Method 1
The eye tracking system is configured to determine a gaze location of a user of the headset
Data Source
AI summary
A system enables a user to query based on a user's gaze by receiving a query from the user and capturing, via an eye tracking system on a headset, the user's gaze location near an object in a local area. The system captures one or more images of the local area with the object and formats the images based in part on a region of interest in the one or more images that includes the object. The system generates a formatted query based in part on the query. The formatted query is provided to a search engine. Information describing the object determined from the one or more formatted images and information describing the query are used by the search engine to determine an answer to the query about the object. The system presents the answer to the query about the object.