3D User Interface Representation for Gaze-Based Cross-Device Interaction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing computing systems are limited by the physical constraints of their displays or touch screens, restricting the ability to interact with user interfaces in a three-dimensional environment.
Innovation Solution
A first electronic device captures a real-world environment, including a second electronic device with a display, and presents a computer-generated environment with a virtual extended display that allows interaction with the second device's user interface, enabling operations based on user gaze and input detection.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a physical display or touch screen is used for user interaction, then the device can display visual information, but the interaction is limited by the physical constraints of the display
Solution Approach 1:
The patent transitions from two-dimensional physical display interaction to three-dimensional virtual environment interaction. Users interact with virtual representations of electronic devices and their displays within a spatially-aware augmented reality environment, adding the dimension of physical space to the interaction model and overcoming the limitations of flat touch screens.
Solution Approach 2:
The system introduces a virtual representation of the second electronic device as an intermediary between the user and the actual device. This virtual proxy allows users to interact with the device's display content through spatial gestures and gaze in the augmented reality environment, mediating the interaction and bypassing physical display constraints.
2Adaptability or versatility
If a virtual extended display is implemented to overcome physical constraints, then interaction capabilities are enhanced, but system complexity increases
Solution Approach 1:
The first electronic device performs multiple functions: it captures the real-world environment, generates virtual representations of other devices, renders the augmented reality scene, tracks user gaze and gestures, and processes spatial interactions. This multi-functionality consolidates the system complexity into a single device rather than requiring separate systems for each function.
Solution Approach 2:
The system automatically captures the environment using the device's camera, identifies and renders virtual representations of detected devices, tracks user gaze and gestures without requiring manual setup, and dynamically adjusts the virtual display based on user interaction. This self-service approach reduces operational complexity and makes the system easier to use despite its technical sophistication.
Data Source
AI summary
Some examples of the disclosure are directed to systems and methods for displaying and interacting with a representation of a user interface of an electronic device in a three-dimensional environment. While presenting a first user interface, a first electronic device presents a representation of a second user interface of a second electronic device based on display data representing the second user interface. In response to detecting a respective event corresponding to user input, if a gaze of a user of the first electronic device is directed to the representation of the second user interface, the first electronic device causes the second electronic device to perform a first operation directed to the second user interface based on the respective event. Alternatively, if the gaze is directed to the first user interface, the first electronic device performs a second operation directed to the first user interface based on the respective event.