Virtual Remote Control in Mixed Reality via Computer Vision
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing virtual reality, augmented reality, and mixed reality systems face challenges in providing a comfortable and natural-feeling experience due to complexities in human visual perception and the impracticality of emulating physical remote controls in augmented reality/mixed reality environments.
Innovation Solution
A wearable system that automatically recognizes physical remotes or parent devices using computer vision techniques, generates a virtual remote with a control panel viewable and interactable by the user, and detects user interactions with virtual elements by tracking body poses and eye gaze.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If physical remote controls are emulated in augmented reality/mixed reality environments, then user interaction with household appliances is enabled, but the user experience becomes uncomfortable and unnatural due to visual perception complexities
Solution Approach 1:
The patent creates a virtual copy of the physical remote control interface that appears on a distant surface (such as a wall or table). This virtual remote control replicates the functionality of physical remotes while eliminating the need for users to hold or manipulate physical devices. The virtual interface is rendered using computer vision and augmented reality techniques, allowing users to interact with household appliances through natural gestures and eye gaze without the discomfort of holding physical controls in AR/MR environments.
Solution Approach 2:
The patent replaces the mechanical interaction of holding and pressing physical remote controls with virtual interaction mechanisms. Users control appliances through gesture recognition, eye tracking, and virtual button activation displayed on distant surfaces. This substitution eliminates the mechanical burden of physical device handling while maintaining full remote control functionality, thereby improving ease of operation in AR/MR contexts.
2Ease of operation
If virtual user input controls are implemented, then physical remote controls are eliminated, but detection accuracy of user interactions becomes challenging
Solution Approach 1:
The patent employs a multi-functional detection system that combines multiple sensing modalities: computer vision for gesture recognition, eye tracking for gaze detection, and virtual interface rendering. This universal approach allows the system to detect various types of user interactions (hand gestures, eye gaze, virtual button presses) through a single integrated framework, thereby maintaining high detection accuracy while eliminating physical remote controls.
Solution Approach 2:
The patent implements real-time feedback mechanisms where the system continuously monitors user gestures and eye gaze, adjusts the virtual remote control interface accordingly, and provides visual confirmation of interactions. This feedback loop ensures accurate detection and response to user inputs, maintaining reliability in the absence of physical controls. The system adapts to user behavior patterns and refines its detection accuracy through ongoing interaction.
3Ease of operation
If virtual remote control interface is rendered on distant surfaces, then natural interaction is improved, but the field of view required to see and interact with the virtual interface increases
Solution Approach 1:
The patent makes the virtual remote control interface dynamic by allowing it to move and resize based on user interaction. The interface can be positioned on distant surfaces to provide natural interaction, but it can also dynamically adjust its position, size, and visibility to accommodate the user's field of view. This dynamic adaptation ensures that the virtual interface remains accessible and viewable without requiring the user to maintain a fixed, narrow field of view, thereby preserving natural interaction while managing the field of view requirement.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A wearable system (200) comprising: a mixed reality display (220); an outward-facing imaging system (464); and a hardware processor programmed to identify a physical remote (1220) associated with a parent device (1210); generate a virtual remote (1260) based at least partly on specifications of the physical remote (1220), the virtual remote comprises a virtual button (1266); cause the display to render the virtual remote (1260); cause the display to render a focus indicator (1262a, 1262b, 1262c), the focus indicator visually connecting the virtual remote (1260) with a real-world view of the parent device (1210); transmit an instruction to the parent device (1210) causing the parent device to perform a function as if the physical remote (1220) is actuated; and cause the mixed reality display to generate a second focus indicator for the virtual button (1266) when movement of the virtual button passes a threshold condition comprising at least one of a threshold position, a threshold duration, or a threshold velocity.