Head-Mounted Touch Sensing on Skin Without Depth Cameras
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing AR/VR input systems lack robustness and accuracy in distinguishing touch inputs across diverse lighting conditions and skin tones without requiring user instrumentation, and they often rely on expensive and inaccurate depth cameras.
Innovation Solution
A system utilizing a head-mounted image sensor to detect touch inputs on the user's skin by analyzing deformation and color changes, providing high-resolution passthrough and real-time touch event recognition without additional hardware.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If depth cameras are used to detect touch inputs, then touch detection capability is provided, but cost increases and accuracy decreases
Solution Approach 1:
The patent uses a standard RGB camera to capture optical images of the user's skin, creating a visual copy of the touch interaction. This optical copy is then processed through machine learning models to detect touch events, replacing the need for expensive depth cameras while maintaining detection capability through image analysis of skin deformation and color changes
Solution Approach 2:
The patent replaces the mechanical/optical depth sensing system with a 2D image-based detection system. Instead of measuring physical depth directly through specialized sensors, the system uses standard camera imaging combined with computational algorithms to infer touch events from visual changes in skin appearance, substituting a simpler optical system for a complex mechanical one
2Reliability
If RGB cameras with illumination are used to detect touches, then touch detection is enabled, but accuracy deteriorates under varying lighting conditions
Solution Approach 1:
The system captures a reference image of the user's skin before the touch event occurs. This preliminary reference image is stored and used for comparison with the current image frame, allowing the system to detect changes caused by touch while compensating for ambient lighting variations. The reference serves as a baseline that adapts to different lighting conditions
Solution Approach 2:
The system uses machine learning models that are trained on diverse lighting conditions and skin tones, incorporating feedback from the reference image comparison. The model adjusts its detection thresholds and parameters based on the visual differences detected between the reference and current frames, enabling robust touch detection across varying environmental conditions
3Measurement precision
If instrumented arms or hand devices are used, then touch detection accuracy improves, but device complexity and user burden increase
Solution Approach 1:
The patent makes the head-mounted display's existing camera serve multiple functions: it provides both the primary display function and touch detection capability. The same imaging sensor used for visual display also captures the optical changes in skin during touch interactions, eliminating the need for separate instrumentation devices and making the system universally applicable to all HMD users
Solution Approach 2:
The system uses the HMD device's own built-in camera to perform touch detection on the user's body. The device serves itself by utilizing its existing imaging capabilities for dual purposes, eliminating the need for external sensors, armbands, or hand-mounted instruments that would burden the user with additional equipment
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The system achieves high accuracy and robustness in touch detection across varying lighting conditions and skin tones, supporting rich input metadata like touch force and finger identification, compatible with mobile hardware.
Implementation Method 1
The change in appearance can result from deformation (i.e. depression in otherwise flat surface)
Implementation Method 2
The change in appearance can result from deformation (i.e. depression in otherwise flat surface) or color (i.e. shadowing or skin color change)
Data Source
AI summary
A system and method provide touch input for electronic devices and utilize and image sensor to detect contact between a finger and an appendage of a user, denoting a touch. Identification of a touch event is based on a change in appearance of the user's skin at the location of contact. The image sensor can be worn on or near the head of the user. Further, the system and image sensor can be incorporated into a device such as an AR/VR headset or smart glasses. No further instrumentation of the user is required to detect touch and the system and method are capable of performing in a range of lighting conditions across varied user skin tones.