Body UI AR Controls Using Hand Collision Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing messaging apps lack the ability to activate user interface elements without physical contact with the device, limiting user interaction in scenarios where the user is not holding the device, such as taking full-body photos or trying on virtual items.
Innovation Solution
Implementing a body UI AR component that detects collisions of objects in camera images to activate user selectable elements, using machine learning to identify anchor points on a person's body and track hand movements to perform actions like recording or capturing images without physical contact.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If traditional touch-based UI elements are used, then the device is easy to operate when held, but the user cannot interact with the device when not holding it (e.g., for full-body photos)
Solution Approach 1:
The patent replaces the mechanical touch-based interaction system with a computer vision-based detection system. The camera captures images of the user's hand gestures, and machine learning algorithms detect and interpret these gestures to trigger UI actions, eliminating the need for physical contact with the device.
Solution Approach 2:
The patent introduces an intermediary system consisting of camera-based hand detection and gesture recognition algorithms. This intermediary translates physical hand gestures into digital commands, enabling touchless interaction while maintaining intuitive control.
2Adaptability or versatility
If the user must hold the device to interact with UI elements, then touch interaction is precise and reliable, but the user cannot take full-body photos or try on virtual items
Solution Approach 1:
The patent substitutes mechanical touch interaction with optical-based gesture recognition. The system uses the camera to capture hand gestures and machine learning models to reliably interpret these gestures, providing a dependable alternative to touch interaction for AR scenarios.
Solution Approach 2:
The patent changes the interaction parameter from physical contact (touch) to spatial gesture recognition. By detecting hand position, orientation, and movement in 3D space, the system enables new interaction modes that work when the device is not held, while maintaining reliability through robust gesture recognition algorithms.
3Ease of operation
If touch-based UI elements are used on camera screen, then the interface is simple and direct, but the user cannot activate elements without physical contact
Solution Approach 1:
The patent replaces the mechanical touch interface with an optical gesture-based interface. The camera captures hand gestures, and the system translates these gestures into UI commands, maintaining interface simplicity while adding touchless interaction capability for scenarios where the device is not held.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The technical problem of providing user interface elements that can be activated without physical contact with any specific device is addressed by configuring an augmented reality component to display a touchless user selectable element that can be activated in response to detecting collision of an object from the output of a digital image sensor of a camera with a user selectable element. The augmented reality component detects collision of the touchless user selectable element with an object from the person image, such as a hand object, and, in response, triggers a predetermined action, such as, for example, starting or stopping recording of the output of a digital image sensor or capturing a still image of the output of a digital image sensor.