Hybrid Touchless UI Navigation Using Finger Vectors and Facial Cues
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
There is a need for an improved mechanism to navigate user interfaces without direct contact, particularly in electronic devices shared by multiple individuals to reduce the risk of transmission and contamination, as highlighted by social distancing measures during the COVID-19 pandemic.
Innovation Solution
A hybrid touchless control mechanism using a combination of finger vector based control, facial expressions, and gestures to navigate user interfaces, employing a computing device with a video capture module, facial region analyzer, and virtual applicator to apply makeup effects without the need for a mouse or touchscreen.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If touchscreen or mouse input is used, then ease of operation is improved, but hygiene and transmission risk are worsened
Solution Approach 1:
The patent replaces mechanical contact input devices (touchscreen, mouse) with a gesture-based control system that uses computer vision to detect hand movements and facial expressions. This substitution eliminates physical contact with shared surfaces while maintaining intuitive control, directly resolving the contradiction between ease of operation and hygiene risk.
Solution Approach 2:
The system introduces an intermediary layer - a camera-based gesture recognition system - that translates physical gestures into digital commands without requiring direct contact with the device. This intermediary mechanism allows users to control the interface from a distance, preserving ease of use while eliminating transmission risks associated with touch surfaces.
2Object-affected harmful factors
If touchless control mechanism is implemented, then hygiene is improved, but device complexity is worsened
Solution Approach 1:
The patent leverages the existing camera module in smartphones and tablets, making it multi-functional by using it both for standard photography/video and for gesture recognition. This approach avoids adding dedicated hardware sensors, thereby reducing the increase in device complexity while still achieving touchless control for improved hygiene.
Solution Approach 2:
The system utilizes the device's own existing hardware components (camera, processor, existing AI frameworks) to enable gesture control, rather than requiring external specialized equipment. This self-service approach minimizes additional complexity while delivering the hygiene benefits of touchless interaction.
3Ease of operation
If traditional touchscreen interface is used, then ease of operation is improved, but transmission risk is worsened
Solution Approach 1:
The patent replaces the mechanical touchscreen interface with a gesture-based control system that operates without physical contact. This substitution maintains operational ease through intuitive hand movements and facial expressions while eliminating the transmission risk associated with touching shared device surfaces.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A computing device captures a live video of a user, determines a location of a facial region of the user by a facial region analyzer, and determines a finger vector type by a finger vector detector based on a direction in which at least one finger is pointing relative to the facial region of the user. Responsive to detecting a first finger vector type within the facial region involving a single finger, a makeup effects toolbar is displayed in the user interface. Responsive to detecting a second finger vector type involving the single finger, a selection tool for selecting a makeup effect in the makeup effects toolbar is displayed. The computing device obtains a makeup effect based on manipulation by the user of the selection tool. Responsive to detecting a target user action, virtual application of the selected makeup effect is performed on the facial region of the user.