Multimodal Near-to-Eye Display Control via Gaze and Gesture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing near-to-eye display control mechanisms, such as motion sensors and gaze tracking systems, are often difficult and inconvenient to use for users, necessitating an improved interaction method that combines gaze tracking and hand gesture detection for efficient control of near-to-eye display devices.
Innovation Solution
A multimodal interaction system that uses a first camera to track the user's gaze for cursor positioning and movement, and a second camera to detect hand gestures when gaze movement is still, allowing for operations like selecting menu options or adjusting cursor positions based on detected gestures, with eye blinking used as an alternative for selection when no hand gesture is detected.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If gaze tracking is used for cursor control, then ease of operation is improved, but reliability is worsened due to difficulty in precise selection
Solution Approach 1:
The patent combines gaze tracking and hand gesture recognition systems into a unified control mechanism. The gaze tracker provides initial cursor positioning while the hand gesture detector confirms selection intent, merging both modalities to achieve reliable and easy operation simultaneously.
Solution Approach 2:
Hand gestures serve as an intermediary confirmation mechanism between gaze positioning and final selection. The system requires both gaze direction and corresponding hand gesture to execute selection, with the hand gesture acting as a mediator that validates the user's selection intent.
2Reliability
If hand gesture detection is added to gaze tracking, then reliability is improved, but device complexity is worsened
Solution Approach 1:
The camera system performs multiple functions: it tracks gaze direction, detects hand gestures, and monitors for hand presence. This multi-functionality allows the same hardware to support both gaze tracking and hand gesture recognition, improving reliability without proportionally increasing device complexity.
Solution Approach 2:
The system dynamically switches between gaze-based control and hand gesture-based control based on detected hand presence. When a hand is detected, the system transitions to gesture-based confirmation; when no hand is present, it relies on gaze tracking alone, creating a flexible and adaptive control system.
3Ease of operation
If multiple control modalities are combined, then ease of operation is improved, but device complexity is worsened
Solution Approach 1:
The control system is segmented into distinct functional modules: gaze tracking module, hand gesture detection module, and selection execution module. Each module handles a specific aspect of control, making the overall system easier to implement and maintain while providing multiple control modalities for improved ease of operation.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The invention relates to a method, comprising tracking the gaze of an eye of a user by a first camera, wherein a position of the gaze determines a place for the cursor on the screen of a near-to-eye display (NED), wherein the movement of the gaze is configured to move the cursor on the screen and when the movement of the gaze is detected to be still or moving only slightly; observing a gesture of one hand of the user by a second camera, and executing a further operation if at least one gesture of the hand is detected. The invention further relates to an apparatus and a computer program product configured to perform the method.