Virtual Interface Hand Tracking Beyond Camera View
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing technologies in mixed reality and virtual reality fail to accurately detect hand motion when the hand is not within the image-captured image, hindering effective operation of virtual interfaces.
Innovation Solution
An information processing system and method that utilizes inertial information from a controller mounted on the hand to determine hand motion, combined with image-captured images, to accurately operate virtual interfaces by adjusting their display based on the detected hand motion and position relative to the virtual interface.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If hand motion detection relies solely on image-captured images, then the system can process visual information, but hand motion cannot be detected when the hand is not in the image-captured image
Solution Approach 1:
The patent merges image-captured image processing with inertial measurement unit (IMU) data processing to detect hand motion. The control unit integrates information from both sources, using the IMU to track hand movements even when the hand is not visible in the image-captured image, thereby extending detection range while maintaining accuracy.
Solution Approach 2:
The inertial measurement unit acts as an intermediary device that provides motion information independent of visual capture. When the hand is not in the image-captured image, the IMU data serves as an intermediary source to continue tracking hand motion, bridging the gap between visual detection limits and motion tracking requirements.
2Ease of operation
If the system uses only image-captured images for hand tracking, then visual interface interaction is possible, but the virtual interface cannot be operated when hand is out of frame
Solution Approach 1:
The patent ensures continuous operation of the virtual interface by switching from image-based detection to IMU-based detection when the hand exits the image-captured frame. The control unit maintains continuous tracking of hand motion and virtual interface interaction without interruption, preserving operational continuity despite changes in detection methodology.
3Reliability
If the system switches between image-based and inertial-based detection, then detection reliability improves, but system complexity increases
Solution Approach 1:
The patent implements a dynamic detection system that adapts its methodology based on real-time conditions. The control unit dynamically switches between image-captured image processing and inertial measurement processing depending on whether the hand is detected in the image frame, allowing the system to maintain reliability while managing complexity through conditional operation.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
Enables accurate detection and operation of virtual interfaces even when the hand is not within the image-captured field, ensuring smooth interaction and control through inertial information integration.
Implementation Method 1
determine motion of a hand of a user on a basis of inertial information of a controller held in or mounted on the hand
Data Source
AI summary
An information processing system determines motion of a hand of a user on the basis of inertial information of a controller held in or mounted on the hand. The information processing system controls a display unit to display a virtual space in which a virtual interface is placed in a real space. The information processing system determines, on the basis of an image-captured image in which the hand is image-captured, whether or not a position of the virtual interface and a position of the hand are in a particular relation. In a case in which the position of the virtual interface and the position of the hand are determined to be in the particular relation in the virtual space, the information processing system changes the display of the virtual interface on the basis of the motion of the hand.