HMD Object Input Using Real-World Contact Detection

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Users find it difficult to perform input operations on objects displayed on head-mounted displays (HMDs) using gestures, as compared to general-purpose devices like personal computers or smartphones.

Innovation Solution

An information processing apparatus and system that enables an HMD to detect contact between a displayed object and an input apparatus in real space, allowing information input to be received by the object through the HMD, by utilizing a processor to control the display of both the object and input apparatus, and detect intentional user interactions.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of operation

If gesture operations are used to input information on objects displayed on HMD, then the 3D model can be browsed by intuitive operations, but the operability of input operation becomes more difficult compared to general-purpose devices

Engineering Contradiction:

Improveoperability of input operationVSAvoidcomplexity of gesture control system

Core Design Contradiction:

Ease of operationVSDevice complexity

Solution Approach 1:

The patent introduces a camera as an intermediary device to capture images of the user's hand gestures and converts them into virtual objects that can be manipulated. This mediator translates physical gestures into digital interactions, making the system more intuitive and easier to operate while reducing the complexity of direct gesture control programming

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent creates virtual copies of the user's hand gestures by capturing images with a camera and generating corresponding virtual objects. These virtual copies allow the system to interpret and respond to gestures without requiring complex direct gesture recognition algorithms, thereby improving ease of operation

Inventive Principle:

Principle #26Copying

2Productivity

If only gesture operations are supported on HMD, then the system maintains simplicity, but the productivity and efficiency of information input are reduced

Engineering Contradiction:

Improveefficiency of information inputVSAvoidvariety of input methods

Core Design Contradiction:

ProductivityVSAdaptability or versatility

Solution Approach 1:

The patent makes the HMD system multi-functional by integrating multiple input methods including gesture operations, voice commands, and interactions with virtual objects. This universal approach allows users to choose different input methods based on their needs, thereby improving productivity without sacrificing adaptability

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The patent implements dynamic input methods that can adapt to different usage scenarios. The system can switch between gesture-based control, voice commands, and virtual object interactions depending on the context, enabling efficient information input while maintaining versatility across different tasks

Inventive Principle:

Principle #15Dynamics

3Measurement precision

If the HMD displays only the object without showing the input apparatus, then the display remains simple, but the user cannot detect contact between the object and input apparatus

Engineering Contradiction:

Improvedetection accuracy of contactVSAvoidcomplexity of contact detection system

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent implements a feedback mechanism where the HMD displays both the virtual object and the physical input apparatus (such as a smartphone or tablet) in the user's field of view. When contact is detected between them, the system provides visual feedback by showing their spatial relationship, enabling precise contact detection without requiring complex additional sensors

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The patent replaces complex mechanical contact sensors with image processing and spatial analysis algorithms. By capturing images of both the virtual object and input apparatus and analyzing their positional relationships in the displayed view, the system achieves accurate contact detection through software rather than complex hardware

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS12535890B2Information processing apparatus, processing system, and non-transitory computer readable medium storing program to improve operability of input operation on object

Publication Date: 2026.01.27 FUJIFILM BUSINESS INNOVATION CORP

AI summary

An information processing apparatus includes a processor configured to perform control for causing an object and an input apparatus that are capable of receiving an information input operation to be visible through a display device, detect contact between the object and the input apparatus, based on information regarding respective positions of the object and the input apparatus that are visible through the display device, and receive information input to the input apparatus, as information input to the object, in a case where the contact is detected.