VR Avatar Hand Pose Estimation During Camera Tracking Loss

Solution Overview

Problem

In virtual reality systems, temporary failures in hand detection by cameras can cause sudden disappearance of user avatars, leading to unnatural movements and user discomfort, and widening the camera view or adding more cameras increases complexity and cost.

Innovation Solution

An information processing device that uses a head information acquisition unit and physical information acquisition unit to maintain the continuity of hand position and posture by estimating based on past sensor data, allowing smooth transitions and natural avatar movements during detection failures.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If the camera view is widened or more cameras are added to maintain continuous hand detection, then detection reliability improves, but device complexity and cost increase

Engineering Contradiction:

Improvehand detection reliabilityVSAvoidcamera system complexity

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent introduces an intermediary estimation mechanism that bridges the gap during detection failures. When the camera fails to detect the hand, the system uses the last detected position combined with head movement information to estimate the current hand position, rather than directly relying on camera detection alone. This intermediary estimation maintains continuity without requiring additional cameras or wider views.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent replaces the mechanical solution of adding more cameras or widening the camera view with a computational estimation approach. Instead of physically expanding the detection system, the system uses algorithmic estimation based on historical data and head movement to maintain detection reliability, substituting physical expansion with intelligent computation.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Reliability

If the camera view is widened or more cameras are added to maintain continuous hand detection, then detection reliability improves, but cost increases

Engineering Contradiction:

Improvehand detection reliabilityVSAvoidsystem cost

Core Design Contradiction:

ReliabilityVSEase of manufacture

Solution Approach 1:

The patent introduces an intermediary estimation mechanism that bridges the gap during detection failures. When the camera fails to detect the hand, the system uses the last detected position combined with head movement information to estimate the current hand position, rather than directly relying on camera detection alone. This intermediary estimation maintains continuity without requiring additional cameras or wider views.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent replaces the mechanical solution of adding more cameras or widening the camera view with a computational estimation approach. Instead of physically expanding the detection system, the system uses algorithmic estimation based on historical data and head movement to maintain detection reliability, substituting physical expansion with intelligent computation.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Device complexity

If the camera frequently loses hand detection, then device complexity remains low, but avatar display naturalness deteriorates

Engineering Contradiction:

Improvecamera system complexityVSAvoidavatar display continuity

Core Design Contradiction:

Device complexityVSStability of the object's composition

Solution Approach 1:

The patent performs preliminary action by continuously tracking head movement and maintaining history of hand positions even when hand detection is lost. This preliminary tracking of related information (head position) prepares the system to estimate hand position during detection failures, ensuring smooth avatar display without sudden interruptions.

Inventive Principle:

Principle #10Preliminary action

Solution Approach 2:

The patent ensures continuity of useful action by maintaining avatar display through estimation during detection failures. Rather than allowing the avatar to suddenly disappear or jump when hand detection fails, the system continuously updates the avatar position using estimation based on head movement and historical data, maintaining smooth and natural display continuity.

Inventive Principle:

Principle #20Continuity of useful action

Data Source

PatentUS12487662B2Information processing device, information processing method, and program

Publication Date: 2025.12.02 SONY GROUP CORP

AI summary

An information processing device (TM) includes a head information acquisition unit (21), a physical information acquisition unit (22), and a display control unit (30). The head information acquisition unit (21) acquires the position and posture of a head of a user. The physical information acquisition unit (22) acquires a position and posture of a portion of a body other than the head of the user. The display control unit (30) controls display of a virtual object corresponding to the portion of the body, on the basis of a history of the position and posture of the portion of the body and on the basis of the position and posture of the head, in a period before the position and posture of the portion of the body are acquired by the physical information acquisition unit (21).