Avatar Camera Mapping Using Facial Landmarks and Head Orientation

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Conventional systems in virtual environments fail to accurately reflect user movements, leading to static backgrounds and potential user discomfort such as eye strain and nausea, due to imprecise gyroscope and accelerometer data in mobile devices.

Innovation Solution

A metaverse application uses facial landmarks to map the orientation of a mobile device to the head orientation of a user, incorporating roll, yaw, and pitch, and adjusts the background perspective accordingly, while also determining distance changes to animate avatars moving closer or farther away.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Device complexity

If conventional systems use static background mapping in virtual experiences, then device complexity is reduced, but realism and user immersion deteriorate

Engineering Contradiction:

Improvesystem complexityVSAvoidrealism of virtual experience

Core Design Contradiction:

Device complexityVSReliability

Solution Approach 1:

The system dynamically updates the background perspective in real-time based on detected head orientation changes. Instead of using a static background mapping, the system continuously adjusts the background view to match the user's head movements, thereby improving realism without requiring overly complex pre-programmed scenarios. This dynamic adaptation resolves the contradiction by making the system flexible and responsive rather than rigid and pre-determined.

Inventive Principle:

Principle #15Dynamics

Solution Approach 2:

The system uses the device's own sensors (accelerometer, gyroscope) to detect head orientation and automatically updates the background perspective without requiring external input or complex processing. The device serves itself by utilizing its built-in capabilities to maintain realism, reducing the need for additional complex systems while preserving immersion.

Inventive Principle:

Principle #25Self-service

2Speed

If conventional systems ignore head orientation changes, then processing speed is improved, but user comfort and immersion deteriorate

Engineering Contradiction:

Improveprocessing speedVSAvoideye strain and nausea

Core Design Contradiction:

SpeedVSObject-affected harmful factors

Solution Approach 1:

The system replaces complex mechanical tracking systems with sensor-based detection using the device's accelerometer and gyroscope. This substitution allows for rapid detection of head orientation changes through electronic sensor data rather than mechanical tracking, maintaining high processing speed while accurately capturing user movements to prevent discomfort.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The system changes the parameter being monitored from full 3D position tracking to specific orientation parameters (roll, pitch, yaw) derived from sensor data. This parameter change simplifies processing while still capturing the essential head movement information needed to update the background perspective, thereby maintaining speed while improving user comfort.

Inventive Principle:

Principle #35Parameter changes

3Power

If conventional systems use simple 2D avatar representation, then rendering performance is improved, but immersion and realism deteriorate

Engineering Contradiction:

Improverendering performanceVSAvoidimmersion quality

Core Design Contradiction:

PowerVSReliability

Solution Approach 1:

The system transitions from 2D avatar representation to 3D spatial background rendering that responds to head orientation. By adding the dimension of dynamic perspective adjustment based on detected head movements, the system enhances immersion without requiring fully complex 3D avatar models. This partial dimensionality change improves realism while maintaining rendering performance.

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

Data Source

PatentUS12633025B2Camera mapping in a virtual experience

Publication Date: 2026.05.19 ROBLOX CORP

AI summary

A metaverse application receives a first frame of a video. The metaverse application determines facial landmarks of the user in the first frame. The metaverse application generates an animation frame that includes an avatar and a background based on the facial landmarks and the first frame. The metaverse application determines a head orientation of the user in the first frame based on the facial landmarks. The metaverse application maps an orientation of the mobile device to the head orientation of the user. For each additional frame of the video subsequent to the first frame, the metaverse application updates the orientation of the mobile device. The metaverse application generates subsequent animation frames that include the avatar and the background based on the updated orientation of the mobile device in relation to the head orientation.