Controllable Avatar Reconstruction With Subject Image Priors

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing methods struggle to create photorealistic and controllable avatars from sparse monocular recordings, such as portrait videos, due to under-constrained reconstruction and the lack of sufficient data for effective training, especially when capturing the challenges of animating facial expressions and generating realistic facial features like wrinkles, which are difficult to capture and produce visual artifacts.

Innovation Solution

A method that initializes a controllable avatar using geometric primitives and a parametric morphable model, and learns parameters based on both the image sets and subject image priors, using a pre-trained image generation model to generate specific content model to personalize it for the animated subject, and learns parameters based on the image generation model, and learns parameters based on both the image set and the subject image priors, using a pre-trained image generation model to enhance the controllable avatar's realism and generalization ability.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Device complexity

If methods use only monocular RGB video as input for avatar reconstruction, then device complexity and data acquisition requirements are reduced, but reconstruction accuracy and photorealism deteriorate due to under-constrained problem

Engineering Contradiction:

Improvedata acquisition requirementsVSAvoidreconstruction accuracy

Core Design Contradiction:

Device complexityVSMeasurement precision

Solution Approach 1:

The patent introduces a pre-trained image generation model as an intermediary component that generates synthetic multi-view images from a single input image. This intermediary bridges the gap between sparse monocular observations and the need for comprehensive 3D reconstruction, allowing the system to infer missing view information without requiring actual multi-view camera setups.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The system performs preliminary action by using a pre-trained image generation model that has been trained on extensive 3D avatar data before actual reconstruction. This pre-training establishes prior knowledge about realistic appearances and 3D geometry, which then guides the reconstruction process when only sparse monocular input is available, effectively preparing the system to handle under-constrained problems.

Inventive Principle:

Principle #10Preliminary action

2Adaptability or versatility

If methods reconstruct 4D avatars from sparse monocular recordings, then adaptability to various poses and expressions is improved, but reliability of reconstruction deteriorates due to lack of constraints

Engineering Contradiction:

Improvegeneralization to novel posesVSAvoidreconstruction reliability

Core Design Contradiction:

Adaptability or versatilityVSReliability

Solution Approach 1:

The patent implements feedback mechanisms where the reconstructed avatar is continuously evaluated against the pre-trained image generation model's expectations. The model provides feedback signals that guide the optimization process, allowing the system to adjust reconstruction parameters to achieve more reliable results while maintaining adaptability to diverse poses and expressions.

Inventive Principle:

Principle #23Feedback

Solution Approach 2:

The system employs parameter changes by dynamically adjusting reconstruction parameters based on the input image characteristics and the pre-trained model's guidance. This allows the reconstruction to adapt to different poses and expressions while maintaining reliability through parameter optimization that incorporates prior knowledge of realistic appearances.

Inventive Principle:

Principle #35Parameter changes

3Adaptability or versatility

If methods capture extreme expressions and facial details from limited observations, then avatar expressiveness is improved, but manufacturing precision of facial features deteriorates due to visual artifacts

Engineering Contradiction:

Improvefacial expression captureVSAvoidfacial feature accuracy

Core Design Contradiction:

Adaptability or versatilityVSManufacturing precision

Solution Approach 1:

The patent applies local quality by differentiating the treatment of various facial regions. The pre-trained image generation model provides region-specific guidance, allowing high-precision reconstruction of critical facial features like eyes, nose, and mouth while maintaining appropriate expressiveness. This localized approach ensures that facial details are captured accurately without generating visual artifacts, even when capturing extreme expressions.

Inventive Principle:

Principle #3Local quality

Data Source

PatentEP4664406A1Method and device for creating a controllable avatar

Publication Date: 2025.12.17 TOYOTA JIDOSHA KK

Figure 1
Figure 2
Figure 3~3(d)

AI summary

A computer-implemented method for creating a controllable avatar of an animated subject from at least one image set (10) of the animated subject, the method comprising: - initializing the controllable avatar; - obtaining at least one subject image prior (Îview) from a pre-trained image generation model (20); learning parameters of the controllable avatar based on the at least one image set (10) and the at least one subject image prior (Îview).