Portrait Animation Using 2D Deformation From a Single Image
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current portrait animation techniques struggle to achieve photorealistic results in real-time on standard mobile devices, with deep learning methods being time-consuming and morphable face models lacking realism, while existing methods require multiple images or videos for accurate animation.
Innovation Solution
A method and system for photorealistic real-time portrait animation on mobile devices using a single target image, involving a 3D face model, sparse correspondence, and 2D deformations to apply facial expressions and head movements directly to the target image, with background restoration for a photorealistic effect.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If deep learning methods are used to re-render a face with different facial expressions, then photorealistic results are obtained, but the process becomes time-consuming and unsuitable for real-time animation on mobile devices
Solution Approach 1:
The patent segments the face into multiple morphable components (identity, expression, pose, illumination) that can be independently manipulated. This allows the system to apply lightweight geometric transformations and texture blending instead of heavy deep learning re-rendering, achieving photorealistic results in real-time on mobile devices.
Solution Approach 2:
The patent introduces a multi-layer composite structure as an intermediary between the source and target images. This composite includes separate layers for identity, expression, pose, and illumination, which are blended using alpha compositing and texture mapping. This intermediary structure enables photorealistic animation without requiring time-consuming deep learning re-rendering.
2Productivity
If morphable face models are used to re-render a face with different facial expressions, then fast processing is achieved, but the resulting face is not photorealistic
Solution Approach 1:
The patent creates a composite image structure that combines multiple semantic components (identity map, expression map, pose map, illumination map) with their corresponding alpha masks. This composite approach allows the system to leverage fast morphable face model transformations while adding photorealistic details through layered texture blending and alpha compositing, resolving the quality-speed tradeoff.
3Measurement precision
If existing portrait animation methods are used, then accurate animation is achieved, but multiple images or videos are required as input
Solution Approach 1:
The patent extracts and separates the essential animation components (expression, pose, illumination) from the source image using facial landmark detection and morphable face model fitting. By extracting only the necessary dynamic components rather than requiring full source and target images/videos, the system achieves accurate animation with simplified single-image input.
Solution Approach 2:
The patent enables the target image to animate itself by projecting the extracted expression and pose parameters from the source image onto the target's facial geometry. The target image's own identity and structure are preserved while it adopts the source's expressions and poses, eliminating the need for multiple input images or videos.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Provided are systems and methods for photorealistic real-time portrait animation. An example method includes receiving a scenario video with at least one input frame. The input frame includes a first face. The method further includes receiving a target image with a second face. The method further includes determining, based on the at least one input frame and the target image, two-dimensional (2D) deformations, wherein the 2D deformations, when applied to the second face, modify the second face to imitate at least a facial expression and a head orientation of the first face. The method further includes applying, by the computing device, the 2D deformations to the target image to obtain at least one output frame of an output video.