Neural 3D Portrait Relighting Under Ambient Video Capture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for creating controllable 3D portraits require controlled lighting conditions, limiting their capture to professional environments and restricting the creation of photo-realistic, animated portraits to devices with specific lighting capabilities.
Innovation Solution
A method for capturing and processing video portraits in any ambient lighting conditions, disentangling facial appearance into intrinsic components, and projecting dynamic face normals to create a controllable, neural 3D portrait using a guided deformation field and neural radiance field.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If controlled lighting conditions are used to create photo-realistic 3D portraits, then the quality and realism of the portrait is improved, but the capture is limited to professional environments and devices with specific lighting capabilities
Solution Approach 1:
The patent changes the lighting parameter from controlled professional lighting to ambient lighting conditions. By using neural radiance fields and learned representations, the system adapts to varying lighting parameters (intensity, direction, color temperature) in ambient environments, eliminating the need for controlled lighting while maintaining portrait quality.
Solution Approach 2:
The patent replaces the mechanical lighting system (physical lights and controlled environments) with a computational approach using neural networks and radiance field models. The system learns to compensate for ambient lighting conditions through data-driven methods rather than physical light control.
2Manufacturing precision
If controlled lighting conditions and professional equipment are used, then photo-realistic portraits can be produced, but the creation process becomes complex and requires specific devices
Solution Approach 1:
The patent substitutes complex physical lighting equipment and controlled environment infrastructure with computational models. Neural radiance fields and deformation fields replace physical light sources, cameras, and positioning systems, dramatically reducing device complexity while maintaining output quality.
Solution Approach 2:
The patent creates a computational copy of the lighting and geometry information through neural representations. Instead of requiring physical light sources and precise camera positioning, the system learns to replicate the appearance and structure of the subject under ambient lighting through learned radiance fields.
3Adaptability or versatility
If ambient lighting conditions are used for capture, then the system becomes adaptable to various environments and mobile devices, but the lighting variability complicates the rendering process
Solution Approach 1:
The patent performs preliminary learning during a training phase where the system captures images under various ambient lighting conditions and learns the lighting characteristics, surface properties, and geometric structure. This preliminary action creates a compact representation that simplifies subsequent rendering operations.
Solution Approach 2:
The patent introduces neural radiance fields as an intermediary representation between the ambient lighting conditions and the final rendered output. This intermediary model absorbs the complexity of variable lighting, providing a simplified interface for controlling and rendering the portrait under different conditions.
Data Source
AI summary
Certain aspects and features of this disclosure relate to providing a controllable, dynamic appearance for neural 3D portraits. For example, a method involves projecting a color at points in a digital video portrait based on location, surface normal, and viewing direction for each respective point in a canonical space. The method also involves projecting, using the color, dynamic face normals for the points as changing according to an articulated head pose and facial expression in the digital video portrait. The method further involves disentangling, based on the dynamic face normals, a facial appearance in the digital video portrait into intrinsic components in the canonical space. The method additionally involves storing and/or rendering at least a portion of a head pose as a controllable, neural 3D portrait based on the digital video portrait using the intrinsic components.