Portrait Image Relighting With Physics-Based Skin and Shadow Rendering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing relighting technologies face challenges in simulating complex lighting effects and non-Lambertian interactions in portrait images, requiring specialized equipment and significant data collection efforts, while deep learning methods fall short in achieving realistic and immersive relighting.
Innovation Solution
A method involving a server-based system that extracts image characteristics using a foreground extraction model, performs reverse rendering to derive normal maps and albedo maps, and generates relighted images through a physics-based model integrated with a self-supervised pre-training framework, utilizing a multi-masked autoencoder for precise reconstruction.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If light stage technology is used to capture lighting diversity, then lighting effect accuracy is improved, but device complexity and data collection requirements increase
Solution Approach 1:
The patent uses a neural network to learn and copy lighting transformation patterns from optical stage data, then applies these learned transformations to new images. This replaces the need for actual optical stage equipment in deployment, achieving accurate lighting effects through software-based transformation rather than hardware-based capture.
Solution Approach 2:
The patent replaces complex mechanical/optical systems (light stage equipment) with a computational approach using neural networks. The system learns lighting transformations from training data and applies them through algorithmic processing, eliminating the need for specialized physical equipment in the actual relighting application.
2Manufacturing precision
If optical stage data collection is performed extensively, then relighting accuracy is improved, but time and effort requirements increase
Solution Approach 1:
The patent performs extensive data collection and model training in advance during an offline phase. The neural network is pre-trained on optical stage data to learn lighting transformations. During actual deployment, the pre-trained model quickly applies learned transformations without requiring additional time-consuming data collection, achieving fast relighting with high accuracy.
3Extent of automation
If deep learning methods are used for relighting, then automation is improved, but realism of lighting effects deteriorates
Solution Approach 1:
The patent introduces an intermediate physical-based rendering model between the neural network and the final relighted image. The neural network predicts lighting parameters, which are then fed into a physics-based rendering model that computes realistic light-surface interactions. This intermediary ensures physically accurate lighting effects while maintaining automation through neural network-driven parameter estimation.
4Productivity
If conventional relighting methods are used, then processing speed is improved, but ability to handle complex non-Lambertian effects deteriorates
Solution Approach 1:
The patent changes the approach from direct pixel manipulation to parameter-based lighting control. The neural network estimates physical lighting parameters (light direction, intensity, color) and material properties, then uses these parameters to compute relighted images through physics-based rendering. This parameter-based approach efficiently handles complex non-Lambertian effects by modeling their underlying physical causes rather than attempting direct pixel-level adjustments.
Data Source
AI summary
Disclosed is a method of generating a relighted image based on an object image according to various embodiments of the present invention for realizing the problems described above. The method includes acquiring a source original image, acquiring image characteristic information based on the source original image, and generating the relighted image based on the source original image, the image characteristic information, and target lighting information, in which the relighted image is an image reflecting a realistic human skin tone, texture, and a shadow effect under the target lighting conditions, and is an image whose a lighting effect is changed compared to the source original image.