Implicit Human Body Relighting From Sparse Video and Dynamic Occlusion
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for building digital human avatars are costly, time-consuming, or unable to effectively relight characters under varying lighting conditions, limiting their application in the mass market.
Innovation Solution
A method and apparatus for rebuilding a relightable implicit human body model using sparse video input, involving spatial point mapping, implicit signed distance field calculations, material information derivation, visibility recording, and ambient lighting representation to create a deformable and drivable model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If acquisition devices are used for modeling virtual digital humans, then production cycle is shortened and manpower is saved, but device complexity and cost increase
Solution Approach 1:
The patent replaces physical acquisition devices (scanners, multi-camera arrays) with an artificial intelligence-based system that processes ordinary video inputs. The neural network models directly reconstruct 3D human body representations from 2D video frames, eliminating the need for specialized mechanical scanning equipment while maintaining modeling efficiency.
Solution Approach 2:
The patent creates a virtual copy of the human body through neural rendering techniques, generating a digital twin that can be manipulated and relighted independently of the original physical subject. This allows reconstruction from simple video copies rather than requiring direct physical scanning.
2Illumination intensity
If previous relighting methods are used, then lighting effects can be achieved, but surface roughness is assumed uniform and relighting quality is insufficient
Solution Approach 1:
The patent applies different surface properties to different body parts by estimating varying roughness values for different regions. The system divides the body into multiple patches and assigns localized material properties, allowing realistic differentiation between smooth skin areas and rougher regions, thereby achieving high-quality relighting that respects local surface characteristics.
3Ease of manufacture
If full-automatic modeling with artificial intelligence is used, then cost is reduced and productivity is improved, but relighting capability is lost
Solution Approach 1:
The patent performs preliminary decomposition of surface materials and lighting conditions during the initial AI-based reconstruction phase. By separating albedo, roughness, and lighting components in advance, the system enables subsequent relighting operations without requiring expensive re-scanning or re-modeling, thus maintaining both low cost and relighting versatility.
Data Source
AI summary
The present invention discloses a method and an apparatus for rebuilding a relightable implicit human body model. A human body is represented as a deformable implicit neural representation, and a geometric shape, material attributes, and ambient lighting of the human body are decomposed to obtain a relightable and drivable implicit human body model. In addition, a volumetric lighting grid including a plurality of spherical Gaussian models is introduced to represent complex lighting with spatial variation, and visible probes capable of changing in position with change in human pose are introduced to record dynamic self-occlusion caused by human motion. With the method, drivable implicit models capable of being used for high-fidelity human body relighting may be generated in cases of sparse video input and even monocular input.