Virtual Relighting Video Streams Using Depth and Surface Normals
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
There is a need to augment live video image streams with various visual effects, such as virtual lighting effects, that can be composited directly into the video image stream in real-time, while being lightweight in terms of computational resources and physically realistic for 3D objects.
Innovation Solution
The solution involves obtaining a video image stream, assigning depth values to the foreground portion of the images, estimating surface normals based on these depth values, augmenting the images with virtual lighting effects, and transmitting the augmented images to another device.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Illumination intensity
If virtual lighting effects are applied to video image streams in real-time, then visual quality and realism are improved, but computational resource requirements increase
Solution Approach 1:
The patent segments the video processing pipeline into distinct stages: depth map generation, surface normal estimation, and virtual lighting application. By dividing the complex task of real-time virtual relighting into separable components, the system can optimize each stage independently, reducing overall computational burden while maintaining visual quality.
Solution Approach 2:
The patent performs preliminary processing by generating depth maps and estimating surface normals in advance before applying virtual lighting effects. This preliminary action allows the system to prepare geometric information that can be reused across multiple lighting scenarios, reducing real-time computational requirements.
2Illumination intensity
If complex visual effects are composited into live video streams, then visual enhancement is improved, but processing time increases
Solution Approach 1:
The patent implements periodic updates of depth maps and surface normals at optimized intervals rather than computing them for every video frame. This periodic action maintains visual enhancement quality while significantly reducing processing time by avoiding redundant computations in temporally similar frames.
Solution Approach 2:
The patent applies virtual lighting effects selectively to specific regions of interest in the video stream, such as foreground objects or faces, rather than processing the entire scene uniformly. This local quality approach concentrates computational resources where they provide maximum visual benefit while reducing overall processing time.
Data Source
AI summary
Devices, methods, and non-transitory computer-readable media are disclosed herein for augmenting live video image streams with “virtual lighting” visual effects. For example, a first electronic device may obtain a video image stream. Then, for each of one or more images in the video stream, the electronic device may: assign depth values to at least a foreground portion of each image; estimate surface normals for at least the foreground portion of each image based, at least in part, on the assigned depth values (and, e.g., using a machine learning (ML)-based model); and augment each image with at least a specified virtual lighting visual effect based, at least in part, on the estimated surface normals. Finally, the first electronic device may transmit the first augmented output image to a second electronic device. The virtual lighting effect may comprise a specification of one or more properties of one or more virtual light sources.