Image Decomposition for Ghosting-Free Motion Estimation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current image-based methods for denoising, upsampling, and frame interpolation in 3D computer graphics face challenges due to the ill-posed nature of the problems, leading to undesirable interference of different effects and ghosting artifacts, especially in complex scenes with specular reflections and refractions.
Innovation Solution
A customizable decomposition framework separates the final pixel color into disjoint components corresponding to subsets of light paths, using motion vectors and auxiliary features like reflectance and surface normals, with temporal extensions of manifold exploration for specular effects and optical flow for accurate motion estimation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If image-based methods (denoising, upsampling, frame interpolation) are used to reduce computational costs, then rendering efficiency is improved, but image quality deteriorates due to interference of different effects and ghosting artifacts
Solution Approach 1:
The patent segments the final pixel color into disjoint components corresponding to different light path subsets (direct lighting, indirect diffuse, specular reflections, refractions). Each component is processed separately through image-based methods, preventing interference between different lighting effects while maintaining high rendering efficiency.
Solution Approach 2:
The patent introduces motion vectors and auxiliary features (reflectance, surface normals, depth) as intermediary data structures that guide the image-based processing. These intermediaries provide auxiliary cues that improve the robustness of denoising, upsampling, and frame interpolation, reducing ghosting artifacts while maintaining computational efficiency.
2Loss of energy
If fewer path samples are used in path tracing, then computational cost is reduced, but image quality deteriorates due to increased noise
Solution Approach 1:
The patent segments the rendering process into path tracing (generating fewer samples) and post-processing (denoising). By separating the lighting calculation from the noise reduction, the system achieves high image quality with reduced computational cost during the rendering phase.
Solution Approach 2:
The patent uses auxiliary features (motion vectors, reflectance, surface normals, depth) as intermediaries to guide the denoising process. These intermediaries provide structural information that enables effective noise reduction even with fewer path samples, maintaining image quality while reducing computational cost.
3Device complexity
If traditional optical flow is used for motion estimation, then computational simplicity is maintained, but accuracy deteriorates in scenes with specular effects and complex lighting
Solution Approach 1:
The patent uses auxiliary features (motion vectors from path tracing, reflectance, surface normals, depth) as intermediaries to enhance optical flow. These intermediaries provide additional constraints and guidance, improving motion estimation accuracy in complex scenes without significantly increasing algorithmic complexity.
Solution Approach 2:
The patent extends optical flow by incorporating additional parameters and constraints from the path tracing process (motion vectors, auxiliary features). This modifies the optical flow algorithm to better handle specular effects and complex lighting while maintaining reasonable computational complexity.
Data Source
AI summary
Particular embodiments perform a light path analysis of an image comprising a scene, wherein the scene comprises at least one refractive or reflective object. The image may be decomposed based on the light path analysis into a plurality of components, each of the components representing a contribution to lighting in the scene by a different type of light interaction. For each of the components, one or more motion vectors are extracted for each of the components in order to capture motion in the scene. Finally, a final contribution of each of the components to the image is computed based on the motion vectors.