Hybrid 2-D/3-D Denoiser for Depth-Aware Bin Alignment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current denoising solutions for three-dimensional (3-D) images, such as deep-Z images, suffer from noise-related artifacts and computational inefficiencies, hindering their use in production environments, while existing 2-D image denoisers are not directly applicable due to misalignment in bin neighborhoods.
Innovation Solution
A hybrid 2-D/3-D neural network architecture that utilizes depth-aware bin indexing and convolution kernels to align bin neighborhoods, improving denoising efficiency and quality in 3-D images.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If conventional deep-Z image denoising filters each bin based on neighboring bins, then denoising is performed, but artifacts such as residual noise or splotches are produced and computational expense increases
Solution Approach 1:
The patent segments the denoising process into two distinct components: a 2-D denoiser that operates on spatial relationships and a 3-D denoiser that operates on depth bin relationships. This segmentation allows each component to specialize in specific types of noise patterns, improving overall denoising quality while distributing computational load efficiently. The 2-D denoiser handles spatial coherence, while the 3-D denoiser handles depth coherence, avoiding the need for a single complex filter to handle all cases.
Solution Approach 2:
The patent merges the outputs of the 2-D and 3-D denoisers through a learned combination mechanism. This merging allows the system to leverage the strengths of both denoisers: the spatial awareness of the 2-D denoiser and the depth awareness of the 3-D denoiser. The combination is optimized to produce high-quality denoised output without requiring either component to be overly complex, thus resolving the contradiction between quality and computational expense.
2Manufacturing precision
If state-of-the-art neural network-based denoisers for flat 2-D images are applied to 3-D images after compositing, then denoising is achieved, but artists must operate on noisy data or have a denoiser run after every compositing operation
Solution Approach 1:
The patent implements preliminary denoising by integrating the denoising operation into the compositing pipeline itself, rather than performing it as a separate post-processing step. The hybrid 2-D/3-D denoiser is applied to the deep-Z image data structure during or immediately after compositing operations, ensuring that artists always work with denoised data without requiring additional processing time. This preliminary action eliminates the need for repeated denoising operations after each compositing step.
3Productivity
If conventional 2-D denoisers are used on 3-D images, then denoising is performed, but denoising quality degrades when non-linear functions are applied to bin colors
Solution Approach 1:
The patent applies local quality by making the denoiser depth-aware, meaning it adapts its filtering behavior based on the depth information in different bins. Instead of applying a uniform 2-D filtering approach, the 3-D denoiser component adjusts its operation according to the local depth characteristics of each bin. This allows the system to maintain high denoising quality even when non-linear functions are applied to bin colors, as the depth-aware filtering preserves the relationships between bins at different depths while still achieving efficient processing through the hybrid architecture.
Data Source
AI summary
A system includes a pre-processor configured to receive three-dimensional (3-D) image data, flatten the 3-D image data to produce corresponding two-dimensional (2-D) image data, and concatenate the 3-D image data and the corresponding 2-D image data to provide concatenated image data. The system further includes an encoder including one or more first neural networks (NNs), the encoder configured to use the one or more first NNs to encode the concatenated image data to provide encoded data, a decoder including one or more second NNs, the decoder configured to use the one or more second NNs to decode the encoded data to provide decoded data, and a reconstructor including a plurality of hybrid 2-D/3-D reconstructors configured to reconstruct the decoded data to provide a denoised 3-D output image corresponding to the 3-D image data.