Runtime Denoising Machine Learning Engine for Ray Tracing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing denoising techniques for ray tracing, such as those using convolutional neural networks, require high-quality training data that may not generalize well to new scenes or environments, leading to suboptimal performance when applied to different settings, like transitioning from land to water-based scenes.
Innovation Solution
A machine learning engine, like a CNN, is continuously trained and updated during runtime by dynamically gathering learning data and refining its weights using a sub-region of the image every frame or every N frames, with new reference regions rendered at high sample count and low sample count inputs, allowing for on-the-fly adaptation and improvement.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If a convolutional neural network is trained on a large set of frames with varying distributed effects from a film, then high-quality denoising results suitable for production are achieved, but the training data may not generalize well to new scenes or environments with significantly different style and content
Solution Approach 1:
The patent implements dynamic adaptation by allowing the neural network to continuously update its weights during runtime based on new input data. Instead of using a static trained model, the system dynamically adjusts its parameters to adapt to different scenes, styles, and content, enabling generalization without retraining from scratch.
Solution Approach 2:
The system changes the parameters (weights) of the neural network during runtime based on the characteristics of new input data. By adjusting these parameters dynamically, the model can adapt to different distributed effects, lighting conditions, and scene types, resolving the contradiction between maintaining high denoising quality and achieving versatility across different environments.
2Manufacturing precision
If a neural network is trained offline with extensive pre-computed reference data, then high-quality denoising is achieved for the training scenes, but extensive pre-computed reference data is required and performance degrades on unseen environments
Solution Approach 1:
The neural network performs self-updating by automatically adjusting its own weights during runtime based on the input data it receives. This self-service mechanism eliminates the need for extensive offline training data and allows the model to continuously improve its performance on-the-fly without requiring large quantities of pre-computed reference data.
Solution Approach 2:
The system performs preliminary adaptation by continuously preparing and updating the network weights in advance during runtime, so that when new scenes or environments are encountered, the model is already adapted and ready to provide high-quality denoising without requiring extensive pre-computed training data for those specific scenarios.
3Productivity
If the neural network weights are fixed after initial training, then the denoising process is fast and computationally efficient, but the model cannot adapt to new environments or varying distributed effects
Solution Approach 1:
The patent makes the previously static neural network weights dynamic by enabling continuous updates during runtime. This dynamic approach allows the model to maintain computational efficiency while adapting to new environments, resolving the contradiction between processing speed and adaptability.
Solution Approach 2:
The system implements continuous useful action by continuously updating the network weights during runtime rather than performing discrete retraining operations. This continuous adaptation maintains high processing speed while enabling the model to adapt to varying distributed effects and new environments throughout operation.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Apparatus and method for improving denoising of images generated by a rendering engine such as a ray tracing engine. For example, one embodiment renders a first plurality of images during runtime using a first sample count. Denoising is performed on the first plurality of images during runtime by a machine-learning engine. In addition, a reference region is generated from one or more of the first plurality of images at a second sample count which is greater than the first sample count. The reference region is then used to perform additional runtime training of the machine-learning engine.