G-Buffer Frame Upscaling with Motion Vectors and Pixel Masking
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing rendering technologies face challenges in efficiently increasing the resolution of frames in the geometry buffer (G-buffer) domain, leading to issues such as ghost effects and aliasing, particularly when upscaling from low to high resolution.
Innovation Solution
The implementation of a pixel masking neural network and a G-buffer reconstruction neural network to insert new pixels, update their values using a motion vector map, and refine them through pixel masking and reconstruction, resulting in a high-resolution G-buffer for improved frame resolution.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional upscaling methods are used to increase frame resolution, then the resolution is improved, but ghost effects and aliasing artifacts occur
Solution Approach 1:
The upscaling process is segmented into multiple stages: first generating a low-resolution G-buffer, then inserting new pixels between original pixels, updating new pixels using motion vectors, masking inaccurate pixels, and finally reconstructing the high-resolution G-buffer. This segmentation allows each stage to address specific aspects of the upscaling problem, preventing ghost effects and aliasing from occurring
Solution Approach 2:
The patent performs preliminary actions by first inserting new pixels and updating their values using motion vectors before the final reconstruction stage. The pixel masking neural network also performs preliminary identification of inaccurate pixels that need correction, allowing the reconstruction network to focus on refining only the necessary areas and avoiding artifacts
2Measurement precision
If high-resolution rendering is performed directly, then the quality is improved, but the computational complexity and processing time increase significantly
Solution Approach 1:
The patent embeds multiple processing stages within each other: the low-resolution G-buffer generation is nested within the upscaling process, which is nested within the pixel insertion stage, which is nested within the motion vector updating stage, and finally nested within the neural network reconstruction stage. This nested structure allows computation to be performed at multiple resolution levels, reducing overall computational complexity while maintaining high-quality output
Solution Approach 2:
Instead of processing all pixels at full resolution from the beginning, the patent performs partial action by first rendering only the low-resolution G-buffer, then selectively inserting and processing only the necessary new pixels. The pixel masking neural network identifies only the inaccurate pixels that require correction, avoiding unnecessary computation on already accurate pixels and reducing overall processing requirements
3Measurement precision
If new pixels are inserted between original pixels, then the resolution is increased, but inaccuracies and artifacts are introduced
Solution Approach 1:
The patent introduces motion vectors as an intermediary mechanism to transfer accurate pixel values from existing pixels to newly inserted pixels. The motion vector map serves as a mediator that guides the updating process, ensuring that new pixels receive values from the correct source pixels based on motion information, thereby maintaining accuracy during the resolution increase process
Solution Approach 2:
The patent implements feedback through the pixel masking neural network, which evaluates the accuracy of pixel values after insertion and updating. The masking output provides feedback information that identifies inaccurate pixels, which then feeds into the reconstruction neural network for correction. This feedback loop ensures that inaccuracies are detected and corrected, maintaining high pixel value accuracy in the final high-resolution output
Data Source
AI summary
A method includes: inserting new pixels between original pixels for each of maps included in a first geometry buffer (or G-buffer) generated from a frame, wherein the maps represent geometric information of a three-dimensional (3D) model of an object included in the frame; generating a second G-buffer by setting values of the new pixels using a motion vector map that may be one of the maps; generating a third G-buffer by combining, with the second G-buffer, a result of updating only values of pixels masked based on an output of a pixel masking neural network to which the second G-buffer may be input; generating a fourth G-buffer by updating values of pixels by inputting the third G-buffer to a G-buffer reconstruction neural network; and update, based on the fourth G-buffer, the resolution of a subsequent frame that follows the frame.