Motion Vector Refinement for Reduced Ghosting in Temporal Supersampling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Temporal Anti-aliasing (TAA) techniques suffer from ghosting artifacts around moving object boundaries due to inaccuracies in motion vectors, leading to lower quality anti-aliased images.
Innovation Solution
A mixed low precision convolutional neural network is employed for temporally amortized supersampling, generating residual motion vectors for pixels without initial motion information, and a residual motion refinement network is trained to improve motion vector accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Stability of the object's composition
If TAA warps sample history using motion vectors to align with current frame, then temporal stability is improved, but ghosting artifacts occur due to motion vector inaccuracies
Solution Approach 1:
The system implements feedback by comparing the warped previous frame with the current frame and using the difference to refine motion vectors. The neural network analyzes the residual errors between warped and actual pixel values, then adjusts motion vectors iteratively to minimize ghosting artifacts while maintaining temporal stability.
Solution Approach 2:
The patent replaces traditional mechanical motion estimation algorithms with a neural network-based system. Instead of relying on hand-crafted motion vector algorithms, the system uses learned patterns from training data to generate more accurate motion vectors, substituting the mechanical algorithmic approach with an intelligent learning-based approach.
2Productivity
If rendering is performed at lower resolutions to maintain performance, then productivity is improved, but image quality deteriorates
Solution Approach 1:
The system transitions from spatial supersampling (rendering at higher spatial resolution) to temporal supersampling (accumulating samples across time). By moving to the temporal dimension, the system achieves anti-aliasing and quality improvement without increasing spatial rendering resolution, thus maintaining performance while enhancing image quality.
Solution Approach 2:
The system performs preliminary warping and accumulation of historical samples before final image composition. By pre-processing and accumulating temporal samples in advance, the system prepares high-quality anti-aliased data that can be composited efficiently, maintaining real-time performance while achieving superior image quality.
3Manufacturing precision
If traditional TAA accumulates temporal samples, then anti-aliasing quality is improved, but ghosting artifacts appear due to visibility and shading changes
Solution Approach 1:
The system applies local quality by treating different regions of the image differently during temporal accumulation. Pixels that have changed visibility or shading are identified and handled separately from stable regions. The neural network learns to apply different weighting and accumulation strategies locally, preserving anti-aliasing quality in stable regions while avoiding ghosting in dynamic regions.
Solution Approach 2:
The system dynamically changes accumulation parameters based on detected visibility and shading changes. Instead of uniform temporal accumulation, the system adjusts accumulation weights, sample selection, and warping parameters according to local scene characteristics. This adaptive parameter adjustment eliminates ghosting artifacts while maintaining anti-aliasing quality.
Data Source
AI summary
A residual network is used to predict a set of residual motion vectors that provide additional motion data for portions of the frame for which motion vectors are not provided, such as animated textures, mirrored/reflected objects, and/or moving objects without motion information.