Splatting-Based Digital Image Synthesis Without Full Neural Networks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional digital image synthesis techniques are computationally intensive, consuming significant resources due to the need to execute neural networks for each frame, making them unsuitable for common usage scenarios, especially with high-resolution or multiple frames.
Innovation Solution
The use of splatting-based digital image synthesis techniques, involving forward warping and backward warping, reduces computational resource consumption by generating warped digital images without relying on the full execution of neural networks, utilizing optical flow generation, splat metrics, and Gaussian kernels to enhance accuracy and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional neural network-based frame interpolation is used, then visual accuracy is improved, but computational resource consumption increases significantly
Solution Approach 1:
The patent segments the image synthesis process into distinct warping stages (forward warping, backward warping) that can be executed separately and efficiently, avoiding the need to execute a complete neural network for each frame while maintaining visual accuracy through structured warping operations
Solution Approach 2:
The patent applies partial action by using only the necessary warping operations (forward and backward warping with merge metrics) rather than executing the full neural network pipeline, achieving sufficient visual accuracy without the excessive computational resources required by conventional approaches
2Reliability
If deep learning techniques are fully executed for each output frame, then visual artifacts are resolved, but processing time becomes untenable
Solution Approach 1:
The patent performs preliminary warping operations (forward warping to create warped source images and backward warping to create warped target images) before the final merge step, allowing visual artifacts to be resolved through structured warping rather than requiring full deep learning execution for each frame, thus reducing processing time while maintaining reliability
3Measurement precision
If high-resolution frames are processed using conventional techniques, then synthesis quality is maintained, but resource consumption becomes unsustainable
Solution Approach 1:
The patent segments the high-resolution frame processing into efficient warping stages that operate independently, allowing synthesis quality to be maintained through structured forward and backward warping operations without requiring the unsustainable resource consumption of conventional full neural network execution on high-resolution frames
Data Source
AI summary
Digital image synthesis techniques are described that leverage splatting, i.e., forward warping. In one example, a first digital image and a first optical flow are received by a digital image synthesis system. A first splat metric and a first merge metric are constructed by the digital image synthesis system that defines a weighted map of respective pixels. From this, the digital image synthesis system produces a first warped optical flow and a first warp merge metric corresponding to an interpolation instant by forward warping the first optical flow based on the splat metric and the merge metric. A first warped digital image corresponding to the interpolation instant is formed by the digital image synthesis system by backward warping the first digital image based on the first warped optical flow.