Visual Perception Radiance Fields for Saliency-Guided Scene Rendering
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural radiation fields and their variants require long network inference times and often ignore significant features around the central visual area, leading to lower rendering quality and longer processing times for generating new perspective images.
Innovation Solution
Construct an initial visual perception radiation field based on a scene image set, incorporating an initial density grid, color grid, and visual saliency grid, using user gaze point information to generate a visual sampling rate map, and determine an image rendering result based on a preset loss function, ultimately outputting a target rendered image.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If neural radiation field method is used for image rendering, then rendering capability is achieved, but network inference time becomes excessively long
Solution Approach 1:
The patent divides the rendering process into two distinct stages: a training stage where the neural radiation field learns from multiple viewpoint images, and an inference stage where pre-computed feature maps are used for rapid rendering. This segmentation allows the heavy computational work to be done during training, leaving the inference phase fast and efficient.
Solution Approach 2:
The patent performs preliminary computations during the training phase, including extracting features from multiple viewpoint images and pre-computing the neural radiation field parameters. These pre-computed features are then stored and reused during inference, eliminating the need for repeated heavy computations and significantly reducing inference time.
2Manufacturing precision
If uniform sampling is used to sample points on light rays, then sampling process is simple, but visually salient areas become undersampled
Solution Approach 1:
The patent applies different sampling strategies to different regions of the image based on visual saliency. High-saliency areas receive denser sampling to capture important visual features, while low-saliency areas use coarser sampling. This local differentiation ensures that visually important regions are adequately sampled without unnecessarily increasing computational cost across the entire image.
Solution Approach 2:
The sampling strategy dynamically adjusts the sampling density based on the computed visual saliency map. Instead of using a fixed uniform sampling pattern, the system adapts the sampling points along light rays according to the saliency distribution, placing more samples where needed and fewer where not required, thus optimizing the balance between accuracy and complexity.
3Manufacturing precision
If first coarse then fine sampling is used, then sampling coverage is improved, but visually salient areas become oversampled
Solution Approach 1:
The patent implements local quality by varying the sampling density according to visual saliency. Rather than applying coarse-then-fine sampling uniformly across the entire image, the system identifies visually salient regions and applies appropriate sampling density only to those areas. This prevents oversampling in low-saliency regions while ensuring adequate coverage in important areas.
Solution Approach 2:
The patent uses visual saliency detection as feedback to guide the sampling process. The saliency map, computed from the image content, provides feedback about which regions are visually important, allowing the sampling algorithm to adjust the number and distribution of sampling points accordingly. This feedback mechanism ensures that sampling resources are allocated efficiently to where they are most needed.
Data Source
AI summary
Embodiments of this disclosure disclose an efficient rendering method for complex scenes based on visual perception radiation fields. One specific mode of carrying out this method comprises: constructing an initial visual perception radiation field; selecting a scene image as a sample image, and performing the following steps: generating a visual sampling rate map; determining an image rendering result based on the visual sampling rate map and the initial visual perception radiation field; determining a target difference value between the image rendering result and rendering data of the sample image; in response to determining that the target difference value is less than a preset difference threshold, determining the initial visual perception radiation field, which has completed training, as a visual perception radiation field; inputting rendering perspective information into the visual perception radiation field to output a target rendered image; controlling a display device to display the target rendered image.