Saliency Prediction for 360-Degree Images Using Graph Convolution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current visual saliency prediction methods for 360-degree images face challenges such as distortion in feature extraction and high computational overhead due to projection transformations and interpolation errors in spherical data processing.
Innovation Solution
A saliency prediction method using a graph convolutional neural network that constructs a spherical graph signal through geodesic icosahedron projection, extracts features with a Chebyshev network, and reconstructs a 360-degree saliency map using a spherical crown based interpolation algorithm, avoiding interpolation errors and reducing computational costs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If 360-degree image is projected onto Euclidean space using cube projection, then feature extraction can be performed using convolutional neural network, but distortion is introduced which affects feature extraction performance
Solution Approach 1:
The patent applies spherical CNN that operates directly on spherical data without projecting to Euclidean space. The convolutional kernel is defined on the spherical surface and rotated with the spherical image, maintaining the natural curvature and avoiding projection distortion while enabling feature extraction.
Solution Approach 2:
The patent replaces the traditional cube projection mechanical transformation with a spherical coordinate-based convolution operation. Instead of mechanically projecting spherical data onto flat planes, the system uses spherical mathematics to perform convolution directly on the spherical surface.
2Measurement precision
If multiple tangent plane images are input into neural network for saliency prediction, then comprehensive feature analysis is achieved, but computational overhead increases significantly
Solution Approach 1:
The patent merges the spherical image into a single equidistant rectangular projection format that preserves spherical topology. Instead of processing multiple separate tangent plane images, the system combines all spherical information into one unified representation that can be processed by the spherical CNN.
Solution Approach 2:
The spherical CNN design allows a single network to process the entire spherical image uniformly, rather than requiring separate processing pipelines for multiple tangent planes. The spherical convolution operation is universally applicable across the entire spherical surface.
3Adaptability or versatility
If spherical image is rotated and resampled for convolution operation, then spherical data can be processed, but interpolation errors are introduced which accumulate and seriously affect model performance
Solution Approach 1:
The patent changes the parameter space from Cartesian coordinates (requiring rotation and resampling) to spherical coordinates. By defining convolution operations in the spherical domain, the system processes spherical data natively without needing to rotate or resample the image, eliminating interpolation errors.
Solution Approach 2:
The spherical coordinate system acts as an intermediary that enables direct convolution operations on spherical data. Instead of rotating and resampling the image to align with a fixed kernel, the spherical coordinate system allows the kernel to be defined and applied naturally in the spherical domain.
Data Source
AI summary
The present disclosure provides a saliency prediction method and system for a 360-degree image based on a graph convolutional neural network. The method includes: firstly, constructing a spherical graph signal of an image of an equidistant rectangular projection format by using a geodesic icosahedron composition method; then inputting the spherical graph signal into the proposed graph convolutional neural network for feature extraction and generation of a spherical saliency graph signal; and then reconstructing the spherical saliency graph signal into a saliency map of an equidistant rectangular projection format by using a proposed spherical crown based interpolation algorithm. The present disclosure further proposes a KL divergence loss function with sparse consistency. The method can achieve excellent saliency prediction performance subjectively and objectively, and is superior to an existing method in computational complexity.