Neural Network Static Feature Maps for Spatial Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Convolutional neural networks struggle to effectively encode spatial information and generalize well to new viewpoints and different cameras due to their limited receptive field and reliance on local filters, which affects their performance in tasks requiring rich and detailed environmental representations for autonomous vehicles and robotic systems.
Innovation Solution
A neural network system is developed that explicitly encodes spatial information by incorporating static feature maps to represent camera intrinsic and extrinsic parameters, including lens distortion and pose, allowing it to generalize across various cameras and viewpoints.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If convolutional neural networks use only local filters with limited receptive field, then the network architecture remains simple and computationally efficient, but the ability to encode spatial information is insufficient
Solution Approach 1:
The patent introduces coordinate channels that add spatial position information as an additional dimension to the feature maps. This allows the network to access global spatial information without fundamentally changing the convolutional operation, effectively resolving the contradiction by encoding spatial information in a new dimensional space rather than increasing network complexity through deeper or wider architectures.
Solution Approach 2:
The patent introduces coordinate maps as intermediary elements that carry spatial position information from the input image to subsequent processing layers. These coordinate maps act as mediators between the local filter operations and the need for global spatial understanding, allowing standard convolutional operations to work effectively while still accessing global spatial context.
2Reliability
If neural networks are trained on specific camera parameters, then they achieve high accuracy for those specific cameras, but they fail to generalize well to new viewpoints and different cameras
Solution Approach 1:
The patent makes the network architecture universal by introducing coordinate channels that work with any camera parameters. The coordinate-based approach creates a common reference frame that is independent of specific camera intrinsic or extrinsic parameters, allowing the same network architecture to process images from any camera viewpoint or camera model without retraining, thus achieving both reliability and adaptability.
Solution Approach 2:
The patent handles camera parameter variability by transforming all camera-specific parameters into a unified coordinate system. Instead of adapting to different camera parameters, the system changes the representation to coordinate-based features that are invariant to camera parameters, allowing the network to generalize across different cameras and viewpoints through parameter transformation rather than parameter adaptation.
3Loss of information
If feature maps encode pixel absolute position using image coordinates, then spatial information is provided, but camera intrinsic parameters and distortion are not accounted for
Solution Approach 1:
The patent transforms the coordinate system from simple image coordinates to a coordinate system that incorporates camera intrinsic parameters and distortion corrections. By changing the parameter representation to account for lens distortion and camera geometry, the spatial measurements become more accurate while still maintaining a regular grid structure that is efficient for neural network processing.
Data Source
AI summary
The disclosure relates to system for processing an image of at least one camera. The camera has predetermined camera parameters including a lens distortion and a camera pose with respect to a predefined reference frame. The system comprises:a trained neural network with a predefined architecture, the neural network being configured to receive the image of the camera as input and to predict in response at least one characteristic, whereinthe neural network architecture comprises at least one static feature map configured to encode the predetermined camera parameters including the lens distortion and/or the camera pose.