3D Positional Encoding for Monocular Driving Perception
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Monocular cameras in automated driving systems face challenges in generating accurate 3D representations of the environment due to inherent limitations in depth perception, making it difficult to enhance perception functionality and make automated driving systems more accessible, cost-effective, and efficient.
Innovation Solution
A computer-implemented method that generates 2D position information for image data from a vehicle-mounted camera, using a multilayer perceptron to output 3D positional encodings, which are then fed into a transformer network to generate perception predictions in a 3D reference frame, incorporating extrinsic, intrinsic, and distortion parameters of the camera.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If monocular cameras are used to capture images, then cost and compactness are improved, but depth perception accuracy deteriorates
Solution Approach 1:
The patent transforms 2D image coordinates into 3D spatial coordinates by introducing a third dimension (depth) through neural network processing. The system embeds 2D positional information into 3D space using learned transformations, enabling depth estimation from monocular images without requiring additional depth sensors.
Solution Approach 2:
The patent introduces neural networks as an intermediary between the monocular camera and the perception system. The neural network processes 2D image data and generates 3D positional encodings, acting as a mediator that translates limited 2D information into comprehensive 3D spatial understanding.
2Measurement precision
If LiDAR is used to generate precise 3D maps, then depth perception accuracy is improved, but cost and system complexity increase
Solution Approach 1:
The patent creates a virtual 3D representation (copy) of the environment using only 2D image data processed through neural networks. Instead of using physical LiDAR sensors to directly measure 3D space, the system generates synthetic 3D positional information from 2D images, achieving similar functional outcomes at lower cost.
Solution Approach 2:
The patent replaces the mechanical LiDAR sensing system with a computational approach using neural networks. Instead of using physical light pulses and time-of-flight measurements, the system uses learned transformations on 2D image data to infer 3D spatial relationships.
3Measurement precision
If transformer networks are used for perception tasks with 3D output, then perception accuracy is improved, but computational complexity increases
Solution Approach 1:
The patent performs preliminary transformation of 2D positional information into 3D positional encodings before feeding data to the transformer network. This preprocessing step embeds spatial structure into the input data, reducing the computational burden on the transformer network during the main perception task.
Data Source
AI summary
A method for making perception predictions for a perception functionality in an automated driving system of a vehicle is disclosed. The method includes generating 2D position information of an image captured by a vehicle-mounted camera. The 2D position information indicates a position of each pixel out of a plurality of pixels of the image, or a position of each patch out of a plurality of patches of the image in the 2D reference frame of the image. Then, feeding the generated 2D position information, extrinsic parameters of the vehicle-mounted camera, intrinsic parameters of the vehicle-mounted camera, and distortion parameters of the vehicle-mounted camera to a multilayer perceptron which process the feed data and output 3D positional encodings. The method further includes feeding the image data and the 3D positional encodings to a transformer network for generating a prediction output in a 3D/2D reference frame of the vehicle.