Neural Rendering Upsampling for Autonomous Driving Simulation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current image simulators for autonomous driving and other applications struggle to accurately replicate real-world environments, leading to discrepancies between virtual and real test results. Additionally, generating high-quality 3D objects for these simulators is time-consuming and costly.
Innovation Solution
A method involving an electronic device that generates an image by obtaining an image sequence of a target scene and information about a first viewing angle. The device then generates rays corresponding to image plane pixels, determines spatial points by sampling these rays, and uses neural networks to render images. This process includes upsampling a rendered image based on a reference image to improve resolution and quality.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If data is collected while a vehicle is moving at significant speed, then data collection efficiency is improved, but the viewing angle is limited and image quality at new angles deteriorates
Solution Approach 1:
The system performs preliminary actions by collecting images from multiple viewing angles before the vehicle moves at high speed. These pre-collected images are stored and later used as reference data to generate images at new viewing angles through neural rendering, avoiding the need to collect data at limited angles during high-speed movement
Solution Approach 2:
The system creates virtual copies of the real-world environment by training a neural network model on pre-collected images from multiple angles. The trained model can then generate synthetic images at any desired viewing angle, effectively copying the visual appearance of the real scene without requiring physical data collection at those specific angles
2Adaptability or versatility
If traditional simulators are used to simulate real environments, then simulation capability is improved, but the accuracy of reproducing real-world scenarios deteriorates
Solution Approach 1:
The system replaces traditional mechanical 3D modeling and rendering approaches with a neural network-based rendering system. Instead of manually creating and manipulating 3D models, the neural network learns the visual appearance of the real environment from images and generates realistic synthetic images, achieving both versatility and accuracy
Solution Approach 2:
The system changes the fundamental parameter of how scenes are represented - from explicit 3D geometric models to implicit neural network representations. This allows the system to capture complex real-world visual details that are difficult to represent with traditional 3D models while maintaining the ability to generate images from various viewing angles
3Manufacturing precision
If high-quality 3D objects are generated for simulators, then rendering quality is improved, but time investment and cost increase
Solution Approach 1:
Instead of manually creating high-quality 3D models through time-consuming processes, the system copies visual information directly from real-world images. The neural network learns to reproduce the appearance, lighting, and textures of real objects by training on images, automatically generating high-quality rendering results without manual 3D modeling effort
Solution Approach 2:
The system performs self-service by automatically training the neural network model on provided images and generating synthetic rendered images without requiring manual intervention for 3D model creation. The entire process from input images to output synthetic images is automated, eliminating the need for expert 3D artists and significantly reducing time investment
Data Source
AI summary
A method of generating an image includes obtaining an image sequence of a target scene and information about a first viewing angle, generating a plurality of rays corresponding to a plurality of pixels of an image plane of the first viewing angle of the target scene, determining a plurality of spatial points by sampling the plurality of rays, generating a first rendered image of the image plane by rendering the plurality of spatial points using a first neural network, determining a reference image of the first rendered image from among a plurality of images of the image sequence, and generating a second rendered image having a second resolution by upsampling the first rendered image using a second neural network and based on the reference image. The plurality of images being captured from the first viewing angle and having a first resolution, and the first rendered image having the first resolution.