Synthetic Driving Image Generation Using 3D Point Cloud Conditions
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing diffusion models for generating driving images lack 3D information, resulting in unnatural images in terms of perspective, object orientation, and shape precision, while inpainting methods fail to provide adequate 3D expression.
Innovation Solution
A method and device that utilize a generative model, specifically a diffusion model, to generate images by combining a masked input image with a generation-condition point cloud derived from LiDAR data, which includes 3D information, to maintain accurate 3D object shapes during image generation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If a diffusion model is used to generate driving images, then the image generation capability is improved, but the 3D information accuracy deteriorates
Solution Approach 1:
The patent combines a diffusion model with a point cloud-based 3D representation system. The diffusion model generates 2D image content while the point cloud provides 3D spatial information, and these two representations are merged through projection and coordinate transformation to produce synthetic images that maintain both natural appearance and accurate 3D geometry.
Solution Approach 2:
The patent introduces a point cloud as an intermediary between the diffusion model and the final synthetic image. The point cloud serves as a mediator that carries 3D information from the real environment and translates it into a format that can guide the diffusion model's image generation process, ensuring 3D accuracy is preserved.
2Device complexity
If a semantic map is used as an image generation condition, then the simplicity of the condition is improved, but the perspective and object orientation accuracy deteriorates
Solution Approach 1:
The patent transitions from using 2D semantic maps to using 3D point clouds as generation conditions. By adding the third dimension (depth) to the representation, the system can accurately capture perspective, object orientation, and spatial relationships while maintaining relatively simple processing through standard point cloud operations.
3Shape
If inpainting is used to generate masked area images, then the natural appearance is improved, but the 3D expression capability deteriorates
Solution Approach 1:
The patent segments the image generation process into two independent but coordinated components: a diffusion model that handles 2D natural appearance generation and a point cloud system that handles 3D information preservation. This segmentation allows each component to specialize in its strength while working together to overcome the limitations of either approach alone.
Data Source
AI summary
A method and apparatus with synthetic image generation are provided. The method includes selecting, from a memory, a sample point cloud corresponding to an object; generating a generation-condition point cloud by combining the sample point cloud with a target point cloud; generating a masked input image by masking, in an input image, an object area corresponding to the sample point cloud; and generating an output image by inputting the masked input image and the generation-condition point cloud to a generative model which uses the generation-condition point cloud as a condition for generating the output image.