Virtual Vehicle Environment Generation Using Camera-LiDAR Fusion

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Current methods for creating virtual vehicle environments for testing highly automated driving functions are labor-intensive and costly, requiring significant personnel effort for manual configuration and object importation.

Innovation Solution

A computer-implemented method using pre-acquired camera image data and LiDAR point cloud data, combined with machine learning algorithms for pixel-based classification and projection, enables the creation of a virtual vehicle environment by classifying and integrating synthetically generated objects into a 3D representation, reducing manual effort and costs.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Ease of manufacture

If manual configuration and import of objects stored in an object library is used for scene building, then the virtual vehicle environment can be created with controlled object quality, but the personnel effort and cost increase significantly

Engineering Contradiction:

Improveease of creating virtual vehicle environmentVSAvoidpersonnel effort and cost

Core Design Contradiction:

Ease of manufactureVSProductivity

Solution Approach 1:

The patent uses real-world camera images and LiDAR point cloud data as templates to automatically generate virtual environment objects. Instead of manually configuring each object from scratch, the system copies geometric and semantic information from real sensor data to create accurate 3D representations, thereby reducing manual effort while maintaining object quality

Inventive Principle:

Principle #26Copying

Solution Approach 2:

The patent replaces the manual mechanical process of object configuration and import with an automated computer vision system. Machine learning algorithms automatically classify pixels in camera images, match them with LiDAR points, and generate 3D objects without human intervention, substituting manual labor with automated processing

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Measurement precision

If pixel-based classification of camera image data using machine learning algorithms is performed, then object classification accuracy improves, but computational complexity and processing time increase

Engineering Contradiction:

Improveobject classification accuracyVSAvoidcomputational complexity

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The patent combines camera image data with LiDAR point cloud data to perform pixel-based classification. By merging information from both sensors, the system achieves more accurate object classification than using camera data alone, as the LiDAR provides complementary geometric and depth information that enhances the classification process

Inventive Principle:

Principle #5Merging (Combining)

Solution Approach 2:

The patent transitions from 2D image classification to 3D point cloud classification by projecting classified pixels onto LiDAR points. This dimensional enhancement allows the system to leverage three-dimensional spatial information for more accurate object identification and reduces ambiguity in classification that exists in 2D images alone

Inventive Principle:

Principle #17Another dimension (Dimensionality change)

3Loss of information

If projection of pixel-based classified camera image data onto LiDAR point cloud data is performed, then the three-dimensional representation is enriched with class and color information, but the data processing complexity increases

Engineering Contradiction:

Improveinformation enrichment of point cloudVSAvoiddata processing complexity

Core Design Contradiction:

Loss of informationVSDevice complexity

Solution Approach 1:

The patent uses image coordinates as an intermediary to match classified pixels with corresponding LiDAR points. By projecting pixels onto the point cloud based on coordinate correspondence, the system efficiently transfers semantic and color information from the 2D image to the 3D point cloud without requiring complex registration or alignment algorithms

Inventive Principle:

Principle #24Intermediary (Mediator)

4Measurement precision

If instance segmentation of classified LiDAR point cloud data is performed for determining real objects, then object detection precision improves, but processing time increases

Engineering Contradiction:

Improveobject detection precisionVSAvoidprocessing time

Core Design Contradiction:

Measurement precisionVSLoss of time

Solution Approach 1:

The patent performs pixel-based classification of camera image data before projecting onto the LiDAR point cloud. This preliminary classification provides pre-processed semantic information that guides the subsequent instance segmentation process, allowing the system to focus computational resources on relevant regions and reduce overall processing time while maintaining high detection precision

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS20230415755A1Computer-implemented method and system for creating a virtual environment for a vehicle

Publication Date: 2023.12.28 DSPACE DIGITAL SIGNAL PROCESSING & CONTROL ENGINEERING GMBH

AI summary

A computer-implemented method and system for generating a virtual environment for a vehicle for testing highly automated driving functions of a motor vehicle. The method comprises projecting the pixel-based classified camera image data onto the pre-acquired LiDAR point cloud data, wherein each point of the LiDAR point cloud, superimposed by classified pixels of the camera image data, in particular having the same image coordinates, is assigned an identical class and an instance segmentation of the classified LiDAR point cloud data for determining at least one real object comprised by a class. A computer program and a computer-readable data carrier are also provided.