GAN-Based Virtual Video Translation for Credible CNN Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The high cost of annotating training images for autonomous driving using Convolutional Neural Networks (CNNs) and the lower credibility of models trained in virtual worlds due to differences between virtual and real-world images.
Innovation Solution
A learning method utilizing a generative adversarial network to transform virtual data into more real-looking data, maintaining consistent frame relationships, with a generating CNN and a discriminating CNN to reduce annotation costs and improve model credibility.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If training images are acquired automatically in a virtual world, then annotation cost is reduced, but the credibility of trained CNNs deteriorates due to differences from real-world images
Solution Approach 1:
A generative adversarial network (GAN) is introduced as an intermediary between virtual and real worlds. The GAN transforms virtual training images into synthesized images that bridge the gap between virtual and real-world appearances, allowing CNNs to be trained on low-cost virtual data while maintaining credibility through realistic image synthesis
Solution Approach 2:
The GAN modifies multiple parameters of virtual images including texture details, lighting conditions, color distributions, and noise characteristics to transform them into real-world-like images. This parameter transformation enables virtual images to maintain their structural advantages while acquiring the visual characteristics of real images
2Quantity of substance
If virtual driving is used to acquire training images, then cost is reduced, but the training images differ from real-world images resulting in lower credibility
Solution Approach 1:
The GAN creates high-fidelity copies of real-world image characteristics by learning from real image datasets. These synthesized copies preserve the essential visual properties of real images while being generated from virtual data, enabling unlimited production of credible training images without additional real-world data collection
3Productivity
If CNNs are trained on virtual images, then annotation cost is reduced, but the models show lower performance on real-world data
Solution Approach 1:
The GAN serves as a translation layer that converts virtual images into a visual language familiar to real-world CNN models. This intermediary transformation allows models trained on virtual data to generalize better to real-world scenarios by presenting data in a realistic visual format
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A learning method for transforming a virtual video on a virtual world to a more real-looking video is provided. And the method includes steps of: (a) a learning device instructing a generating CNN to apply a convolutional operation to an N-th virtual training image, N-th meta data and (N-K)-th reference information to generate an N-th feature map; (b) the learning device instructing the generating CNN to apply a deconvolutional operation to the N-th feature map to generate an N-th transformed image; (c) the learning device instructing a discriminating CNN to apply a discriminating CNN operation to the N-th transformed image to generate a category score vector; (d) the learning device instructing the generating CNN to generate a generating CNN loss by referring to the category score vector and its corresponding GT, and to perform backpropagation by referring to the generating CNN loss to learn parameters of the generating CNN.