Multi-Task Network Training With Mixed Real and Virtual Batches
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training multi-task neural networks, such as those performing depth prediction and semantic segmentation from monocular images, faces challenges due to the difficulty in acquiring labeled training data, especially when manual annotation is required for certain tasks, and relying solely on virtual data does not provide the same level of training as real data.
Innovation Solution
A mixed-batch training approach that combines real and virtual data, where real data includes pairs of monocular images and virtual data includes synthetic images with ground-truth labels, allowing the network to generate separate outputs for real and virtual losses, which are then combined to update the network, thereby leveraging the strengths of both data types.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If manual annotation is used for supervised training, then training accuracy for tasks like semantic segmentation improves, but the complexity and time required for data preparation increases significantly
Solution Approach 1:
The patent creates virtual copies of real-world scenes using 3D models and rendering engines. These synthetic images serve as training data with automatically generated ground truth labels, eliminating the need for manual annotation while providing sufficient training accuracy for multi-task learning
Solution Approach 2:
The system generates its own training data through automated rendering pipelines that create synthetic images with inherent ground truth information. This self-service approach to data generation removes the dependency on external manual labeling processes
2Productivity
If only virtual data is used for training, then data acquisition becomes easier and faster, but the training performance and generalization capability deteriorates
Solution Approach 1:
The patent merges virtual and real data into a unified training pipeline. Synthetic images provide efficient data generation with ground truth labels, while real images provide authentic visual characteristics. The mixed-batch training combines both sources to achieve high training performance and efficient data acquisition
Solution Approach 2:
The training dataset is constructed as a composite of virtual and real data, leveraging the strengths of each type. The composite approach allows the model to learn from the structured precision of synthetic data while adapting to the natural variations present in real-world images
3Measurement precision
If multiple tasks are trained simultaneously with separate labeled data, then each task can achieve optimal performance, but the overall system complexity and data requirements increase
Solution Approach 1:
The patent creates a universal training framework where a single multi-task network processes multiple tasks simultaneously. The mixed-batch training approach provides unified supervision signals that work across different tasks (depth prediction, semantic segmentation, etc.), reducing the need for separate training pipelines for each task
Data Source
AI summary
Systems, methods, and other embodiments described herein relate to training a multi-task network using real and virtual data. In one embodiment, a method includes acquiring training data that includes real data and virtual data for training a multi-task network that performs at least depth prediction and semantic segmentation. The method includes generating a first output from the multi-task network using the real data and second output from the multi-task network using the virtual data. The method includes generating a mixed loss by analyzing the first output to produce a real loss and the second output to produce a virtual loss. The method includes updating the multi-task network using the mixed loss.