Scene Simulator Training With Real and Simulated Multi-Agent Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Simulators face difficulties in training automated driving systems (ADS) due to limited datasets for complex agent interactions, volatile lighting, and edge cases, leading to incomplete or darkened scene renderings, which hinder performance in environments with multiple agents.
Innovation Solution
A learning system trains a scene simulator using diverse datasets from real and simulated agents, synthesizing observations to generate 3D scenes and employing reinforcement learning to improve model estimates, reducing design costs and enhancing realism.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If simulators use limited datasets for training, then training efficiency is improved, but scene rendering completeness deteriorates
Solution Approach 1:
The patent combines real-world sensor data from vehicles with simulated agent interaction data to create a comprehensive training dataset. This merging of data sources allows the simulator to maintain complete scene rendering information while achieving training efficiency through the structured integration of diverse data types.
Solution Approach 2:
The trained scene simulator is designed to serve multiple functions: it renders complete scenes for training ADS, generates synthetic training data, and supports various driving scenarios including edge cases. This multi-functionality allows the single simulator system to address multiple needs without requiring separate specialized systems for each function.
2Manufacturing precision
If simulators use diverse datasets including edge cases, then scene rendering quality is improved, but dataset acquisition complexity increases
Solution Approach 1:
The patent uses real vehicle sensor data as a template or copy of actual driving scenarios, then replicates and extends these scenarios through simulated agent interactions. This copying approach allows the system to generate diverse edge case data without requiring physical collection of every possible scenario, thereby improving rendering quality while managing acquisition complexity.
Solution Approach 2:
The system pre-collects real-world sensor data from vehicles before simulation training begins. This preliminary data collection establishes a foundation of authentic driving scenarios that can then be efficiently expanded through simulation, reducing the overall complexity of acquiring diverse edge case data during the training process.
3Reliability
If simulators train with realistic multi-agent interactions, then ADS performance is improved, but computational resources required increase
Solution Approach 1:
The patent segments the training process into distinct phases: first training with real sensor data, then progressively introducing simulated agent interactions. This segmentation allows the computational workload to be distributed and managed in stages, enabling realistic multi-agent training while controlling resource consumption through structured progression.
Solution Approach 2:
The system implements a progressive training approach where simulated agent interactions are introduced gradually rather than all at once. This partial action strategy allows the ADS to benefit from realistic multi-agent training improvements while managing computational resources by controlling the intensity and scope of simulated interactions at each training stage.
Data Source
AI summary
System, methods, and other embodiments described herein relate to training a scene simulator for rendering 2D scenes using data from real and simulated agents. In one embodiment, a method includes acquiring trajectories and three-dimensional (3D) views for multiple agents from observations of real vehicles. The method also includes generating a 3D scene having the multiple agents using the 3D views and information from simulated agents. The method also includes training a scene simulator to render scene projections using the 3D scene. The method also includes outputting a 2D scene having simulated observations for a driving scene using the scene simulator.