VR Pedestrian Interaction Data for Autonomous Vehicle Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for training autonomous vehicles lack sufficient data for complex and dangerous pedestrian-vehicle interaction scenarios, particularly due to the limitations of using keyboard controllers and VR headsets, which fail to accurately replicate real-world pedestrian behaviors.
Innovation Solution
A human-in-the-loop VR simulator, JaywalkerVR, is used to generate high-quality pedestrian-vehicle interaction data by tracking real human movements in virtual environments, incorporating yaw angles and other natural behaviors, to create the CARLA-VR dataset for training autonomous vehicles.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If keyboard controllers or joysticks are used to control pedestrian avatars in virtual environments, then data collection can be performed, but the control freedom is restricted and cannot accurately reproduce real walking behaviors
Solution Approach 1:
The patent replaces mechanical control devices (keyboard controllers, joysticks) with a VR headset-based body tracking system. The VR headset uses sensors to track head rotation, yaw angles, and body movements, automatically generating pedestrian behavior data without requiring manual mechanical control. This substitution enables natural, unrestricted pedestrian movements that accurately reflect real-world behavior patterns.
2Productivity
If public datasets are used for training, then training can be performed with available data, but the datasets lack complex and dangerous scenes needed for robust autonomous vehicle training
Solution Approach 1:
The patent performs preliminary action by proactively generating synthetic pedestrian-vehicle interaction data in virtual environments before actual autonomous vehicle deployment. The system pre-collections diverse scenarios including dangerous and long-tail cases (jaywalking, pedestrians walking along vehicles, close contacts) that are rare in public datasets, thereby preparing comprehensive training data in advance to improve model robustness.
Solution Approach 2:
The patent creates virtual copies of real-world pedestrian-vehicle interaction scenarios in a simulated environment. By rendering synthetic data that replicates complex interactions (pedestrians jaywalking, walking alongside vehicles, dangerous close contacts), the system generates diverse training samples that complement limited public datasets, enabling more comprehensive model training without requiring additional real-world data collection.
3Reliability
If structured data collection with human subjects is performed on public roads, then complex scenarios can be captured, but it poses safety risks to participants
Solution Approach 1:
The patent introduces a virtual environment as an intermediary between human subjects and real-world dangerous scenarios. Participants interact with simulated vehicles and pedestrians through a VR system, allowing collection of authentic behavioral data (head yaw angles, body movements, decision-making patterns) without exposing them to actual traffic hazards. This intermediary approach preserves data quality while eliminating safety risks.
Data Source
AI summary
A method and system for generating virtual pedestrian-vehicle interaction data includes generating a virtual reality environment in virtual reality device, generating a scenario in the virtual reality environment, the scenario comprising virtual vehicle movements, displaying the scenario in a virtual reality device, storing virtual reality movements relative to the scenario, the virtual reality movements comprising at least a yaw movement, communicating the virtual vehicle movements to a simulator controller, communicating the virtual vehicle movements to the simulator controller, associating the virtual reality movements, the virtual vehicle movements and the scenario to form pedestrian-vehicle data, and training an autonomous vehicle system using the pedestrian-vehicle data.