Remote Driving Awareness Using Audio-Visual Object Detection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Teleoperated remote driving systems face challenges in providing adequate environment awareness to operators, limiting their ability to make safe driving decisions due to the complexity of combining data from various sensors, which can result in overlooked objects or unsafe driving behaviors.
Innovation Solution
The implementation of audio and vision-based systems that process data from multiple sensors to identify relevant objects, providing visual and audio notifications to the teleoperator, along with safety tunnel parameters to ensure safe driving behaviors, by using sensors like audio sensors, imaging devices, and position sensors to enhance awareness and automate safe driving actions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If multiple sensors are combined to provide comprehensive environment awareness, then the completeness of environmental information is improved, but the system complexity increases making it difficult to process and analyze the data
Solution Approach 1:
The patent introduces an intermediary system that processes sensor data before presenting it to the teleoperator. This intermediary layer automatically filters, prioritizes, and presents only relevant environmental information through visual and audio notifications, reducing the complexity of raw sensor data while maintaining comprehensive environmental awareness.
Solution Approach 2:
The patent replaces manual analysis of complex sensor data with automated computer vision and audio processing systems. These systems automatically identify, track, and prioritize relevant objects and events, substituting the need for complex human cognitive processing of multi-sensor data streams.
2Loss of information
If all sensor data is presented to the teleoperator, then complete environment awareness is achieved, but the operator becomes overwhelmed and may overlook critical objects
Solution Approach 1:
The patent applies local quality by providing different types of information presentation based on the specific object or event. Critical objects receive prominent visual and audio notifications, while less important elements are presented with lower priority. This differentiated presentation approach maintains complete environmental awareness while managing operator attention effectively.
Solution Approach 2:
The system provides feedback to the teleoperator through visual and audio notifications that highlight relevant objects and events. This feedback mechanism guides the operator's attention to critical elements in the environment without requiring them to process all raw sensor data, thus maintaining awareness while reducing cognitive load.
3Reliability
If automated safety systems are implemented to ensure safe driving behaviors, then driving safety is improved, but the system complexity and computational requirements increase
Solution Approach 1:
The patent implements preliminary action by pre-defining safety tunnel parameters and automated safety protocols before driving operations begin. These pre-established safety rules and parameters enable the system to automatically enforce safe driving behaviors without requiring complex real-time decision-making algorithms, thus improving safety while managing system complexity.
Data Source
AI summary
Systems and methods to increase environment awareness in remote driving applications may include a vehicle having an imaging device and a teleoperator station in communication with each other via a network. For example, audio data may be received from the vehicle and processed to identify known sounds associated with unseen objects in the environment. In addition, imaging data may be received from the vehicle and processed to identify known but unheard objects in the environment. Based on the identified sounds and/or objects, visualizations of the objects may be generated and presented to a teleoperator, and sounds associated with the objects may be amplified, synthesized, and/or emitted to the teleoperator to increase environment awareness.