Autonomous Vehicle Remote Assistance Interface for ML Navigation Constraints
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Autonomous vehicles (AVs) often encounter error events that require remote assistance, but the data provided by AVs to remote assistance operators is not intuitive and can be difficult for humans to understand, due to its non-visual and complex nature.
Innovation Solution
The system translates non-visual outputs from AI/ML models of AVs into visual data that can be easily understood by remote assistance operators, providing a remote assistance interface that allows operators to interact with the AV and overcome error events.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of information
If non-visual data from AI/ML models is provided to remote assistance operators, then the AV can provide detailed operational data for analysis, but the data becomes difficult for humans to understand and interpret
Solution Approach 1:
The patent introduces an intermediary translation system that converts non-visual AI/ML model outputs into visual representations. This intermediary layer translates complex numerical data, cost maps, and model predictions into visual formats such as annotated images, graphical overlays, and visualized cost surfaces that remote assistance operators can intuitively understand while preserving the complete operational information from the AV's AI/ML models.
Solution Approach 2:
The patent replaces the mechanical/abstract data presentation system with a visual information system. Instead of presenting raw numerical data and non-visual model outputs, the system substitutes these with visual representations including annotated sensor data, graphical cost maps, and visualized navigation constraints, making the information accessible to human operators who process visual information more effectively.
2Loss of information
If complex AI/ML model outputs are transmitted to remote assistance systems, then comprehensive navigation information is available, but the system complexity increases making integration difficult
Solution Approach 1:
The translation system serves as an intermediary that simplifies the integration process by handling the complex task of converting diverse AI/ML model outputs into standardized visual formats. This intermediary layer absorbs the complexity of different model types and data formats, presenting a unified visual interface to remote assistance operators without requiring complex integration logic in the remote assistance system itself.
Solution Approach 2:
The patent transforms the parameter representation from non-visual numerical data to visual parameters. AI/ML model outputs such as cost maps, navigation constraints, and predictions are converted into visual parameters that can be displayed graphically, changing the form of data representation while preserving the underlying information content and making it accessible to human operators.
3Ease of operation
If visual data is used to represent AI/ML outputs, then operator understanding improves, but data processing and translation complexity increases
Solution Approach 1:
The translation system is integrated into the AV's existing data processing architecture, allowing it to self-serve by utilizing the AV's own computational resources to perform the translation. The AV's computing system generates the visual representations from its AI/ML model outputs, reducing the translation complexity burden on external remote assistance systems and enabling the AV to prepare operator-friendly visual data autonomously.
Solution Approach 2:
The translation system is designed to handle multiple types of AI/ML model outputs universally, including cost maps, navigation constraints, predictions, and sensor data. By creating a multi-functional translation capability that can process diverse data types through a unified visual representation framework, the system reduces overall translation complexity compared to implementing separate translation mechanisms for each data type.
Data Source
AI summary
Systems and techniques are provided for translating outputs from machine learning (ML) models of an autonomous vehicle (AV) and generating a remote assistance (RA) interface based on the translated outputs. An example method can include receiving an output generated by an ML model of an AV, the output indicating a navigation action and/or a likelihood that the AV will implement the navigation action in a scene associated with the AV; in response to an error event experienced by the AV, establishing an RA session between the AV and an RA system; translating the output into a navigation constraint associated with the navigation action, the navigation constraint representing a condition deemed to restrict a behavior of the AV in the scene to the navigation action or trigger the AV to implement the navigation action; and generating an RA interface comprising user interface (UI) data representing the navigation constraint.