Natural-Language Sensor Queries for Rare Automated Driving Scenarios
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing automated driving systems face challenges in effectively training and evaluating rare and dynamic scenarios due to the long tail problem, requiring extensive human labor for data annotation and trigger design, which is costly and inefficient.
Innovation Solution
A query system using sensor data embeddings and query embeddings in a multi-dimensional space to identify and capture dynamic scenarios through natural language prompts, enabling efficient data collection and evaluation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If human annotations or handcrafted triggers are used to identify rare scenarios, then the quality and precision of scenario identification is improved, but the cost and time required increases significantly
Solution Approach 1:
The system enables self-service by allowing the automated driving system to automatically identify and tag rare scenarios using its own sensor data and neural network predictions, without requiring external human annotators. The system queries its own historical data to find scenarios matching desired criteria, making the annotation process autonomous
Solution Approach 2:
The patent introduces an intermediary query system that acts as a mediator between human developers and the vast dataset. This query system with natural language processing capabilities allows developers to specify scenario requirements without manually annotating data, translating human intent into automated data retrieval
2Reliability
If human engineers design and verify triggers for each scenario, then the reliability of scenario capture is improved, but the complexity and labor cost increases
Solution Approach 1:
The patent replaces the mechanical process of manual trigger design and verification with an automated information processing system. Neural networks and query processing algorithms substitute for human engineer analysis, automatically identifying scenarios based on sensor data patterns without requiring manual trigger engineering
Solution Approach 2:
The system creates simplified copies or representations of complex scenarios through query embeddings that capture essential scenario characteristics. These query representations serve as substitutes for detailed manual trigger designs, enabling scenario identification through pattern matching rather than explicit rule engineering
3Quantity of substance
If millions of hours of driving data are used for training, then the quantity of training data is improved, but the neural network overfits to nominal cases rather than rare scenarios
Solution Approach 1:
The patent extracts rare scenarios from the vast dataset of million-hour driving data. Instead of using all available data equally, the query system identifies and extracts only the relevant rare tail scenarios that match specified criteria, creating a focused training subset that prevents overfitting to common nominal cases while maintaining sufficient sample size for rare events
Data Source
Figure 1
Figure 2
Figure 3
AI summary
The present disclosure relates, among other things, to computer-implemented method (100) comprising: obtaining (5102) a sequence of sensor data comprising sensor data captured by one or more sensors of a vehicle for a plurality of time instances; generating (S104) a set of sensor data embeddings for the sequence of sensor data, wherein each sensor data embedding of the set of sensor data embeddings is generated by processing sensor data for one or more time instances of the plurality of time instances, through one or more sensor data embedding networks having been trained to process sensor data and to output corresponding sensor data embeddings for the sensor data in a multi-dimensional space; obtaining (S106) two or more query embeddings, the two or more query embeddings having been generated by processing a query through one or more query embedding networks trained to process queries and to output corresponding one or more query embeddings for each query in the multi-dimensional space, and wherein each of the one or more query embedding networks has been trained in association with at least one of the one or more sensor data embedding networks so as to relate to the same multi-dimensional space; obtaining (S108) a logical statement associated with the query for which the two or more query embeddings have been generated; and in response to each of the two or more query embeddings matching a respective sensor data embedding of the set of sensor data embeddings, and the sensor data, corresponding to said matching sensor data embeddings, fulfilling the logical statement, performing (5110) an action of a set of defined actions.