ADAS Performance Assessment Using Scenario-Based Objective Metrics
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The lack of a consistent metric for evaluating the performance of advanced driver assistance systems (ADAS) and automated driving systems (ADS) makes it difficult to objectively compare and improve their performance across different configurations and scenarios, as key performance indicators (KPIs) are context-dependent and often specific to individual systems.
Innovation Solution
A method and system that generate assessment indicators for ADAS/ADS systems by identifying real-world and simulated scenarios, using scenario parameters to calculate objective metrics, allowing for the comparison and improvement of performance across variously configured systems through KPI plots and AI-based computational methods.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If KPIs are calculated for variously configured ADAS/ADS systems, then performance evaluation is enabled, but comparability of KPIs deteriorates due to context-dependency and system-specific configurations
Solution Approach 1:
The patent transforms subjective KPI values into objective metrics by changing the parameter representation from system-specific numeric values to standardized scores based on percentile rankings. This allows performance evaluation across different system configurations while maintaining comparability through the standardized scoring mechanism.
Solution Approach 2:
The patent introduces scenario parameters and percentile-based scoring as intermediary elements between the raw KPI data and the final performance assessment. These intermediaries normalize the data from different system configurations, enabling objective comparison while preserving the adaptability to handle various system types.
2Measurement precision
If system-specific KPIs are used for each ADAS/ADS configuration, then performance assessment for individual systems is accurate, but objective comparison across different systems becomes difficult
Solution Approach 1:
The patent changes the parameter representation from absolute KPI values to relative percentile scores. This transformation maintains the accuracy of individual system assessment while enabling reliable comparison across different systems, as the percentile ranking contextualizes each system's performance relative to others in the same scenario.
Solution Approach 2:
The patent creates equipotential conditions for comparison by standardizing all KPIs to a common scoring scale (1-100) based on percentile rankings. This allows systems with different configurations to be compared on equal footing, as each system is evaluated against the same reference distribution of performance values.
3Measurement precision
If comprehensive scenario parameters are collected for objective assessment, then assessment accuracy is improved, but data processing complexity and computational requirements increase
Solution Approach 1:
The patent extracts only the essential scenario parameters needed for performance assessment from the comprehensive set of available data. By selecting and focusing on the most relevant parameters (such as those affecting safety, comfort, and efficiency), the system achieves objective assessment without requiring processing of all possible data elements, thus reducing computational complexity.
Solution Approach 2:
The patent segments the assessment process into distinct modules: scenario identification, KPI calculation, percentile ranking, and scoring. This segmentation allows each component to handle specific tasks independently, reducing the overall computational burden while maintaining comprehensive assessment capability through the coordinated operation of specialized sub-systems.
Data Source
AI summary
A method for objective assessment of performance of an ADAS/ADS system (210) of a vehicle (200) for a defined driving task in at least one selected scenario (SZi). The method includes identifying real-world scenarios (SZri) from data captured in real-time by sensors (220) while traveling on a test path with the vehicle (200) or from stored data; generating simulated scenarios (SZsi) from a simulation module (400). The method continues by calculating an assessment indicator (570) for at least one real-world scenario (SZri) and/or an assessment indicator (570) for at least one simulated scenario (SZsci) from an assessment module (500). The assessment indicator (570) represents the performance of the ADAS/ADS system (210) for the defined driving task. The method then includes generating (S50) evaluation results (750) from an evaluation module (700).