Robustness Evaluation for Data-Based Sensor Models
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data-based sensor models, particularly those based on neural networks, lack predictability and robustness against interferences, making them unsuitable for safety-sensitive systems like motor vehicle control, as they cannot guarantee output within certain ranges and struggle to evaluate robustness effectively.
Innovation Solution
A method to evaluate the robustness of trained data-based sensor models using unlabeled validation input datasets, applying two robustness criteria: a distance-based criterion for temporal shifts and an epsilon environment-based criterion, to determine the proportion of robust datasets, thereby ensuring continuous control and monitoring of sensor models in real-world operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If data-based sensor models (neural networks) are used to evaluate sensor signal time series, then change-point time detection capability is improved, but predictability and robustness against interferences deteriorate
Solution Approach 1:
The patent segments the evaluation process into multiple independent robustness criteria (temporal shift criterion, epsilon environment criterion, interference criterion). Each criterion evaluates a specific aspect of model robustness separately, allowing comprehensive assessment without compromising the overall reliability of the system.
Solution Approach 2:
The patent applies preliminary actions by introducing validation datasets with predefined interferences and perturbations before actual deployment. The model undergoes robustness evaluation using synthetic interferences (noise, outliers, temporal shifts) to pre-assess its reliability, ensuring it meets safety requirements before real-world application.
2Measurement precision
If data-based sensor models are used for evaluating sensor signal time series, then detection accuracy is improved, but comprehensibility and quality assessment capability deteriorate
Solution Approach 1:
The patent implements feedback mechanisms where the robustness evaluation results are fed back into the model deployment decision process. The quantified robustness metrics provide comprehensible feedback about model quality, enabling stakeholders to understand model reliability and make informed decisions about deployment in safety-sensitive applications.
3Reliability
If robustness evaluation with multiple criteria is performed, then reliability assessment is improved, but computational complexity and evaluation time increase
Solution Approach 1:
The patent allows for partial evaluation by enabling selective application of robustness criteria based on specific application requirements. Users can choose to evaluate only the most relevant criteria (e.g., only temporal shift criterion for time-sensitive applications), reducing computational complexity while maintaining essential reliability assessment.
Data Source
AI summary
A computer-implemented method determines a degree of robustness for a robustness of a provided, trained, data-based sensor model for evaluating an input dataset having at least one signal time series in order to determine a model output representing a change-point time. The method includes providing a plurality of unlabeled validation input datasets to the sensor model, and determining a plurality of robust validation input datasets of the plurality of unlabeled validation input datasets that satisfy a first robustness criterion and/or a second robustness criterion. The method further includes determining a proportion of the plurality of robust validation input datasets out of the plurality of unlabeled validation input datasets in order to obtain the degree of robustness.