Machine Learning Result Evaluation Using Synthetic Temporal Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Computer-based machine learning systems in safety-critical applications, such as autonomous driving, generate unreliable results when operating on different data sets than those used for training and validation, leading to unexpected outcomes due to the lack of real-time uncertainty value generation and temporal evolution consideration.
Innovation Solution
A method that involves generating synthetic elements based on real elements to emulate temporal scene evolution, processing these elements through the machine learning system, and comparing results to determine a confidence measure by evaluating the deviation and discrepancies between real and synthetic outcomes.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If machine learning systems operate on different data sets than training data, then real-time processing capability is maintained, but result reliability deteriorates due to unexpected outcomes
Solution Approach 1:
The patent creates synthetic data sets that copy and transform real data sets through mathematical operations (addition, multiplication, inversion). These synthetic data sets mimic the statistical properties and temporal evolution patterns of real data, allowing the system to maintain reliability when processing unseen real data while preserving real-time processing capability.
Solution Approach 2:
The patent transforms training data parameters by applying random linear transformations and temporal evolution models to generate synthetic data. This parameter transformation allows the system to adapt to different data distributions without retraining, maintaining both reliability and real-time processing speed.
2Speed
If uncertainty values are generated in real time, then system responsiveness is improved, but confidence in results deteriorates due to lack of temporal evolution consideration
Solution Approach 1:
The patent pre-computes temporal evolution models and uncertainty relationships during system initialization and training. These pre-computed models are then applied to real-time data without requiring complex real-time calculations, maintaining both responsiveness and confidence in results.
Solution Approach 2:
The patent implements a feedback mechanism where synthetic data sets are continuously refined and updated based on their relationship to real data sets. This feedback loop ensures that uncertainty values remain accurate and confident while the system maintains real-time processing capability.
3Measurement precision
If synthetic data sets are generated to evaluate confidence, then measurement precision of result reliability is improved, but device complexity increases
Solution Approach 1:
The patent designs a universal synthetic data generation framework that serves multiple functions: training, validation, and confidence measurement. The same mathematical transformations and temporal evolution models used to generate synthetic data also serve as the basis for confidence estimation, eliminating the need for separate complex evaluation systems.
Solution Approach 2:
The patent uses simple copying and transformation operations to create synthetic data sets that replicate the essential characteristics of real data. These operations (addition, multiplication, inversion) are computationally inexpensive compared to complex simulation or generation models, providing accurate confidence measurement without significantly increasing device complexity.
Data Source
AI summary
A computer-implemented method for evaluating results of a computer-based machine learning system. The method includes receiving a data set of real elements, wherein each real element of the data set of real elements corresponds to a real observation; receiving a first result that was calculated by the computer-based machine learning system using the data set of real elements; generating a data set of synthetic elements in relation to the data set of real elements; transmitting the data set of synthetic elements; receiving a second result calculated by the computer-based machine learning system using the data set of synthetic elements; comparing the first result to the second result; determining, based on the first result and/or comparing the first result to the second result, one or more validity values characterizing a confidence measure of the calculated first result.