Synthetic Training Data Realism Checking With Confidence Intervals
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for assessing the realism of synthetic training data for machine learning models, such as the FID score, require a large number of data points and are not efficient in detecting deviations from real data, potentially leading to biased models.
Innovation Solution
A method using confidence intervals to compare synthetic and real data, determining upper and lower limits through repeated training and inference, and detecting systematic deviations to assess the realism of synthetic training data.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the FID score method is used to assess the realism of synthetic training data, then the assessment correlates well with subjective human assessment of realism, but a large number of data points (order of 10,000) are required for calculation
Solution Approach 1:
The patent extracts only the essential statistical features (mean and variance) from the synthetic training data that are most relevant for detecting systematic deviations from real data distributions. By focusing on these key statistical parameters rather than analyzing all 10,000+ data points, the method achieves realistic assessment with significantly reduced computational requirements
Solution Approach 2:
The patent introduces confidence intervals as an intermediary statistical tool to bridge the comparison between synthetic and real data. By calculating confidence intervals for statistical features and checking for systematic deviations, the method provides a rigorous statistical framework that maintains assessment accuracy while requiring fewer data points than direct FID score computation
2Reliability
If existing assessment methods are used to detect deviations from real data, then some level of deviation detection is achieved, but the detection efficiency is insufficient and may lead to biased models
Solution Approach 1:
The patent performs preliminary statistical analysis by calculating confidence intervals for the mean and variance of synthetic training data before model training. This preliminary assessment of whether statistical features fall within expected ranges allows early detection of systematic deviations, preventing biased model training while maintaining high reliability
Solution Approach 2:
The patent implements a feedback mechanism where confidence interval results from statistical feature analysis feed into the model training decision process. If systematic deviations are detected (statistical features outside confidence intervals), the process provides feedback to reject or adjust the synthetic data, ensuring only realistic data trains the model
Data Source
AI summary
A method for evaluating a degree of realism of synthetic training data for a machine learning model includes (i) providing the synthetic training data, wherein the synthetic training data is described by a statistical quantity, wherein the synthetic training data simulates sensor data, (ii) determining an upper limit of a confidence interval for the statistical value on the basis of the synthetic training data as part of a training of the machine learning model, (iii) providing real data, the real data also being described by the statistical variable, the real data comprising sensor data, the sensor data resulting from the detection of at least one sensor, (iv) determining a lower limit of the confidence interval for the statistical value on the basis of the real data in the context of an inference of the machine learning model, the lower limit being determined continuously from the start of the inference, and (v) checking the degree of realism of the synthetic training data on the basis of a comparison of the continuously determined lower limit with the determined upper limit, wherein a systematic deviation of the synthetic training data from the real data is detected. A computer program, a device, and a storage medium for this purpose is also disclosed.