AI Model Validation via Reverse Data Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In artificial intelligence modeling, particularly deep learning, the similarity between training and test datasets can lead to false high model performance due to the lack of independence between these sets, resulting in unreliable model validation.
Innovation Solution
A computer-implemented method is introduced that generates a training dataset using reverse data generation from a trained AI model, compares it with a test dataset, and removes data with predefined relationships to create a new test dataset, ensuring the validity of the AI model by evaluating its performance on a more independent dataset.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the test dataset is used to evaluate the trained AI model, then the model performance can be measured, but the similarity between test dataset and training dataset leads to false high performance assessment
Solution Approach 1:
Instead of directly using the original test dataset to evaluate the model, the patent inverts the approach by first generating synthetic training data from the model using reverse data generation, then comparing this generated data with the original training data to identify and remove similar samples from the test dataset. This inversion ensures that the evaluation process itself creates the independent test set rather than relying on pre-existing potentially biased splits.
Solution Approach 2:
The patent creates a copy of the training process by using the trained model to generate synthetic training data. This copied training data is then used to identify samples in the test dataset that are too similar to the original training data, allowing removal of these samples to create a more independent evaluation set without losing valuable evaluation coverage.
2Reliability
If data from test dataset with predefined relationship to training dataset is removed, then data independence is improved, but the test dataset size is reduced
Solution Approach 1:
By generating synthetic training data copies through reverse data generation, the patent creates a reference set that mirrors the characteristics of the original training data. This allows systematic identification and removal of overlapping samples from the test dataset while preserving samples that are truly independent, thus maintaining optimal test set size with improved independence.
Solution Approach 2:
The generated synthetic training data acts as an intermediary between the original training data and the test dataset. It serves as a mediator to identify which test samples are too similar to training samples, enabling precise removal of only the problematic samples rather than arbitrary reduction of the test set.
Data Source
AI summary
Validity of a trained artificial intelligence model is verified. The verifying the validity includes generating a training dataset from the trained artificial intelligence model using reverse data generation of the trained artificial intelligence model. The training dataset generated using the reverse data generation is compared with a test dataset used to evaluate the trained artificial intelligence model. The comparing is to determine a relationship between the training dataset that was generated and the test dataset. Data from the test dataset determined to have a predefined relationship with the training dataset is removed to obtain a new test dataset. The new test dataset is used to verify the validity of the trained artificial intelligence model.