Deep Learning Model Verification with Generated Robustness Tests
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning models are sensitive to noise or distortions, leading to accuracy deterioration and unexpected performance failures due to nonintuitive characteristics and inherent blind spots, making them less transparent and challenging to deploy efficiently.
Innovation Solution
A method and system for verifying deep learning models by creating a replica environment, uploading reference datasets, and determining performance scores using generative deep learning models to analyze robustness against adversarial, correlated, fooling, outlier, and augmented datasets.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If deep learning models are deployed to handle laborious tasks with high computational power, then productivity and efficiency are improved, but the models become sensitive to noise or distortions causing accuracy deterioration
Solution Approach 1:
The patent applies preliminary action by creating a replica environment before actual deployment to verify data control flow and model behavior. This advance verification step allows identifying potential accuracy issues before the model handles real tasks, preventing future reliability problems while maintaining high productivity.
Solution Approach 2:
The patent creates a replica environment that copies the deployment parameters and data flow characteristics. This virtual copy allows testing and verification without affecting actual task processing, enabling accuracy validation while maintaining high productivity in the real system.
2Productivity
If deep learning models are trained through backpropagation to achieve human-level performance, then productivity is improved, but the models develop nonintuitive characteristics and blind spots reducing transparency
Solution Approach 1:
The patent introduces an intermediary verification layer between the trained model and its deployment. This replica environment acts as a mediator to observe and verify data control flow, making the nonintuitive characteristics and blind spots detectable while preserving the high productivity achieved through backpropagation training.
3Productivity
If deep learning models are integrated with day-to-day workloads to enhance efficiency, then productivity is improved, but unexpected failures occur due to connection with non-obvious nuances of data distribution
Solution Approach 1:
The patent performs preliminary verification of data control flow in a replica environment before integrating the model with actual workloads. This advance step identifies potential performance failures caused by data distribution nuances, ensuring reliable integration while maintaining high productivity.
Data Source
AI summary
A method and system of verifying deep learning models is disclosed. A computing device creates a replica environment based on one or more deployment parameters. Data control flow of the trained DL model is verified in the replica environment. A reference model performance score is determined based on processing of a reference dataset by the trained DL model. A plurality of test datasets is determined based on the reference dataset using a generative deep learning model. A test model performance score is determined based on processing of each of the plurality of test datasets by the trained DL model. Data performance of the trained DL model is verified based on a comparison of the test model performance score corresponding to each of the plurality of test datasets and the reference model performance score corresponding to the reference dataset.