Proxy Model Detection for Generative Training Data Leakage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing membership inference techniques for generative machine learning models have limitations, including requiring knowledge of internal components and low average accuracies, making it difficult to accurately detect training data usage and prevent data leakage.
Innovation Solution
The system compares test samples to generative model samples using an epsilon threshold distance, determining if they are within an epsilon environment, and employs methods like Monte Carlo integration and Kernel Density Estimation to calculate the probability of test samples being part of the training data, sending alerts if the probability exceeds a threshold.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If existing membership inference techniques are used to detect training data usage, then the detection process can be performed, but the accuracy is low and internal model components are required
Solution Approach 1:
The patent extracts the detection capability from the internal model structure by using independently trained proxy models. Instead of requiring access to the target model's internal components, the system uses external proxy models that replicate the target model's behavior patterns, thereby separating the detection function from the model's internal architecture.
Solution Approach 2:
The patent introduces proxy models as intermediary entities between the detector and the target model. These proxy models serve as mediators that capture the essential characteristics of the target model without requiring direct access to its internal components, enabling accurate detection while maintaining independence from the target model's structure.
2Measurement precision
If existing membership inference techniques are used, then training data detection can be performed, but the average accuracy remains low
Solution Approach 1:
The patent creates proxy models that are universally applicable to detect training data across different target models. The proxy models are trained on diverse datasets and can be applied to various generative models without requiring model-specific adjustments, thereby improving both accuracy and reliability through generalizable detection patterns.
Solution Approach 2:
The patent performs preliminary training of proxy models on extensive datasets before actual detection. This preliminary action involves training the proxy models to recognize training data patterns across multiple domains and model types, which significantly improves their accuracy and reliability when deployed for actual detection tasks.
Data Source
AI summary
Various examples are directed to systems and methods for detecting training data for a generative model. A computer system may access generative model sample data and a first test sample. The computer system may determine whether a first generative model sample of the plurality of generative model samples is within a threshold distance of the first test sample and whether a second generative model sample of the plurality of generative model samples is within the threshold distance of the first test sample. The computer system may determine that a probability that the generative model was trained with the first test sample is greater than or equal to a threshold probability based at least in part on whether the first generative model sample is within the threshold distance of the first test sample, the determining also based at least in part on whether the second generative model sample is within the threshold distance of the first test sample.