ML Retraining Data Partitioning to Prevent Training-Test Leakage
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing supervised learning models become obsolete due to data drift, leading to inaccurate classification and performance decay when not retrained with updated data, and current retraining methods are computationally intensive and prone to data leaks.
Innovation Solution
A system and method for automatically constructing training and testing data sets that optimize the use of new data for retraining models, avoiding data leaks by maintaining separate partitions and using bipartite graphs to determine optimal data inclusion.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If newly received product data is randomly assigned to training or testing data, then data assignment is simple, but data leakage occurs and model effectiveness is reduced
Solution Approach 1:
The patent segments the data assignment process into distinct phases: first identifying the testing data set from newly received data, then identifying the training data set from remaining data. This segmentation prevents data leakage by ensuring that data used for testing is strictly separated from training data, while still allowing comprehensive use of new product information for model retraining.
Solution Approach 2:
The patent applies preliminary action by first determining the testing data set before determining the training data set. This preliminary identification of testing data ensures that no data intended for testing accidentally leaks into the training set, while still allowing all remaining new data to be utilized for training purposes.
2Reliability
If manual analysis is used to assign data to training or testing sets, then data leakage is avoided, but computational intensity increases and real-time operation is prevented
Solution Approach 1:
The patent implements self-service by enabling the system to automatically perform data assignment without requiring manual analysis. The computer system autonomously identifies testing data and training data from newly received product data using predefined criteria and algorithms, thereby maintaining data integrity while eliminating the computational burden and time constraints associated with manual processes.
Solution Approach 2:
The patent replaces the mechanical manual analysis process with an automated computer-based system that uses algorithmic methods to identify and assign data. This substitution maintains rigorous data integrity checks while dramatically improving computational efficiency and enabling real-time model retraining operations.
3Use of energy by moving object
If a classification model is not retrained with new data, then computational resources are saved, but model performance decays due to data drift
Solution Approach 1:
The patent applies partial action by retraining the classification model only with the identified training data set from newly received product data, rather than using all available data or requiring complete retraining from scratch. This selective approach maintains model accuracy by incorporating new product information while conserving computational resources through targeted updates.
Solution Approach 2:
The patent implements parameter changes by updating the model with new data that reflects current product information and consumer preferences. This allows the model to adapt to data drift by changing its learned parameters based on recent data, thereby maintaining accuracy without requiring complete retraining.
4Ease of manufacture
If outdated data is used to train the model, then training is simpler, but the model becomes ineffective for current product data
Solution Approach 1:
The patent implements dynamics by making the training data adaptive rather than static. The system dynamically identifies and selects the most relevant training data from newly received product data, ensuring that the model is trained on current, relevant information rather than outdated data. This dynamic approach maintains training simplicity while significantly improving model adaptability to current products and consumer preferences.
Data Source
AI summary
A computer-implemented method and system for optimally retraining a supervised machine learning model based on newly received data. The method comprises receiving, from a requestor device, a new data set for updating a previously-trained model generated using a first training data set and tested using a first testing data set. Then, the new data set is checked for components having an association to both the first training data set and the first testing data set; and where such components are found, they are deleted. Once all of the components of the new data have been examined, remaining components of the new data set are assigned to one of the first training or testing data set in dependence upon a relationship connectivity therewith to form at least one of an updated testing and training data set for building the updated model.