Moving-Average Model Selection for Domain Generalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Machine learning models struggle with domain generalization, exhibiting unreliable performance on distribution-shifted test domains and high stochasticity, making deployment in real-world settings unreliable.
Innovation Solution
A training mechanism that maintains a moving average of model parameters during training, starting after a threshold of iterations, and selects optimal model states for ensembling, generating an ensembled machine learning model that improves performance across different domains.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If machine learning models are trained on training data from a specific domain, then they achieve good performance on data from the same domain, but they exhibit unreliable performance and high stochasticity on distribution-shifted test domains
Solution Approach 1:
The patent segments the training process into multiple independent training runs with different initializations and hyperparameter settings. Each run produces a separate model state, and the ensemble combines predictions from multiple such states. This segmentation allows the system to capture diverse solution paths and reduce stochasticity when deploying to distribution-shifted domains.
Solution Approach 2:
The patent merges multiple model states from different training runs into an ensemble predictor. The ensemble combines predictions from individual models through averaging or voting mechanisms. This merging strategy leverages the diversity of different training trajectories to improve reliability and reduce variance on out-of-domain data while maintaining adaptability across domains.
2Measurement precision
If standard training procedures are used without domain generalization techniques, then the training process is simple and fast, but the model selection becomes unreliable due to high stochasticity
Solution Approach 1:
The patent performs preliminary actions by conducting multiple training runs with different initializations and hyperparameter configurations before final model selection. This preliminary exploration of the solution space allows the system to identify stable model states that generalize well, improving measurement precision in model selection. The complexity introduced is managed through automated ensemble construction and selection criteria.
Solution Approach 2:
The patent implements feedback mechanisms by evaluating model performance on validation sets and using this information to select the best ensemble configurations. The selection process provides feedback on which model states and ensemble sizes yield the most reliable predictions, allowing the system to adapt to the specific characteristics of the training data and task while reducing stochasticity in model selection.
Data Source
AI summary
A computing device may perform training of a set of machine learning models on a first data set associated with a first domain. In some examples, the training may include, for each machine learning model of the set of machine learning models, inputting, as values for a set of parameters of the respective sets of parameters and for an iteration of a set of iterations, a moving average of the set of parameters calculated over a threshold number of previous iterations. The computing device may select a set of model states that are generated during the training of the plurality of machine learning models based on a validation performance of the set of model states performed during the training. The computing device may then generate an ensembled machine learning model by aggregating the set of machine learning models corresponding to the set of selected model states.