Domain Generalisation Model Selection via Shifted Validation Splits
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning models struggle with domain generalization, as hyperparameter choices based on in-distribution performance are often sub-optimal for out-of-distribution test domains, and current methods for hyperparameter optimization, such as using OOD validation sets, are still inadequate.
Innovation Solution
A method involving generating multiple machine learning models with different hyperparameters, architectures, or initializations, and partitioning training and validation data sets using kernel k-means clustering to maximize domain shift, allowing for selection of the model with the highest accuracy without additional metadata.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If hyperparameter choices are based on in-distribution performance, then training accuracy is improved, but out-of-distribution test accuracy deteriorates
Solution Approach 1:
The patent applies preliminary action by creating a validation set that is deliberately constructed to represent out-of-distribution conditions before model evaluation.通过使用kernel k-means聚类算法最大化训练集和验证集之间的域偏移，在模型评估之前就预先准备好能够反映真实测试条件的验证环境，从而使hyperparameter选择能够同时考虑训练性能和泛化性能
Solution Approach 2:
The patent introduces an intermediary validation set that acts as a bridge between training data and test data. This validation set is constructed to have domain characteristics similar to the test domain but is distinct from both training and test sets, serving as an intermediate evaluation environment that better predicts test performance
2Adaptability or versatility
If traditional OOD validation sets are used for hyperparameter optimization, then model selection is improved, but accuracy performance remains sub-optimal
Solution Approach 1:
The patent applies parameter changes by fundamentally altering how the validation set is constructed. Instead of using traditional random sampling or simple domain-based splitting, the patent uses kernel k-means clustering to maximize domain shift between training and validation sets, thereby creating a validation set that better reflects out-of-distribution conditions and leads to superior accuracy performance
Data Source
Figure 1
Figure 2
Figure 3
AI summary
There is described a computer implemented method of training a machine learning model for domain generalisation. The method comprises: generating (202) a plurality of machine learning models which are different to each other; generating (204) a training data set and validation data set by partitioning a source data set using kernel k-means clustering to maximally domain shift the training set with respect to the validation set; training (206) and validating (208) the plurality of machine learning models; and selecting (210) the best performing model.