Neural Network Minibatch Sampling for Better Domain Generalization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing deep learning techniques for neural network training suffer from poor domain generalization due to noisy distance estimates in minibatches, leading to underperformance in real-world applications, particularly in bioacoustic monitoring tools deployed in new environments.
Innovation Solution
A method for training neural networks that involves generating minibatches by determining dissimilarities in feature embeddings and selecting instances probabilistically based on these dissimilarities, using techniques like k-means clustering or determinantal point processes to ensure diversity and representation, followed by updating the network.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If standard random sampling is used to generate minibatches, then the training process is simple and fast, but the distance estimates become noisy and domain generalization performance deteriorates
Solution Approach 1:
The method pre-computes feature embeddings for all training instances before minibatch sampling. This preliminary action creates a stable reference framework that enables dissimilarity-based sampling without adding complexity during the training loop, improving distance estimate quality while maintaining training efficiency
Solution Approach 2:
The patent introduces feature embeddings as an intermediary representation layer between raw data and the training process. By sampling based on dissimilarities in this embedding space rather than raw data space, the method achieves better distance estimates without directly complicating the sampling mechanism itself
2Adaptability or versatility
If invariance regularisation is applied with small minibatches, then domain adaptation is attempted, but the noisy distance estimates result in negligible or negative impact on training
Solution Approach 1:
The method changes the sampling parameter distribution by using dissimilarity-based probabilistic sampling instead of uniform random sampling. This parameter change in the sampling strategy ensures that minibatches contain more diverse and representative samples, stabilizing the distance estimates used in invariance regularisation and improving domain generalization reliability
Solution Approach 2:
The patent incorporates feedback by using the computed dissimilarities in feature embedding space to guide the sampling process. This feedback mechanism ensures that minibatches are constructed with explicit consideration of domain diversity, making the invariance regularisation more effective and training more stable across domains
3Adaptability or versatility
If vanilla empirical risk minimisation is used, then training is straightforward, but domain generalization performance is poor due to sensitivity to distributional shifts
Solution Approach 1:
The patent introduces dynamics into the sampling process by making minibatch composition adaptive based on feature dissimilarities. Rather than static random sampling, the method dynamically adjusts which samples are included based on their relative positions in feature space, improving domain generalization while adding controlled complexity to the training process
Data Source
Figure 1
Figure 2
Figure 3
AI summary
There is described a computer implemented method (200) of training a neural network. The method comprises obtaining (202) a set of feature embeddings for the training dataset, sampling (204) the training dataset to generate a minibatch, wherein the sampling comprises determining (206) a dissimilarity between instances of the training dataset in the feature embedding space, and selecting (208) instances for inclusion in the minibatch based on the determined dissimilarities, wherein instances having larger dissimilarity have a greater probability of being jointly selected for inclusion in the minibatch, and updating (210) the network.