Neural Network Training via Neighborhood Cluster Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional batch construction methods for training artificial neural networks often result in overfitting and negatively impact test accuracy due to uneven index overlap patterns between training batches and the full data distribution, leading to adjustments in embedding weights that may not accurately reflect the entire data set.
Innovation Solution
The method involves selecting a training batch of points from a dataset, forming a neighborhood of points that share at least one index with the batch, and using cluster analysis to choose a cluster of points for training, effectively freezing weights in the embedding layer not part of the batch, thereby improving the training process and reducing overfitting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional batch construction methods are used for training, then training speed is maintained, but overfitting occurs and test accuracy deteriorates
Solution Approach 1:
The patent introduces an intermediary selection process between the full dataset and the training batch. Instead of directly using conventional batches, the system forms a neighborhood of candidate points and selects an intermediate batch that balances representation of the full data distribution with training efficiency, thereby reducing overfitting while maintaining training speed
Solution Approach 2:
The patent changes the parameters of batch construction by forming neighborhoods around data points and selecting batches based on their ability to represent the full data distribution. This parameter change in batch selection strategy prevents overfitting while maintaining acceptable training speed
2Manufacturing precision
If embedding weights are adjusted to accommodate training batches with uneven index overlap patterns, then training convergence is achieved, but the embedding weights do not accurately reflect the entire data set
Solution Approach 1:
The patent applies local quality by forming neighborhoods around individual data points and selecting training batches based on local density and representation characteristics. This ensures that embedding weights are adjusted based on locally representative samples rather than uniformly random batches, improving the accuracy of weight adjustments while maintaining overall data distribution representation
Solution Approach 2:
The system incorporates feedback by evaluating how well training batches represent the full data distribution and adjusting batch selection accordingly. This feedback mechanism ensures that embedding weight adjustments accurately reflect the entire dataset rather than biased samples
Data Source
AI summary
The disclosed computer-implemented method may include (1) selecting, for training of an artificial neural network (ANN), a training batch of points from within a dataset of training points, each training point comprising a plurality of sets of values, where each value corresponds to an index into an embedding space included in the ANN, (2) forming, from the dataset of training points, a neighborhood of training points associated with the training batch such that each member of the neighborhood shares at least one index with at least one training point included in the training batch, (3) choosing, via a cluster analysis method, a cluster of points from the neighborhood of training points associated with the training batch, and (4) training the ANN using the chosen cluster of points from the neighborhood of points associated with the training batch. Various other methods, systems, and computer-readable media are also disclosed.