Stratified Sampling for Neural Network Training Data
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The training of 3D neural networks is negatively impacted by the variance introduced by randomly sampled training data parameters, leading to inconsistent gradient computations across training steps.
Innovation Solution
The use of stratified sampling to uniformly distribute training data parameters across their sampling range, reducing variance in gradient computations and improving training performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If random sampling is used for training data parameters, then sampling simplicity is maintained, but gradient computation variance increases
Solution Approach 1:
The sampling range is divided into multiple strata or intervals, and samples are drawn from each stratum separately. This segmentation of the sampling process reduces variance by ensuring coverage across the entire parameter range while maintaining the simplicity of random sampling within each stratum.
Solution Approach 2:
The sampling method changes from simple random sampling to stratified random sampling, where the distribution of samples across parameter ranges is controlled through stratification. This parameter change in the sampling approach reduces gradient computation variance while preserving operational simplicity.
2Ease of manufacture
If random sampling is used for training data parameters, then implementation ease is maintained, but training performance deteriorates
Solution Approach 1:
The parameter space is segmented into strata, allowing implementation of stratified sampling that maintains ease of implementation through systematic random sampling within each stratum while significantly improving training performance by reducing gradient variance.
Solution Approach 2:
Strata are predetermined based on the parameter ranges before sampling occurs. This preliminary action of defining strata boundaries enables the sampling process to be both easy to implement and high performing, as the stratification structure is prepared in advance and guides the random sampling process.
Data Source
AI summary
Apparatuses, systems, and techniques to train one or more neural networks using stratified sampled training data parameters. In at least one embodiment, one or more stochastic training data parameters may be stratified sampled from one or more sampling ranges to compute a gradient for updating the one or more neural networks.