Noise-Enhanced CNN Training via Noisy EM Algorithm
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The backpropagation training algorithm for convolutional neural networks (CNNs) is computationally slow, especially when processing images, and existing methods have not provided fundamental solutions to speed up this process effectively.
Innovation Solution
The Noisy CNN algorithm injects carefully chosen noise into the network, leveraging the equivalence of backpropagation with the expectation-maximization algorithm to speed up training by finding a separating hyperplane in the noise space, which can accelerate convergence and improve accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If backpropagation training algorithm is used for CNNs, then training can be performed with linear time complexity in the number of training samples, but the training process is computationally slow
Solution Approach 1:
The patent applies noisy EM algorithm which intentionally introduces noise into the training process. The noise, which could be considered harmful, actually accelerates convergence by helping the algorithm escape local optima and explore the parameter space more effectively. This converts the harmful effect of noise into a beneficial acceleration of training.
Solution Approach 2:
The patent modifies the standard backpropagation algorithm by changing key parameters: introducing noise variance control, adjusting the EM iteration structure, and modifying the likelihood function. These parameter changes enable the training process to converge faster while maintaining linear time complexity scaling.
2Measurement precision
If more training data is used to improve accuracy, then classification performance improves, but training time increases linearly
Solution Approach 1:
The noisy EM algorithm achieves better classification accuracy with smaller training datasets by using noise-enhanced sampling. Instead of requiring exhaustive training on large datasets, the algorithm uses partial data with noise-induced exploration to achieve comparable or superior performance, reducing the linear time penalty associated with larger datasets.
Data Source
AI summary
A learning computer system may include a data processing system and a hardware processor and may estimate parameters and states of a stochastic or uncertain system. The system may receive data from a user or other source. Parameters and states of the stochastic or uncertain system are estimated using the received data, numerical perturbations, and previous parameters and states of the stochastic or uncertain system. It is determined whether the generated numerical perturbations satisfy a condition. If the numerical perturbations satisfy the condition, the numerical perturbations are injected into the estimated parameters or states, the received data, the processed data, the masked or filtered data, or the processing units.