Hardware Pseudorandom Number Generator for Neural Network Overfitting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks face challenges in generating randomness efficiently, particularly in large datasets, where existing methods for implementing randomizing techniques using CPU or GPU are non-trivial and may not fully realize their performance benefits, and overfitting issues arise due to the complexity of networks with millions of parameters.
Innovation Solution
An execution unit for a processor is developed to efficiently generate random values from a predetermined, almost Gaussian distribution using a hardware pseudorandom number generator and normalizing circuitry, which produces bit strings that can be summed to achieve a desired probability distribution, optimizing for hardware efficiency and performance in neural networks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If CPU or GPU are used to implement randomizing techniques in neural networks, then randomness can be generated, but the performance benefits are not fully realized and the implementation is non-trivial
Solution Approach 1:
The patent replaces software-based random number generation on CPU/GPU with dedicated hardware circuitry that generates random values directly through electrical signals and logic operations, eliminating the need for complex software implementations and achieving full hardware acceleration
Solution Approach 2:
The patent introduces a dedicated random value generation unit as an intermediary component between the neural network processor and memory systems, providing random values through a specialized interface that simplifies integration while enabling efficient hardware-based randomness generation
2Adaptability or versatility
If large networks with millions of parameters are used, then neural network capability increases, but overfitting occurs where the network remembers training samples rather than extracting relevant features
Solution Approach 1:
The patent changes the statistical parameters of the random values generated, specifically using non-uniform probability distributions (such as Gaussian or Laplacian distributions) instead of uniform distributions, to provide appropriate regularization effects that prevent overfitting while maintaining network capability
Solution Approach 2:
The patent implements mechanisms where random values are dynamically introduced during training based on feedback from the network's performance, adjusting the amount and distribution of randomness to optimize both learning capability and generalization
Data Source
Figure 1
Figure 1A
Figure 2
AI summary
The present relates to invention deals with an execution unit configured to execute a computer program instruction to generate random numbers based on a predetermined probability distribution. The execution unit comprises a hardware pseudorandom number generator configured to generate at least randomised bit string on execution of the instruction and adding circuitry which is configured to receive a number of bit sequences of a predetermined bit length selected from the randomised bit string and to sum them to produce a result.