Hardware Dropout Selector Element for Neural Network Overfitting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing artificial neural networks face overfitting issues due to excessive focus on input training data, leading to a loss of generality, and current dropout techniques lack effective hardware-based implementations to address this problem.
Innovation Solution
A hardware-based dropout apparatus using a selector element and transistors is implemented, allowing target currents to selectively flow to neurons or ground nodes based on a driving voltage and threshold voltage, enabling on-chip operation without requiring communication with external circuits, and is integrated into a neural network circuit system to prevent overfitting.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If dropout technique is applied to prevent overfitting in artificial neural networks, then generalization performance is improved, but hardware implementation complexity increases due to lack of effective hardware-based solutions
Solution Approach 1:
The patent replaces software-based dropout implementation with a hardware-based circuit implementation using selector elements and transistors. The selector element (221) and transistor (211, 212) circuit systematically controls current flow to neurons, substituting the mechanical/software random deactivation with an electrical hardware mechanism that achieves the same dropout effect while enabling on-chip operation.
Solution Approach 2:
The patent introduces a selector element (221) as an intermediary component between the driving unit and the neuron circuit. This selector element acts as a mediator that controls whether current flows to the neuron or is redirected to ground, enabling the dropout function without requiring complex external control circuits.
2Adaptability or versatility
If multiple hidden layers are used to learn complicated data structures, then learning capability is improved, but the number of parameters and learning time increase significantly
Solution Approach 1:
The patent applies dynamic dropout during the learning process by randomly deactivating neurons in each epoch. This dynamic adjustment of network structure during training helps the network generalize better when learning complicated data structures across multiple hidden layers, reducing overfitting and improving learning efficiency for complex patterns.
3Reliability
If conventional dropout is implemented without hardware optimization, then overfitting prevention is achieved, but power efficiency deteriorates due to repeated communication with external circuits
Solution Approach 1:
The patent implements self-service dropout functionality directly on the neural network chip using integrated selector elements and transistor circuits. Each neuron circuit includes its own dropout control mechanism, eliminating the need for external circuit communication and enabling the system to perform dropout operations autonomously, thereby significantly improving power efficiency.
Solution Approach 2:
The patent extracts the dropout control functionality from external circuits and integrates it directly into the neuron circuit on-chip. By taking out the dropout mechanism and embedding it within the neural network hardware, the system eliminates power-consuming external communications while maintaining overfitting prevention capability.
Applied Scientific Principles
This section explains which scientific principles are used to turn an abstract innovation direction into a practical engineering solution.
Function Achieved in This Case
The solution effectively prevents overfitting by randomly turning on or off neurons during the learning process, improving power efficiency and enabling the use of dropout in complex neural networks for applications like autonomous driving and image processing with limited data, while maintaining performance across varying training data sizes.
Implementation Method 1
a driving unit which includes a selector element turned on or off according to a size relationship of an applied driving voltage and a threshold voltage, and controlling the switch unit
Implementation Method 2
a switch unit disposed on the bit line of a hardware-based artificial neural network, and provided to allow a target current which flows to the bit line to selectively flow to a predetermined neuron or ground node constituting the artificial neural network
Data Source
AI summary
Disclosed are an apparatus for implementing hardware-based dropout for an artificial neural network using a selector element and a neural network circuit system using the same, and an apparatus for implementing hardware-based dropout for an artificial neural network using a selector element according to an exemplary embodiment of the present disclosure may include: a switch unit disposed on the bit line of a hardware-based artificial neural network, and provided to allow a target current which flows to the bit line to selectively flow to a predetermined neuron or ground node constituting the artificial neural network; and a driving unit which includes a selector element turned on or off according to a size relationship of an applied driving voltage and a threshold voltage, and controlling the switch unit.