Hybrid Delta Modulator Neuron for Variable Threshold Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks face challenges in implementing a simple and effective way to incorporate varying threshold neurons, which are essential for time-dependent signal analysis in applications like speech recognition and handwriting analysis.
Innovation Solution
A hybrid delta modulator is introduced, which includes a weighting circuit and a modulator circuit with a differencing element, quantizer, and filter network, allowing for a quantized output that represents the sum-of-products signal and exhibits memory of its prior state, enabling the neuron to adapt its threshold based on input and derivative.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If a canonical Elman RNN is used to provide feedback for time-dependent signal analysis, then the network can recognize evolution over time, but the device complexity increases due to explicit feedback loops and prior state variables
Solution Approach 1:
The patent extracts the memory function from the explicit feedback loop structure and implements it through a delta modulator's inherent state retention mechanism. The differencing element and quantizer in the modulator naturally maintain prior state information without requiring separate feedback connections, thereby simplifying the overall network architecture while preserving time-dependent analysis capabilities.
Solution Approach 2:
The delta modulator acts as an intermediary component between the weighting circuit and the output, introducing a differencing element that implicitly handles temporal dependencies. This mediator structure allows the network to process time-dependent signals without the complexity of explicit RNN feedback loops, as the modulator's internal state differences capture the necessary temporal information.
2Adaptability or versatility
If a spiking neural network with varying threshold is used, then the neuron can adapt its threshold based on input and recover to steady state, but the ease of manufacture decreases due to the complexity of implementing varying threshold neurons
Solution Approach 1:
The patent replaces the complex mechanical or electronic threshold modulation mechanism with a mathematical transformation approach. The differencing element in the delta modulator naturally produces a varying threshold effect by comparing current and prior states, eliminating the need for complex threshold control circuitry while achieving the same adaptive behavior.
Solution Approach 2:
The patent changes the operational parameters of the neuron by introducing a differencing operation that dynamically adjusts the effective threshold based on the difference between current and prior states. This parameter change approach allows the neuron to exhibit variable threshold behavior through simple subtraction and quantization operations rather than complex threshold control mechanisms.
3Adaptability or versatility
If a hybrid delta modulator is used to separate integral and gain functions, then the output pattern can be tailored to include signal and rate of change, but the device complexity increases due to the additional circuit components
Solution Approach 1:
The patent merges the integral and gain functions into a single delta modulator circuit structure. The differencing element inherently performs integration of the difference signal, while the quantizer provides gain control, combining multiple functions into one compact unit that reduces overall circuit complexity despite the versatile output capabilities.
Solution Approach 2:
The delta modulator is designed as a universal component that can simultaneously perform multiple functions: differencing, integration, quantization, and adaptive thresholding. This multi-functional design allows the same circuit structure to tailor output patterns to include both signal magnitude and rate of change without requiring separate dedicated circuits for each function.
Data Source
AI summary
A hybrid delta modulator that can be used as a variable threshold neuron in a neural network is described. The hybrid delta modulator exhibits a memory of the prior state of the modulator, similar to a delta modulator, and receives a sum-of-products signal from a weighting circuit and generates a quantized output stream that represents the sum-of-products signal, potentially including an activation function and offset. With appropriately selected components, the hybrid delta modulator separates the integral function of the feedback from the gain function. Further, the gain can be selected, and the characteristic of the output pattern can be tailored to include an arbitrary combination of the input and the rate of change of the input. The use of a hybrid delta modulator of the present approach provides a simpler solution and better performance than many prior art neurons.