1.5-Branch Bipolar Morphological Networks With Layer-by-Layer Distillation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training bipolar morphological neural networks using gradient methods based on backpropagation of error is challenging due to the use of maximum operations, which only update four weight values per neuron per iteration, and the complex computational structure of bipolar morphological neurons.
Innovation Solution
A layer-by-layer knowledge distillation method is employed to transform classical neural networks into bipolar morphological networks, using a simplified 1.5-branch model for bipolar morphological neurons, and incorporating continuous approximations of the maximum operation during training to enhance training efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If bipolar morphological neurons use maximum operations for computation, then hardware complexity is reduced, but training difficulty increases significantly
Solution Approach 1:
The patent introduces continuous approximations of the maximum operation (such as log-sum-exp or softplus functions) as intermediaries during the training phase. These approximation functions are differentiable and allow gradient-based optimization, serving as a bridge between the non-differentiable maximum operation and the requirements of backpropagation training. After training, the approximation is replaced with the exact maximum operation for deployment.
Solution Approach 2:
The patent transforms the bipolar morphological neuron parameters by shifting inputs and weights to positive values only. This parameter transformation simplifies the neuron structure from four computational branches to a single branch, making the training process more tractable while preserving the essential functionality of the bipolar morphological computation.
2Speed
If bipolar morphological neurons update only four weight values per iteration, then computational speed improves, but training precision deteriorates
Solution Approach 1:
By using continuous approximations of the maximum operation during training, the patent enables all weight parameters to be updated via gradient descent, rather than limiting updates to only four weight values. This intermediary approach maintains computational efficiency while significantly improving training precision through full parameter optimization.
3Adaptability or versatility
If bipolar morphological neurons use four computational branches, then functionality is preserved, but device complexity increases
Solution Approach 1:
The patent applies parameter transformation by shifting all inputs and weights to positive values. This transformation allows the bipolar morphological neuron to be implemented with a single computational branch instead of four, reducing device complexity while preserving the essential functionality through the mathematical equivalence of the transformed operations.
Data Source
AI summary
Bipolar morphological (BM) neural networks can be used to improve performance over classical artificial neural networks, at the inference stage, using specialized hardware. Accordingly, embodiments introduce a 1.5-branch BM neuron model to increase the computational efficiency of the inference process. However, it can be difficult to train BM neural networks using classical training methods. Therefore, embodiments construct such a 1.5-branch BM neural network using layer-by-layer knowledge distillation. In an embodiment, the construction of the 1.5-branch BM neural network is further improved using maximum approximation.