Logistic Output Neurons for High Capacity Classification
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network classifiers, particularly those using softmax neurons, are limited when dealing with a large number of classifications, as they restrict coding options to unit bit vectors, leading to reduced efficiency in storing and classifying patterns.
Innovation Solution
A neural network architecture that incorporates logistic output neurons with random bipolar coding, allowing for more efficient pattern storage and classification by using fewer output neurons, and employs a deep-sweep training method to improve classification accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If softmax neurons with K unit bit vectors are used for classification, then the network can handle K classifications, but the number of output neurons must be at least K, leading to increased device complexity when K is large
Solution Approach 1:
The patent transitions from 1-in-K coding (one-hot encoding) to binary coding schemes where K classifications are represented by log2(K) binary neurons. This dimensional reduction allows the network to handle the same number of classifications with fewer output neurons, directly resolving the contradiction between adaptability and device complexity.
Solution Approach 2:
The patent changes the coding parameter from unit bit vectors (softmax) to binary vectors (logistic), fundamentally altering how classifications are represented. This parameter change enables more efficient encoding where M = log2(K) binary neurons can represent K classifications, reducing the output layer size while maintaining classification capacity.
2Adaptability or versatility
If more output neurons are used to increase classification capacity, then more patterns can be classified, but the network requires more parameters to train, increasing loss of information during training
Solution Approach 1:
By changing from softmax to binary logistic coding, the patent reduces the number of parameters in the output layer from O(K) to O(log2(K)). This parameter reduction decreases the information loss during training while maintaining the ability to classify K patterns, as the binary coding scheme is more parameter-efficient.
3Reliability
If softmax activation function is used, then the network produces probability vectors for K classes, but the coding is restricted to K unit bit vectors, limiting the efficiency of pattern representation
Solution Approach 1:
The patent changes the activation function from softmax to binary logistic, which fundamentally alters the output coding scheme. Instead of being restricted to K unit bit vectors, the network can now use binary vectors from a much larger space (2^M possible vectors where M = log2(K)), dramatically improving pattern storage efficiency while maintaining classification reliability.
Solution Approach 2:
The patent moves from the K-dimensional simplex space of softmax probabilities to a binary hypercube space where more efficient packing of classification vectors is possible. This dimensional change enables better utilization of the output space, improving productivity without sacrificing reliability.
Data Source
AI summary
A neural network architecture for classifying input data is provided. The neural network architecture includes an input block, an output block, and at least one hidden block interposed between the input block and the output block. Characteristically, each neuron of an input block output neuron layer, an output block input neuron layer, an output block output neuron layer, a hidden block input neuron layer and a hidden block output neuron layer, independently applies a logistic activation function or an activation function that is the sum of a logistic activation function and a linear term or an activation function that is the sum of a logistic activation function and a quasi-linear term.