Kernelized Classification Layer for Nonlinear Neural Network Boundaries
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks struggle with nonlinearly separable classes, as they rely on linear classifiers in the space of feature vectors, limiting their classification accuracy, especially when the feature extractor lacks capacity to learn fully linearly separable features.
Innovation Solution
Introduce a kernelized classification layer in neural networks that automatically learns an optimal positive definite kernel from a space of radial kernels, enabling nonlinear classification by embedding feature vectors into a higher-dimensional Reproducing Kernel Hilbert Space (RKHS) without explicit computation, using a modified softmax loss function and backpropagation.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If a linear classifier is used in the space of feature vectors, then the neural network structure is simple and computationally efficient, but the classification accuracy deteriorates when classes are nonlinearly separable
Solution Approach 1:
The patent applies dimensionality change by mapping feature vectors into a Reproducing Kernel Hilbert Space (RKHS) where classes become linearly separable. The kernelized classification layer learns an optimal positive definite kernel that embeds inputs into this higher-dimensional space, enabling linear separation of nonlinearly separable classes while maintaining computational efficiency through kernel tricks.
2Reliability
If the feature extractor is given more capacity to learn linearly separable features, then classification accuracy improves, but the network complexity and training difficulty increase
Solution Approach 1:
The patent introduces a kernelized classification layer as an intermediary between the feature extractor and the final classification decision. This layer learns an optimal kernel that transforms the feature space, allowing the feature extractor to remain relatively simple while achieving high classification accuracy through the kernel-induced nonlinear transformation.
3Productivity
If a predefined linear classifier is used during training, then the training process is straightforward and fast, but the model cannot adapt to nonlinear classification boundaries
Solution Approach 1:
The patent makes the classification layer dynamic by enabling it to learn an optimal positive definite kernel during training through backpropagation. The kernelized classification layer adapts its kernel parameters based on the training data, allowing the model to automatically adjust to nonlinear classification boundaries while maintaining reasonable training speed through efficient gradient computation.
Data Source
AI summary
A method includes receiving, by a computing device, training data to train a neural network, wherein the training data comprises a plurality of inputs and a plurality of corresponding labels. The method also includes mapping, by a representation learner of the neural network, the plurality of inputs to a plurality of feature vectors. The method additionally includes training a kernelized classification layer of the neural network to perform nonlinear classification of an input feature vector into one of a plurality of classes, wherein the kernelized classification layer is based on a kernel which enables the nonlinear classification, and wherein the kernel is selected from a space of positive definite kernels based on application of a nonlinear softmax loss function to the plurality of feature vectors and the plurality of corresponding labels. The method further includes outputting a trained neural network comprising the representation learner and the trained kernelized classification layer.