Multivariate Nonlinear Activation Function for XOR Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural networks rely on fixed canonical nonlinear activation functions, which may not accurately represent the complexity of brain neurons, limiting their ability to perform sophisticated operations like exclusive OR (XOR) and generalizing well to various noise and adversarial attacks.
Innovation Solution
Implementing a multivariate nonlinear activation function using a multilayer perceptron (MLP) within a neural network, merging it with an outer network through slice and concatenation operations, and pretraining with reinforcement learning to enhance learning efficiency and robustness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If fixed canonical nonlinear activation functions (sigmoid, tanh, ReLU) are used, then the neural network structure is simple and easy to implement, but the network cannot accurately represent the complexity of brain neurons and cannot perform sophisticated operations like XOR
Solution Approach 1:
The patent transforms the static, fixed activation function into a dynamic, learnable multivariate nonlinear activation function. The activation function is parameterized by a small neural network with learnable weights and biases, allowing it to adapt its behavior during training. This enables the activation function to dynamically adjust its characteristics to perform sophisticated operations like XOR while maintaining computational efficiency.
Solution Approach 2:
The patent changes the parameters of the activation function from fixed canonical forms to learnable multivariate parameters. By introducing a parameterized form with learnable weights (W) and biases (b) in the underlying linear transformations, the activation function can transform its behavior to match complex neural computations. This parameter transformation allows the same functional form to adapt to different computational requirements.
2Productivity
If a parameterized multivariate nonlinear activation function is implemented, then the network can learn complex patterns faster and perform sophisticated operations, but the training process becomes more complex requiring pretraining and merging with outer networks
Solution Approach 1:
The patent applies pretraining to the inner network parameters before merging with the outer network. This preliminary training phase allows the multivariate nonlinear activation function to learn basic complex patterns and computational capabilities in advance. By performing this preliminary action, the network reduces the overall training complexity when integrated into the complete system, as the activation function parameters are already optimized for fundamental operations.
Solution Approach 2:
The patent segments the neural network into an inner network (handling the multivariate nonlinear activation function) and an outer network (handling the main computational task). This segmentation allows independent training and optimization of each component. The inner network can be pretrained separately, and then merged with the outer network through systematic integration of parameters and computations, reducing the overall training complexity.
3Reliability
If conventional activation functions are used, then the network structure is straightforward, but the network shows poor robustness against noise and adversarial attacks
Solution Approach 1:
The patent creates a composite activation function by combining multiple linear transformations with different weight matrices and bias vectors within a single activation unit. This composite structure, formed by the multivariate nonlinear activation function with multiple inputs and internal computations, provides enhanced robustness against noise and adversarial attacks. The composite nature allows the activation function to capture complex input patterns and resist perturbations that would affect simpler activation functions.
Data Source
AI summary
Disclosed are an exclusive OR (XOR) operation learning probability of a multivariate nonlinear activation function and a practical application method thereof. A learning method of an activation function performed by a computer device may include constructing an inner network using a multivariate nonlinear activation function; and training a combination model generated by merging the constructed inner network and an outer network.