Neural Network Hidden Layer Function Weighting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks often select activation, normalization, denoising, and regularization functions based on empirical knowledge rather than theoretical grounds, leading to suboptimal performance due to the lack of adaptation to specific input data.
Innovation Solution
An information processing apparatus that weights and combines various functions usable in a hidden layer of a neural network, adjusts these weights during learning using error back propagation, and produces a new function optimized for the specific data, enhancing learning accuracy and robustness.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If existing activation functions (ReLU, sigmoid) are selected based on empirical knowledge, then the neural network can be implemented with standard functions, but the function is not always adapted to the specific input data leading to suboptimal performance
Solution Approach 1:
The patent implements dynamic function selection by introducing a gating mechanism that adaptively chooses between different activation functions (ReLU, sigmoid, tanh) based on the input data characteristics. The gate network dynamically determines the appropriate function for each neuron, allowing the network to adapt to different input distributions and optimization scenarios rather than using a fixed activation function throughout.
Solution Approach 2:
The patent changes the parameters of the activation function selection by introducing learnable gate parameters that control the mixing ratio between different activation functions. These parameters are optimized during training to find the optimal combination of functions for each specific input data, thereby improving adaptability without manual intervention.
2Adaptability or versatility
If a single activation function is used throughout the network, then the implementation is simple, but it cannot be optimized for different layers or input data types
Solution Approach 1:
The patent creates a universal activation function module that can perform multiple functions by selecting from different activation types (ReLU, sigmoid, tanh, leaky ReLU) through a gating mechanism. This multi-functional design allows the same network architecture to adapt to different data types and layer requirements without requiring separate networks for each scenario.
Solution Approach 2:
The patent introduces a gate network as an intermediary component that mediates between the input data characteristics and the activation function selection. The gate network analyzes the input data and produces control signals that determine which activation function should be applied, thereby enabling adaptive function selection while maintaining a unified network structure.
3Reliability
If empirical knowledge is used to select activation functions, then the selection process is straightforward, but the theoretical justification is lacking and performance is not optimized
Solution Approach 1:
The patent implements feedback mechanisms where the gate network continuously monitors the performance and characteristics of the neural network during training, and adjusts the activation function selection accordingly. The loss gradient flows back through the gate network, enabling it to learn which activation functions produce better performance for different input patterns, thereby improving reliability through data-driven decisions rather than empirical guesses.
Solution Approach 2:
The patent enables the neural network to self-determine the optimal activation functions through the trained gate mechanism. Instead of relying on external expert knowledge for function selection, the network automatically learns and selects the most appropriate activation functions based on the specific characteristics of the input data and task requirements, improving both reliability and autonomy.
Data Source
AI summary
To cause a more appropriate function to be applied to a hidden layer in a neural network.An information processing apparatus including a memory and one or a plurality of processors, wherein the memory stores: a learning model using a neural network; each function usable in a hidden layer of the neural network; and a first function that is produced by weighting each of the functions, and the one or a plurality of processors: acquire prescribed learning data; apply the first function commonly to a prescribed node group in a hidden layer of the learning model; perform learning by inputting the acquired prescribed learning data to the learning model in which the first function has been applied to the hidden layer; when learning the learning model, update a parameter of a neural network of the learning model by error back propagation, based on a supervisor label of the prescribed learning data; adjust each weight of the first function when the parameter of the neural network is updated; and produce, after the learning model is learned, a second function in which each of the adjusted weights is set to the first function.