Neural Network Learning Device Linearization Strategy
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network learning methods, such as those described in Patent Literature 1, perform learning as if the problem is linearly non-separable, even when the learning target is linearly separable, leading to increased computational complexity due to the use of nonlinear activation functions.
Innovation Solution
A neural network learning device that includes a learning unit, a linearization quantity determination unit, and an aggregation unit. The linearization quantity determination unit adjusts a parameter in the activation function to bring it closer to a linear function, and the aggregation unit replaces the activation function with a linear function when it converges, allowing for weight aggregation across layers using the linear function.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If nonlinear activation functions are used in neural network learning, then the network can solve linearly non-separable problems, but the computational complexity increases
Solution Approach 1:
The patent applies dynamics by making the activation function adaptable - it can switch between nonlinear and linear forms based on the learning progress. The activation function is designed to be dynamic, transitioning from a nonlinear state (when complex pattern recognition is needed) to a linear state (when the problem becomes linearly separable or near-separable), thereby reducing computational complexity while maintaining problem-solving capability.
Solution Approach 2:
The patent employs parameter changes by introducing a control parameter that determines the degree of nonlinearity in the activation function. By adjusting this parameter during learning, the system can transform the activation function from highly nonlinear to nearly linear, effectively controlling the computational complexity while preserving the ability to handle both linearly separable and non-separable problems.
2Adaptability or versatility
If multiple hidden layers are used in the neural network, then the model can represent complex relationships, but the amount of calculation cannot be reduced
Solution Approach 1:
The patent makes the activation function dynamic across different layers and learning stages. In layers or stages where the data is already linearly separable, the activation function transitions to a linear form, effectively reducing the computational burden of those specific layers while maintaining the multi-layer structure for representing complex relationships where needed.
Solution Approach 2:
The patent applies local quality by allowing different activation function characteristics in different parts of the network. Specifically, certain layers or neurons can have nonlinear activation functions when needed for complex pattern recognition, while other layers can use linear activation functions when the local data representation is already sufficient, thereby optimizing computation locally across the network architecture.
Data Source
AI summary
The learning unit 81 learns a neural network. The linearization quantity determination unit 82 determines linearization quantity, which is a parameter included in an activation function used in the neural network, and which is a parameter that brings the activation function closer to a linear function by increasing or decreasing itself. The aggregation unit 83 replaces the activation function, which is determined to converge to a linear function by increasing or decreasing the linearization quantity, with the linear function, and aggregating weights among layers using the replaced linear function. The learning unit 81 calculates evaluation value based on output by the neural network in learning the neural network, and the linearization quantity determination unit 82 changes the linearization quantity when the evaluation value satisfies the predetermined criterion.