Ruler Activation Function Capsule Network Sparsity
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The existing Squash activation function in capsule networks suffers from insufficient activation state sparsity and low updating speed when the activation state corresponds to high values, leading to poor performance in neural networks.
Innovation Solution
A new processing method and apparatus for a neural network model that uses a Ruler activation function, which iteratively updates the initial activation input value by performing affine transformation, weighted summation, and applying a modulus-based activation function to improve sparsity and updating speed.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If the Squash activation function is used in capsule network, then the neuron can be expanded into vector representation, but the activation state sparsity is insufficient and the updating speed is low when activation state corresponds to high value
Solution Approach 1:
The patent changes the parameter representation from scalar to vector by introducing direction information in capsule nodes. The activation function is redesigned to operate on vector inputs, using modulus length and directional components to achieve both sparsity and fast updating speed through iterative optimization.
Solution Approach 2:
The patent introduces an iterative update mechanism where the activation function dynamically adjusts the activation state through multiple iterations. The updating process is made dynamic by allowing the activation state to evolve from initial values through repeated applications of the activation function until convergence, enabling fast updates even for high activation states.
2Ease of operation
If the Squash activation function is used in capsule network, then the neuron can be expanded into vector representation, but the performance of the neural network is low
Solution Approach 1:
The patent segments the activation function into distinct computational components: modulus length calculation, directional component extraction, and iterative update steps. This segmentation allows each component to be optimized independently, improving overall network performance while maintaining vector representation capabilities.
Solution Approach 2:
The patent implements a feedback mechanism through iterative updates where the output of the activation function is fed back as input for the next iteration. This feedback loop enables the network to refine activation states progressively, leading to better convergence and improved performance compared to single-pass activation functions.
Data Source
Figure 1
Figure 2
Figure 3~4
AI summary
The disclosure provides a processing method and an apparatus of a neural network model, and relates to a field of computer technologies. The method includes: obtaining and converting input data of the ith processing layer into a plurality of capsule nodes; performing affine transformation on the plurality of the capsule nodes to generate a plurality of affine nodes; determining an initial activation input value according to the plurality of the affine nodes, and inputting the initial activation input value into an activation function to generate an initial activation output value; re-determining the initial activation input value according to an affine node corresponding to the initial activation output value, and inputting the re-determined initial activation input value into the activation function to regenerate the initial activation output value; repeating the acts for a preset number of times to determine the latest initial activation output value as an activation output value.