Neural Network Activation Function Lookup Table Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network technologies require extensive storage space for lookup tables corresponding to each activation function, leading to inefficient processing performance due to the need for multiple lookup tables and increased computational complexity.
Innovation Solution
The method converts data from a target activation function to a preset activation function using a conversion relationship, allowing the use of a single lookup table for the preset function, thereby reducing storage occupancy and improving processing efficiency by expressing the target function through the preset function.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If multiple lookup tables are used for different activation functions, then processing accuracy is maintained, but storage space consumption increases and processing efficiency decreases
Solution Approach 1:
The patent applies universality by designing a single lookup table that can serve multiple activation functions. The conversion module enables the system to transform data between different activation function formats, allowing one lookup table to handle sigmoid, tanh, and other activation functions through mathematical conversion relationships, thus eliminating the need for separate lookup tables for each function.
Solution Approach 2:
The patent utilizes parameter changes by applying mathematical transformations to convert data between different activation function domains. The conversion module modifies data parameters (scaling, shifting, or other transformations) to map values from one activation function's range to another, enabling a single lookup table to be used across multiple activation function types.
2Adaptability or versatility
If multiple lookup tables are maintained for different activation functions, then function-specific processing is preserved, but computational complexity increases
Solution Approach 1:
The conversion module acts as an intermediary between the input data and the single lookup table. It performs necessary mathematical transformations on the input data according to the target activation function, then queries the unified lookup table, and finally applies inverse transformations to produce the output. This mediator approach maintains function-specific processing while simplifying the overall system structure.
Solution Approach 2:
The system achieves versatility through a universal lookup table that can process multiple activation functions. The conversion module adapts the input data to match the lookup table's expected input range for the desired activation function, allowing the same table to serve multiple purposes without requiring separate tables for each function type.
3Speed
If separate lookup tables are used for each activation function, then processing speed is maintained for specific functions, but overall processing efficiency decreases
Solution Approach 1:
The patent merges multiple separate lookup tables into a single unified lookup table. By combining the storage resources of multiple tables into one, the system reduces memory access overhead and improves cache utilization. The conversion module handles the necessary transformations, allowing the merged structure to maintain or even improve processing speed while enhancing overall system efficiency.
Solution Approach 2:
The conversion module applies parameter changes to adapt input data to the unified lookup table's format. By transforming data parameters on-the-fly, the system maintains processing speed for different activation functions while benefiting from the efficiency gains of a consolidated lookup table structure, thereby improving overall processing efficiency.
Data Source
AI summary
Embodiments of the present disclosure relate to a data processing method and apparatus for a neural network. The neural network is provided with at least one activation function. A method may include: converting, in response to that an activation function acquiring current data is a target function, based on a conversion relationship between the target function and a preset function, the current data into input data of the preset function; finding out first output data of the preset function with the input data as an input in a lookup table corresponding to the preset function; obtaining second output data of the target function with the current data as an input by conversion based on the conversion relationship and the first output data; and outputting the second output data.