Non-Uniform Activation Function Approximation Table
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for approximating activation functions in neural networks using discrete sample values result in lower accuracy and inefficient resource usage, particularly when the number of samples is small, leading to wasted memory resources due to uniform segmentation of the function domain.
Innovation Solution
A computing apparatus and method that generates an approximation table with input samples spaced farther apart from a reference point, allowing for more efficient use of resources by varying neighboring sample intervals based on the activation function's slope, thereby maintaining accuracy without unnecessary sample density in regions of low change.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If the number of samples is increased to secure sufficient approximation accuracy over the entire domain, then approximation accuracy is improved, but memory resources are wasted due to unnecessarily large number of samples in regions where function slope does not change much
Solution Approach 1:
The patent applies local quality by varying the sample interval based on the local characteristics of the activation function. Specifically, smaller intervals are used in regions where the function slope changes rapidly (high curvature) and larger intervals are used in regions where the slope is relatively stable (low curvature). This non-uniform sampling strategy ensures sufficient approximation accuracy where needed while reducing the total number of samples in regions where high density is not required, thereby resolving the contradiction between accuracy and memory usage.
2Ease of manufacture
If uniform segmentation is used to simplify table generation, then ease of manufacture is improved, but approximation accuracy deteriorates and function shape reproducibility is impaired when number of samples is small
Solution Approach 1:
The patent applies parameter changes by modifying the sampling interval parameter based on the activation function's characteristics. Instead of using a constant interval throughout the domain, the method dynamically adjusts the interval parameter according to the function's slope and curvature at different points. This allows the table generation process to maintain simplicity while achieving high approximation accuracy, as the interval adjustment is based on straightforward evaluations of the activation function's derivatives.
3Device complexity
If larger neighboring sample interval is used to reduce memory usage, then device complexity is reduced, but approximation accuracy deteriorates in regions where function slope changes rapidly
Solution Approach 1:
The patent applies dynamics by making the sample interval adaptive rather than fixed. The interval dynamically adjusts based on the local properties of the activation function (specifically the slope and curvature) at each region. In regions with rapid changes, the interval automatically becomes smaller to maintain accuracy, while in regions with stable behavior, the interval becomes larger to save memory. This dynamic adaptation resolves the contradiction by allowing the system to optimize between accuracy and memory usage based on actual function characteristics.
Data Source
AI summary
There is provided a computing apparatus that includes: a retaining unit configured to retain an approximation table that approximately represents an activation function of a neural network, the approximation table mapping between a plurality of discrete input samples of the activation function and output samples respectively corresponding to the plurality of input samples; and a computing unit configured to convert an input value of activation function computation to an output value using the approximation table retained by the retaining unit when the activation function is selected for the activation function computation. The plurality of input samples of the approximation table are set such that input samples more distant from a reference point in the domain of the activation function have a larger neighboring sample interval.