PIM Activation Function Approximation with Interval-Based Parallel Selection
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional artificial neural networks face increased computational costs due to the use of high-complexity activation functions, which are not efficiently accelerated by existing methods.
Innovation Solution
An apparatus and method that utilize approximate activation functions approximated by intervals based on input values, allowing for reduced computational cost and accelerated operations through parallel processing using PIM (Processing-in-memory).
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If high-complexity activation functions are used in neural networks, then model accuracy and capability are improved, but computational cost increases
Solution Approach 1:
The patent divides the application range of the activation function into multiple intervals and creates separate approximate activation functions for each interval. This segmentation allows the system to use simple linear approximations locally while maintaining the ability to represent complex nonlinear behavior globally, thus reducing computational cost while preserving model accuracy.
Solution Approach 2:
The patent changes the parameter representation by using interval indicators and one-hot encoding to select among multiple approximate activation functions. Instead of directly computing complex activation functions, the system transforms the input into interval-based parameters that guide selection of pre-computed approximate functions, significantly reducing computational complexity.
2Reliability
If complex activation functions are used, then neural network performance is improved, but operation speed decreases
Solution Approach 1:
The patent pre-computes multiple approximate activation functions for different intervals and stores them in advance. During inference, the system only needs to determine the input's interval and retrieve the corresponding pre-computed function, avoiding real-time complex calculations and significantly accelerating operation speed while maintaining performance.
Solution Approach 2:
The patent creates multiple copies of approximate activation functions for different intervals and stores them in memory. Instead of computing a single complex activation function, the system copies and stores simplified linear approximations for each interval, enabling fast retrieval and parallel processing that accelerates operation speed.
3Device complexity
If approximate activation functions are used, then computational cost is reduced, but accuracy may be compromised
Solution Approach 1:
The patent applies local quality by creating different approximate activation functions tailored to specific intervals of the input range. Each interval receives a customized linear approximation that is optimized for its local characteristics, ensuring high accuracy within each interval while maintaining low computational cost overall.
Solution Approach 2:
The patent uses multiple approximate activation functions (excessive action) rather than a single approximation, allowing the system to select the most appropriate approximation for each input. This partial application of different approximations ensures that accuracy is maintained across the entire input range without requiring a single complex global approximation.
4Productivity
If multiple approximate activation functions are stored in memory, then parallel processing is enabled, but memory usage increases
Solution Approach 1:
The patent transforms the problem from computing activation functions sequentially to organizing them in a dimensional structure with interval indicators and one-hot encoding. This dimensional organization enables parallel processing across multiple intervals simultaneously while using memory efficiently through structured storage and selective retrieval based on interval indicators.
Data Source
AI summary
The present disclosure provides an apparatus and method for accelerating activation function, wherein the application range of the activation function is divided into multiple intervals, and an interval indicator is obtained indicating an interval in which an input value is included by using multiple interval detection operators to select one approximate activation function among multiple approximate activation functions approximated in each interval, an approximate activation function being set for the interval designated by the interval indicator is obtained, and the input value is input into the obtained approximate activation function to obtain an activation operation result. According to the apparatus and method for accelerating activation function of of the present disclosure, not only can an interval according to an input value for selecting an approximate activation function be easily determined, but also the operation can be accelerated by performing activation operations simultaneously in parallel.