Activation Function Hardware Circuit for Neural Network Processing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Neural networks are computationally intensive and slow due to the need for executing multiple activation functions, particularly as the number of layers increases, leading to significant processing time and resource consumption.

Innovation Solution

Implementing dedicated hardware circuits that approximate activation functions using linear functions over specific ranges, allowing for faster processing by determining the appropriate range based on a subset of input bits, thereby reducing the complexity and power consumption of processing operations.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If multiple activation functions are executed in software for neural network processing, then the neural network can perform accurate computational operations, but the processing speed decreases and computational complexity increases

Engineering Contradiction:

Improvecomputational accuracyVSAvoidprocessing speed

Core Design Contradiction:

Measurement precisionVSProductivity

Solution Approach 1:

The patent replaces software-based activation function execution with dedicated hardware circuits. Each activation function type (ReLU, sigmoid, tanh, etc.) has a corresponding hardware circuit that directly computes the result through electrical signals, eliminating the need for software interpretation and multiple sub-operations. This substitution of mechanical/software processes with electrical/hardware processes dramatically improves processing speed while maintaining computational accuracy.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent divides the neural network processing into separate dedicated hardware circuits for each activation function type. Instead of using a general-purpose processor to handle all activation functions sequentially through software, the system segments the computation into parallel hardware paths, each optimized for a specific activation function, thereby improving overall processing throughput.

Inventive Principle:

Principle #1Segmentation

2Measurement precision

If multiple activation functions are executed in software for neural network processing, then the neural network can perform accurate computational operations, but the number of sub-operations increases significantly

Engineering Contradiction:

Improvecomputational accuracyVSAvoidnumber of sub-operations

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Productivity

If dedicated hardware circuits are implemented for activation functions, then processing speed improves significantly, but circuit complexity increases

Engineering Contradiction:

Improveprocessing speedVSAvoidcircuit complexity

Core Design Contradiction:

ProductivityVSDevice complexity

Solution Approach 1:

The patent implements different hardware circuit designs optimized for specific activation function characteristics. For example, ReLU circuits use simple conditional routing, while sigmoid circuits use lookup tables or approximate computational structures. Each circuit is locally optimized for its specific function rather than using a universal design, reducing overall circuit complexity while maintaining high processing speed.

Inventive Principle:

Principle #3Local quality

Solution Approach 2:

The patent employs approximation techniques that change the computational parameters of activation functions. Instead of implementing exact mathematical formulas, the circuits use piecewise linear approximations, lookup tables, or truncated series expansions that reduce computational complexity while maintaining sufficient accuracy for neural network operations.

Inventive Principle:

Principle #35Parameter changes

4Productivity

If dedicated hardware circuits are implemented for activation functions, then electrical power usage increases, but processing speed improves

Engineering Contradiction:

Improveprocessing speedVSAvoidelectrical power usage

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

Inventive Principle:

Principle #3Local quality

Data Source

PatentUS11475305B2Activation function functional block for electronic devices

Publication Date: 2022.10.18 ADVANCED MICRO DEVICES INC

AI summary

An electronic device has an activation function functional block that implements an activation function. During operation, the activation function functional block receives an input including a plurality of bits representing a numerical value. The activation function functional block then determines a range from among a plurality of ranges into which the input falls, each range including a separate portion of possible numerical values of the input. The activation function functional block next generates a result of a linear function associated with the range. Generating the result includes using a separate linear function that is associated with each range in the plurality of ranges to approximate results of the activation function within that range.