ANN Activation Function Using Bit-Wise Operations

Solution Overview

Problem

Current artificial neural networks (ANNs) face challenges in implementing sigmoid and tanh activation functions due to resource constraints, particularly in low-power applications, as they require significant resources and are impractical for embedded systems like SoC in industrial or automotive applications.

Innovation Solution

The implementation of a piece-wise linear saturating activation function using digital bit-wise operations, such as bit-slicing, bit-shifting, increment, decrement, addition, and subtraction operations, which eliminates the need for multiplication, division, or look-up tables, allowing for efficient computation on resource-constrained devices.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Reliability

If traditional sigmoid or tanh activation functions are used in ANNs, then the network can achieve good performance in image recognition and other tasks, but the hardware resources and energy consumption increase significantly making them impractical for low-power embedded applications

Engineering Contradiction:

ImproveperformanceVSAvoidenergy consumption

Core Design Contradiction:

ReliabilityVSUse of energy by moving object

Solution Approach 1:

The patent transforms the continuous sigmoid/tanh activation functions into discrete piecewise linear approximations by changing the mathematical parameters and representation. This allows the network to maintain functional performance while enabling efficient digital hardware implementation that reduces energy consumption in embedded systems

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The patent replaces the traditional floating-point mathematical computation mechanism with a digital bit-wise operation mechanism. By substituting multiplication and division operations with bit-shifting and logical operations, the system achieves lower energy consumption while maintaining computational functionality

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

2Reliability

If traditional sigmoid or tanh activation functions are used in ANNs, then the network can achieve good performance, but the hardware requirements and device complexity increase making them impractical for embedded systems

Engineering Contradiction:

ImproveperformanceVSAvoidhardware requirements

Core Design Contradiction:

ReliabilityVSDevice complexity

Solution Approach 1:

The patent transforms the continuous sigmoid/tanh activation functions into discrete piecewise linear approximations by changing the mathematical parameters and representation. This allows the network to maintain functional performance while enabling efficient digital hardware implementation that reduces energy consumption in embedded systems

Inventive Principle:

Principle #35Parameter changes

Solution Approach 2:

The patent replaces the traditional floating-point mathematical computation mechanism with a digital bit-wise operation mechanism. By substituting multiplication and division operations with bit-shifting and logical operations, the system achieves lower energy consumption while maintaining computational functionality

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

3Device complexity

If look-up tables or polynomial approximations are used to implement activation functions, then the computation can be simplified, but the use of look-up table memories and multiply/divide operations makes them sub-optimal for lower-power applications

Engineering Contradiction:

Improvecomputation simplificationVSAvoidenergy consumption

Core Design Contradiction:

Device complexityVSUse of energy by moving object

Solution Approach 1:

The patent replaces the traditional floating-point mathematical computation mechanism with a digital bit-wise operation mechanism. By substituting multiplication and division operations with bit-shifting and logical operations, the system achieves lower energy consumption while maintaining computational functionality

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent divides the activation function computation into discrete piecewise linear segments that can be efficiently implemented using bit-slicing and conditional operations. This segmentation eliminates the need for complex polynomial calculations or large look-up tables while maintaining accuracy

Inventive Principle:

Principle #1Segmentation

Data Source

PatentUS12039432B2Artificial neural network activation function

Publication Date: 2024.07.16 INFINEON TECHNOLOGIES AG

AI summary

An artificial neural network (ANN) apparatus can include processing component circuitry that receives linear inputs, and removes linearity from the one or more linear inputs based on an S-shaped saturating activation function that generates a continuous non-linear output. The neurons of the ANN comprise digital bit-wise components configured to transform the linear inputs into the continuous non-linear output.