Sigmoid Function Circuit Using Hyperbolic Tangent Approximation

Solution Overview

Problem

Existing implementations of sigmoid functions in machine learning networks are inefficient in terms of area and power consumption, making them unsuitable for high-performance and low-cost deployments, particularly in integrated circuits like coarse-grain reconfigurable architectures.

Innovation Solution

The sigmoid function is approximated using a combination of hyperbolic tangent and exponential functions, with a comparator to divide the input domain, allowing for parallel circuit implementation and reduced computational power and area requirements.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If typical arithmetic logic unit circuits or look-up tables are used to implement sigmoid function, then the function can be computed, but the implementation involves significant latencies and consumes substantial computational power and area

Engineering Contradiction:

Improvesigmoid function computation accuracyVSAvoidcomputational power and area consumption

Core Design Contradiction:

Measurement precisionVSDevice complexity

Solution Approach 1:

The input domain is divided into multiple segments based on threshold values. For each segment, a simplified linear approximation function is used instead of the complete sigmoid function. This segmentation allows the system to trade off between approximation accuracy and computational complexity, achieving low latency and reduced power consumption while maintaining acceptable accuracy for machine learning applications

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

The sigmoid function parameters are changed by using different linear approximation equations for different input ranges. Instead of computing the transcendental sigmoid function across the entire domain, the system switches between different linear parameter sets (slopes and intercepts) depending on the input value, dramatically reducing computational requirements

Inventive Principle:

Principle #35Parameter changes

2Speed

If look-up table is compiled to form combinational logic circuit to provide sigmoid function result, then high speed computation is achieved, but the circuit becomes large and consumes significant power

Engineering Contradiction:

Improvesigmoid function computation speedVSAvoidcircuit area and power consumption

Core Design Contradiction:

SpeedVSArea of stationary object

Solution Approach 1:

The input domain is divided into multiple segments based on threshold values. For each segment, a simplified linear approximation function is used instead of the complete sigmoid function. This segmentation allows the system to trade off between approximation accuracy and computational complexity, achieving low latency and reduced power consumption while maintaining acceptable accuracy for machine learning applications

Inventive Principle:

Principle #1Segmentation

Solution Approach 2:

Instead of using a large, permanent look-up table that occupies significant circuit area, the system uses lightweight conditional logic to select between different linear approximation functions. This disposable approach computes the result on-the-fly using simple arithmetic operations rather than storing pre-computed values in a large memory structure

Inventive Principle:

Principle #27Cheap short-living objects (Disposable)

Data Source

PatentUS11327923B2Sigmoid function in hardware and a reconfigurable data processor including same

Publication Date: 2022.05.10 SAMBANOVA SYSTEMS INC

AI summary

A functional unit for a data processor comprises an input register to store a variable X; a first circuit, having an input connected to the input register and an output, to generate a value eX on its output; a second circuit, having an input connected to the input register and an output, to generate an output which is a value (tan h(X/2)+1)/2 on its output; a comparator, having an input connected to the input register and an output, to generate a line on its output based on a comparison between X and a constant; and a selector to select between inputs connected to the outputs of the first circuit and the second circuit, in response to the output of the comparator, and provide an output representing a value sigmoid(X).