Neural Network Activation Circuits With Parallel Function Multiplexing

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Existing neural networks rely on software-based computation of activation functions, which is computationally inefficient and high-power consumption.

Innovation Solution

Implementing parallel compute circuits with a multiplexor to compute activation functions efficiently, using dedicated hardware for specific activation functions and caching weights and biases for quick switching between activation functions.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Productivity

If software-based computation is used for activation functions, then implementation flexibility is maintained, but computational efficiency and power consumption deteriorate

Engineering Contradiction:

Improvecomputational efficiencyVSAvoidpower consumption

Core Design Contradiction:

ProductivityVSUse of energy by moving object

Solution Approach 1:

The patent replaces software-based computation (mechanical/electronic processing) with dedicated hardware circuits (ASICs, FPGAs, or neural processing units) that are specifically designed to compute activation functions. This substitution of computational paradigm achieves significant improvements in computational efficiency and power consumption by utilizing specialized hardware architecture optimized for the specific mathematical operations required by neural network activation functions.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Solution Approach 2:

The patent implements reconfigurable hardware circuits that can dynamically change their operational parameters and configuration to support different activation functions. By using programmable logic devices or reconfigurable neural processing units, the system can alter its computational behavior to match different neural network requirements while maintaining the performance benefits of hardware acceleration.

Inventive Principle:

Principle #35Parameter changes

2Speed

If dedicated hardware circuits are implemented for each activation function, then computational speed improves, but device complexity increases

Engineering Contradiction:

Improvecomputation speedVSAvoidcircuit complexity

Core Design Contradiction:

SpeedVSDevice complexity

Solution Approach 1:

The patent implements universal hardware circuits that can perform multiple activation function computations through reconfiguration. Instead of creating separate dedicated circuits for each activation function, the system uses a single reconfigurable hardware architecture that can be programmed or configured to implement different activation functions (such as ReLU, sigmoid, tanh, softmax) as needed, thereby reducing overall device complexity while maintaining high computation speed.

Inventive Principle:

Principle #6Universality (Multi-functionality)

Solution Approach 2:

The patent employs dynamic reconfiguration capabilities in the hardware circuits, allowing the system to adapt its structure and operation in real-time based on the required activation function. This dynamic approach enables the hardware to transform between different computational modes, providing the flexibility of multiple specialized circuits while maintaining a compact, unified physical implementation.

Inventive Principle:

Principle #15Dynamics

3Adaptability or versatility

If multiple activation functions are supported with switching capability, then model flexibility improves, but control complexity increases

Engineering Contradiction:

Improveactivation function switching flexibilityVSAvoidcontrol signal complexity

Core Design Contradiction:

Adaptability or versatilityVSDevice complexity

Solution Approach 1:

The patent implements preliminary configuration mechanisms where the desired activation function is selected and configured before the computational task begins. By pre-configuring the hardware circuit with the appropriate activation function parameters and settings, the system avoids the need for complex real-time switching control during computation, thereby reducing control complexity while maintaining the ability to support multiple activation functions for different neural network models.

Inventive Principle:

Principle #10Preliminary action

Data Source

PatentUS12417388B1Hardware implementations of activation functions in neural networks

Publication Date: 2025.09.16 NATARAJ BINDIGANAVALE S

AI summary

Circuitry for performing neural-network calculations includes a plurality of compute circuits, arranged in parallel with respective inputs and outputs, to receive function arguments for a node of a neural network on their respective inputs, compute values of a plurality of activation functions using the function arguments, and provide the values on their respective outputs. Each compute circuit of the plurality of compute circuits is to compute the values of a respective activation function of the plurality of activation functions. The circuitry also includes a multiplexor to select between the respective outputs of the plurality of compute circuits and to provide the values on a selected output as activation-function values for the node of the neural network, based on an activation-function selection signal.