Shared Lookup Table Circuit for Multiple Neural Activation Functions

Resolve Bottlenecks,
Find Innovative SolutionsGenerate Solutions

Solution Overview

Problem

Deep neural networks face challenges in processing activation functions efficiently at reduced memory costs, as existing methods often require multiple lookup tables for different activation functions, increasing memory usage.

Innovation Solution

Implementing a shared lookup table that processes multiple activation functions by converting input values and function values between different activation functions using predetermined address and function value conversion rules, reducing the need for separate lookup tables and thus lowering memory costs.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If multiple separate lookup tables are used for different activation functions, then processing accuracy for each activation function is maintained, but memory costs increase

Engineering Contradiction:

Improveprocessing accuracyVSAvoidmemory costs

Core Design Contradiction:

Measurement precisionVSQuantity of substance

Solution Approach 1:

The patent implements a shared lookup table that can serve multiple activation functions (Sigmoid, Tanh, ReLU) by converting their input values to a common reference function's input space. This allows a single lookup table to perform the function of multiple separate tables, reducing memory requirements while maintaining processing accuracy through mathematical conversion relationships between different activation functions.

Inventive Principle:

Principle #6Universality (Multi-functionality)

2Quantity of substance

If a shared lookup table is used for multiple activation functions, then memory costs are reduced, but processing complexity increases due to value conversion requirements

Engineering Contradiction:

Improvememory costsVSAvoidprocessing complexity

Core Design Contradiction:

Quantity of substanceVSDevice complexity

Solution Approach 1:

The patent introduces a reference activation function as an intermediary that mediates between different activation functions. By converting all activation function inputs to the reference function's input space and using corresponding conversion relationships, the system simplifies the overall structure despite the mathematical transformations required. The reference function acts as a common denominator that enables sharing while managing complexity.

Inventive Principle:

Principle #24Intermediary (Mediator)

3Speed

If separate lookup tables are used for each activation function, then processing speed is maintained through direct lookup, but device complexity increases

Engineering Contradiction:

Improveprocessing speedVSAvoiddevice complexity

Core Design Contradiction:

SpeedVSDevice complexity

Solution Approach 1:

The patent merges multiple separate lookup tables into a single shared lookup table by establishing mathematical relationships between different activation functions. This combining approach reduces device complexity by eliminating redundant storage structures while maintaining processing speed through efficient conversion and lookup operations. The unified structure reduces memory access overhead and simplifies the overall system architecture.

Inventive Principle:

Principle #5Merging (Combining)

Data Source

PatentUS11928575B2Activation function processing method, activation function processing circuit, and neural network system including the same

Publication Date: 2024.03.12 SK HYNIX INC

AI summary

An activation function processing method includes processing a first activation function in a first mode by referring to a shared lookup table that includes a plurality of function values of the first activation function; and processing a second activation function in a second mode by referring to the shared lookup table, the second activation function being a different function than the first activation function.