Integer Activation Function Circuit for Low-Power Neural Hardware
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing machine learning technologies face challenges in efficiently implementing non-linear activation functions on devices with limited power, space, and speed resources, particularly in dedicated devices like ASICs and FPGAs, leading to accuracy and usefulness limitations.
Innovation Solution
Implementing activation functions using integer representation and combinatorial logic, employing oversampling and binary filtering operations, which can be processed without floating-point units, allowing for efficient implementation on devices like ASICs and FPGAs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If floating-point units are used to implement activation functions, then computational accuracy is improved, but device complexity and power consumption increase
Solution Approach 1:
The patent substitutes floating-point arithmetic operations with integer arithmetic operations and combinatorial logic circuits. Specifically, it replaces complex floating-point units with simpler integer adders, shifters, and logic gates that can implement activation functions like sigmoid and tanh through bit-manipulation operations, thereby reducing device complexity while maintaining acceptable accuracy
Solution Approach 2:
The patent changes the numerical representation parameter from floating-point to fixed-point integer format. By representing neural network weights and activations as integers with specific bit widths (e.g., 8-bit or 16-bit integers), the system eliminates the need for floating-point hardware while preserving computational accuracy through careful quantization and scaling operations
2Measurement precision
If floating-point units are used to implement activation functions, then computational accuracy is improved, but power consumption increases
Solution Approach 1:
The patent replaces power-hungry floating-point arithmetic units with energy-efficient integer arithmetic circuits. The implementation uses simple integer adders, subtractors, and bit-shift operations that consume significantly less power than floating-point units, making the system suitable for mobile and embedded devices with limited power budgets
Solution Approach 2:
The patent employs approximate computation methods that sacrifice minimal accuracy for substantial power savings. By using integer-based activation function approximations rather than precise floating-point calculations, the system accepts small accuracy trade-offs in exchange for dramatically reduced power consumption, enabling deployment on battery-powered devices
3Device complexity
If integer representation and combinatorial logic are used, then resource requirements are reduced, but implementation complexity increases
Solution Approach 1:
The patent segments the activation function computation into discrete bit-level operations that can be independently implemented. By breaking down the sigmoid and tanh functions into sequences of integer additions, shifts, and logical operations, the design becomes modular and easier to manufacture using standard CMOS logic cells, reducing overall implementation complexity
4Device complexity
If integer representation and combinatorial logic are used, then resource requirements are reduced, but accuracy may be compromised
Solution Approach 1:
The patent carefully adjusts the integer representation parameters (bit width, scaling factors, and quantization levels) to maintain accuracy. By using sufficient bit widths (e.g., 16-bit integers) and appropriate scaling, the system achieves results that closely match floating-point computations while using only integer hardware, thus preserving accuracy while reducing resource requirements
Data Source
Figure 1~4
Figure 2
Figure 3
AI summary
An apparatus and method is disclosed, the apparatus comprising means for: receiving first input data at a sample rate ƒs, the first data representing fixed-length integer values from part of a machine learning model and providing second input data at an oversampling rate Nƒs, wherein N is an integer greater than one, the second input data representing a sequence of non-repeating integers. The apparatus may further comprise means for summing the first and second input data to produce first output data, wherein the summed output is limited within a predetermined first output range C, subtracting the second input data from the first output data to produce second output data, wherein the second output data is limited within a predetermined second output range and processing the second output data to perform a binary filtering operation to produce third output data. Symmetric and Asymmetric embodiments are disclosed.