Quantized Softmax Layer Using Distance-Based Lookup Table
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep learning neural networks require significant memory and processing resources for softmax operations, especially when using floating-point data types, which can exceed the capabilities of end devices like phones and cameras, and previous attempts to simplify lookup tables have resulted in increased computational costs and memory usage.
Innovation Solution
Implementing a quantized softmax layer with a single compact lookup table that uses a distance-based index instead of separate tables for each input size, reducing memory and processing resources by removing redundant entries and using fixed-point data types, which degrades accuracy by less than 1% while significantly reducing resource usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If floating-point data types are used for softmax operations, then accuracy is maintained, but memory and processing resources are excessively consumed
Solution Approach 1:
The patent changes the data type parameter from floating-point to fixed-point representation, and transforms the softmax calculation into a lookup table-based approach. This parameter change reduces memory usage from multiple gigabytes to less than 100 kilobytes while maintaining acceptable accuracy through quantization and distance-based indexing.
2Measurement precision
If separate lookup tables are created for each input size, then accuracy is improved, but device complexity and memory usage increase
Solution Approach 1:
The patent merges multiple input-specific lookup tables into a single unified lookup table by using distance-based indexing. Instead of having separate tables for different input sizes, the system calculates distances from maximum values and uses these distances as indices into one compact table, reducing complexity while preserving accuracy.
Solution Approach 2:
The patent introduces a new dimension for indexing by using distance from maximum value instead of direct input value indexing. This dimensional transformation allows a single lookup table to serve multiple input sizes by indexing based on the relative distance from the maximum element, rather than requiring separate tables for each input configuration.
3Quantity of substance
If lookup tables are simplified to reduce memory usage, then resource consumption decreases, but computational costs increase
Solution Approach 1:
The patent performs preliminary actions by pre-calculating and storing softmax values in a compact lookup table during an offline phase. The distance-based indexing structure is pre-constructed, allowing runtime operations to simply compute distances and perform table lookups, thereby reducing online computational costs while maintaining low memory usage.
Data Source
AI summary
Quantized softmax layers in neural networks are described. Some embodiments involve receiving, at an input to a softmax layer of a neural network from an intermediate layer of the neural network, a non-normalized output comprising a plurality of intermediate network decision values. Then for each intermediate network decision value of the plurality of intermediate network decision values, the embodiment involves: calculating a difference between the intermediate network decision value and a maximum network decision value; requesting, from a lookup table, a corresponding lookup table value using the difference between the intermediate network decision value and the maximum network decision value; and selecting the corresponding lookup table value as a corresponding decision value. A normalized output is then generated comprising the corresponding lookup table value for said each intermediate network decision value of the plurality of intermediate network decision values.