Neural Networks Using Lookup Tables for Discrete Weights
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks require significant computing resources and memory due to their large number of parameters, making them unsuitable for resource-constrained devices such as hearing aids.
Innovation Solution
The neural network is structured and trained to use discrete activation values and weight values, replacing multiplication operations with lookup table accesses, and converting values to fixed-point representation to eliminate floating-point operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If a deep neural network with many parameters is used to achieve high accuracy, then the network performance is improved, but the memory usage and computing resources required increase significantly
Solution Approach 1:
The patent changes the parameter representation from continuous floating-point values to discrete quantized values. Weights and activation functions are restricted to discrete sets of values, enabling compact storage in lookup tables while preserving network accuracy. This parameter transformation resolves the contradiction by reducing memory requirements without significantly degrading network performance.
Solution Approach 2:
The patent creates simplified copies of the neural network computation process using lookup tables instead of actual multiplication operations. The lookup tables store pre-computed results for discrete weight and activation value combinations, replacing the need to store and process large numbers of continuous parameters. This copying approach maintains accuracy while dramatically reducing memory usage.
2Measurement precision
If floating point multiplication operations are used to achieve precise computations, then the calculation accuracy is improved, but the computing resource consumption increases
Solution Approach 1:
The patent substitutes the mechanical floating-point multiplication operation with a lookup table access operation. Instead of performing actual multiplication computations that require floating-point units and significant processing power, the system retrieves pre-computed values from lookup tables based on discrete weight and activation value indices. This substitution dramatically reduces computing resource consumption while maintaining calculation accuracy through the use of discrete value sets.
Solution Approach 2:
The patent performs multiplication operations in advance during the lookup table construction phase. All possible products of discrete weight values and activation values are pre-computed and stored in the lookup table. During network inference, these pre-computed values are simply retrieved rather than calculated in real-time, eliminating the need for floating-point multiplication operations during actual computation and significantly reducing processing requirements.
3Measurement precision
If a large number of weights with unique values are stored to maintain network accuracy, then the model precision is improved, but the storage requirements increase
Solution Approach 1:
The patent transforms the weight parameter from continuous floating-point values to discrete quantized values drawn from a small finite set. This parameter change enables weights to be stored as integer indices rather than full-precision floating-point numbers. Combined with the lookup table approach, this dramatically reduces storage requirements while maintaining model precision through the structured discretization of the weight space.
Data Source
AI summary
The present disclosure provides systems and methods that train and use neural networks that can be run with no multiplications and no floating point operations. In particular, according to one aspect of the present disclosure, the respective non-linear and continuous activation functions typically used by the nodes of a neural network can be replaced with custom activation functions that output one of a discrete number of activation values. Likewise, according to another aspect of the present disclosure, the neural network can be trained such that each of its weights equals one of a discrete number of weight values. Taken together, this enables replacement of the typical multiplication process associated with computing a node of the network with a simple, and much faster, lookup process. In particular, a lookup table can store the result of multiplying each unique pair of activation value and weight value.