Product-Sum Circuit Power-of-2 Quantization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for reducing the processing load in neural networks, such as binarization and log domain conversion, face challenges with quantization granularity and efficiency, particularly as the number of dimensions increases, leading to performance degradation.
Innovation Solution
An information processing apparatus and method that performs product-sum operations using input values and weight coefficients quantized by power expressions, with exponents represented as fractions, allowing for efficient multiplication using addition operations and maintaining granular quantization.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If binarization using +1 or -1 is performed to reduce processing load, then the processing load is reduced, but the quantization granularity becomes rough as the number of dimensions of the weigh coefficient increases
Solution Approach 1:
The patent changes the parameter representation from binary values (+1, -1) to power-of-2 values (2^0, 2^1, 2^2, ...). This parameter transformation allows the system to maintain coarse quantization (few distinct values) while improving the effective granularity through the exponential spacing of power-of-2 values, which better preserves relative magnitude relationships in high-dimensional spaces.
Solution Approach 2:
The patent introduces a new dimensional aspect by using exponential representation (power-of-2 scaling) rather than linear binary values. This transforms the quantization from a linear scale to an exponential scale, effectively adding a logarithmic dimension to the value representation that improves granularity perception while maintaining computational efficiency.
2Device complexity
If log domain conversion is used to convert multiplication into addition, then multiplication is avoided, but there is more room for reducing the processing load
Solution Approach 1:
The patent extracts the multiplication operation entirely by representing all values in power-of-2 form. Instead of converting multiplication to addition via log domain (which requires antilog operations), the system directly computes products as sums of exponents, eliminating the need for complex log/antlog conversions and enabling simpler hardware implementation.
Solution Approach 2:
The patent substitutes the mechanical multiplication operation with a simpler exponential addition operation. By representing values as 2^x, the product of two values becomes 2^(x+y), replacing complex multiplication circuitry with simple addition circuits, thereby reducing device complexity and increasing processing efficiency.
3Measurement precision
If power expression quantization with fractional exponents is used, then quantization granularity is maintained in high-dimensional spaces, but the computational complexity increases
Solution Approach 1:
The patent uses fractional exponents with a fixed denominator (e.g., 2^(n/4) where n is an integer) to represent quantized values. This parameterization allows the system to maintain fine granularity control while enabling efficient computation through pre-computed lookup tables and simple integer arithmetic, avoiding the need for complex floating-point operations.
Solution Approach 2:
The patent performs preliminary quantization of weight coefficients and input values to power-of-2 form before the actual product-sum operation. By pre-computing and storing the exponent values in a standardized format, the system reduces the computational complexity during the actual neural network inference, as the expensive multiplication operations have already been transformed into simple exponent additions.
Data Source
AI summary
An apparatus includes: a product-sum operation circuit that executes a product-sum operation with a plurality of input values quantized by power expression and plurality of weigh coefficients quantized by power expression corresponding to respective input values, in which an exponent of each of the input values is expressed by a fraction having a predetermined divisor in a denominator, an exponent of each of the weigh coefficients is expressed by a fraction having the divisor in a denominator, the product-sum operation circuit executes the product-sum operation using a plurality of addition multipliers based on a remainder when a value obtained by adding a numerator related to the exponent of each of the input values and a numerator related to the exponent of each of the weigh coefficients is divided as a dividend, and each of the addition multipliers is a floating-point number with an exponent part having a radix of 2.