N-Hot Quantization for DNN Coefficient Decomposition
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep neural networks (DNNs) face a high processing load due to large calculation amounts, leading to decreased recognition rates, particularly because coefficients close to zero are densely expressed while those near the maximum value are sparsely expressed, affecting accuracy.
Innovation Solution
An information processing apparatus that expresses either the input value or weight coefficient using a combination of addition or subtraction of n or less first bit strings, where each bit string includes one '1', reducing the calculation load while maintaining recognition accuracy by decomposing coefficients into 'n-hot' quantization values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If coefficients are expressed using logarithmic calculation to reduce processing load, then calculation amount is reduced, but coefficients close to maximum value are sparsely expressed causing recognition rate to decrease
Solution Approach 1:
The patent segments the coefficient representation into multiple bit strings, where each bit string represents a specific power of 2. This segmentation allows the system to express coefficients as sums of these bit strings, enabling both efficient calculation (by selecting only necessary bit strings) and precise representation (by combining multiple bit strings) simultaneously.
Solution Approach 2:
The patent applies different representation strategies to different ranges of coefficient values. By using n-hot encoding where each bit string corresponds to a specific power of 2, the system provides dense representation for coefficients that require multiple bit strings and sparse representation for coefficients that require fewer bit strings, optimizing both precision and calculation efficiency for different value ranges.
2Measurement precision
If the number of coefficients is increased to achieve high recognition rate, then recognition rate is improved, but calculation amount increases leading to high processing load
Solution Approach 1:
The patent uses n-hot encoding where n is greater than 1, allowing each coefficient to be represented by multiple bit strings. This partial redundancy enables the system to use fewer total coefficients to achieve the same recognition rate, as each coefficient can be more efficiently represented and processed through the bit string decomposition method.
Solution Approach 2:
The patent changes the parameter representation from traditional floating-point or integer formats to a binary decomposition format using multiple bit strings. This parameter transformation allows the system to maintain recognition accuracy while reducing the computational complexity of coefficient operations, as addition and subtraction become simpler operations on the bit string representations.
Data Source
AI summary
An information processing apparatus (10, 10A, and 10B) includes a calculation unit (120, 120A, and 120B) that performs integration of an input value and a weight coefficient. Any one of the input value and the weight coefficient is expressed by a combination of addition or subtraction of n or less first bit strings. The first bit strings include one bit of 1.