Neural Network Partial Sum Compression via Exponent Mantissa Encoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network operations, such as calculating convolutions, are burdensome due to high storage and bandwidth requirements for partial sums in processing circuits.
Innovation Solution
A computer-implemented method and system that compresses partial sums during neural network inference operations by converting them into a compressed format using a customized encoding method, specifically representing partial sums as integers with an exponent and mantissa, reducing the bit width and storage needs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If partial sums are stored in conventional formats during neural network inference, then calculation accuracy is maintained, but storage requirements and bandwidth consumption increase significantly
Solution Approach 1:
The patent changes the representation parameters of partial sums by converting them from conventional fixed-point or floating-point formats to a customized compressed format with reduced precision. This involves modifying the bit-width and numerical representation parameters to achieve compression while maintaining acceptable accuracy for neural network inference operations.
Solution Approach 2:
The patent applies partial compression to partial sums by selectively compressing only those partial sums that meet certain criteria (e.g., magnitude thresholds or distribution characteristics). This allows the system to achieve significant storage reduction for the majority of partial sums while maintaining full precision for edge cases, thus balancing storage efficiency with calculation accuracy.
2Measurement precision
If partial sums are stored in conventional formats during neural network inference, then calculation accuracy is maintained, but bandwidth for data transfer increases
Solution Approach 1:
The patent reduces bandwidth consumption by changing the data representation parameters of partial sums from full-precision formats to compressed formats with fewer bits. This parameter transformation enables the same computational accuracy to be achieved with significantly reduced data transfer requirements between memory and processing units.
Solution Approach 2:
The patent selectively applies compression to partial sums based on their importance and distribution, transferring only compressed versions of less critical partial sums across the system boundary. This partial action approach reduces overall bandwidth consumption while maintaining sufficient accuracy for the majority of computational operations.
3Quantity of substance
If partial sums are compressed to reduce storage and bandwidth, then storage requirements and bandwidth are reduced, but calculation accuracy may be compromised
Solution Approach 1:
The patent employs customized encoding methods that change the numerical representation parameters of partial sums in a controlled manner. By adjusting parameters such as bit-width, rounding modes, and representation formats, the system achieves compression while preserving the essential numerical properties needed for accurate neural network inference.
Solution Approach 2:
The patent applies partial compression strategies where full precision is maintained for critical partial sums (e.g., those exceeding threshold values or belonging to important computational paths) while compressed representation is used for less critical partial sums. This selective approach ensures that accuracy is preserved where needed while achieving significant storage reduction overall.
Data Source
Figure 1A
Figure 1B
Figure 1C
AI summary
A method for performing a neural network operation. In some embodiments, method includes: calculating a first plurality of products, each of the first plurality of products being the product of a weight and an activation; calculating a first partial sum, the first partial sum being the sum of the products; and compressing the first partial sum to form a first compressed partial sum.