Quantized NNP Matrix-Vector Operations for Concordant Software Layers
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks face discrepancies in results due to differences in precision formats between software and hardware implementations, leading to errors over time.
Innovation Solution
A method and system for generating concordant software instructions that match the operations performed on hardware/firmware by quantizing data and using a neural network processor to perform matrix-vector multiply operations, ensuring consistent results across different precision formats.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If quantized data is used in hardware neural network processors, then computation speed and efficiency are improved, but precision discrepancies and result inconsistencies between software and hardware implementations occur
Solution Approach 1:
The patent applies parameter changes by systematically modifying the precision format parameters in software to match hardware quantization behavior. It transforms software from using high-precision floating-point formats to using quantized integer formats with specific bit widths (e.g., 8-bit, 16-bit) and scaling factors, thereby aligning computational parameters between software and hardware to eliminate precision discrepancies while maintaining computation speed benefits
Solution Approach 2:
The patent introduces an intermediary translation layer that converts hardware quantized operations into equivalent software operations. This intermediary mechanism includes mapping functions that translate hardware-specific quantized matrix-vector multiply operations into software-compatible forms, ensuring that software simulations accurately reflect hardware behavior without requiring actual hardware execution
2Productivity
If different precision formats are used between software and hardware, then hardware-specific optimizations are achieved, but errors accumulate over time in neural network computations
Solution Approach 1:
The patent applies preliminary action by pre-determining the quantization parameters (bit width, scaling factors, zero-point values) before neural network execution. It performs ahead-of-time conversion of weight matrices and activation functions into quantized formats, and pre-computes scaling factors that will be used during inference, thereby eliminating runtime precision errors and ensuring consistent results across software and hardware platforms
Solution Approach 2:
The patent implements feedback mechanisms that compare software simulation results with hardware execution results to detect and correct precision drift. It uses reference models that run in software with known correct answers to validate hardware quantized computations, and adjusts quantization parameters based on observed errors to maintain computational accuracy over time
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Systems and methods for deriving a concordant software neural network layer are provided. A method includes receiving first instructions configured to, using a neural network processor (NNP), process a first set of data corresponding to a neural network layer, where the NNP is configured to quantize the first set of the data to generate a set of quantized data and then perform matrix-vector multiply operations on the set of quantized data using a matrix-vector-multiplier incorporated within hardware associated with the NNP to generate a first set of results. The method further includes processing the first instructions to automatically generate second instructions configured for use with at least one processor, different from the NNP, such that the second instructions, when executed by the at least one processor to perform matrix multiply operations, generate a second set of results that are concordant with the first set of results.