Neural Network Configuration for Bit-Identical Hardware Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks face challenges in efficiently training on inference hardware due to hardware-specific errors and stochastic elements, leading to over-saturation or under-saturation effects, which are not reproducible and require extensive study to understand.
Innovation Solution
A method to configure a neural network by ascertaining noise parameters to ensure bit-identical output between training and inference hardware, using error ascertainment and back propagation to model noise distributions, allowing training on training hardware without considering rounding errors.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If neural networks are trained offline on training hardware with floating point computing, then training accuracy is improved, but hardware cost and complexity increase
Solution Approach 1:
The system separates training operations from inference operations by using different hardware types. Training is performed on powerful floating-point training hardware, while inference is performed on simpler fixed-point inference hardware. This segmentation allows each component to be optimized for its specific function, reducing overall system complexity and cost.
Solution Approach 2:
The patent creates a virtual model of the inference hardware's fixed-point arithmetic behavior and incorporates it into the training process. This copying approach allows training to account for inference hardware characteristics without requiring the training hardware to physically replicate the inference hardware, thus maintaining training accuracy while avoiding hardware duplication costs.
2Device complexity
If fixed point values are used in inference hardware to save chip surface and operating costs, then device complexity is reduced, but output precision deteriorates due to rounding errors
Solution Approach 1:
The patent applies preliminary quantization and rounding operations during the training phase on floating-point hardware to simulate the fixed-point arithmetic behavior of inference hardware. This preliminary action allows the training process to learn and compensate for the precision limitations of fixed-point arithmetic, ensuring that the trained model produces accurate results when deployed on resource-constrained inference hardware.
Solution Approach 2:
The system changes the numerical representation parameters by using different precision levels for training versus inference. Training uses higher-precision floating-point values, while inference uses lower-precision fixed-point values. The training process is modified to account for this parameter change, ensuring consistency between training and deployment conditions.
3Stability of the object's composition
If noise parameters are ascertained to ensure bit-identical output between training and inference hardware, then output consistency is improved, but training complexity increases
Solution Approach 1:
The patent implements a feedback mechanism where the quantization error between floating-point training outputs and simulated fixed-point inference outputs is calculated and used to adjust the training process. This feedback loop continues iteratively until bit-identical output is achieved, ensuring consistency while automating the complexity management through systematic error correction.
Solution Approach 2:
The patent replaces the need for physical inference hardware during training with a virtual model that simulates fixed-point arithmetic behavior. This substitution allows the training process to account for inference hardware characteristics without the complexity of physical hardware-in-the-loop testing, achieving output consistency through software-based simulation and error modeling.
Data Source
AI summary
A method for configuring a neural network. The method includes: feeding image data to the neural network implemented on a training hardware; feeding the image data to a neural network implemented on an inference hardware; ascertaining a deviation between output data of the training hardware and output data of the inference hardware; and ascertaining noise parameters for the neural network in such a way that after feeding the image data to the neural network implemented on the training hardware and after feeding image data to the neural network implemented on the inference hardware, the output data of the inference hardware and the output data of the training hardware are bit-identical.