Dynamic Precision Adjustment for Neural Network Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks require high computational resources for inference, leading to increased system costs, making them unaffordable for organizations that cannot support the hardware demands, especially since the high number of computations necessary for generating predicted output values is resource-intensive.
Innovation Solution
The method involves reducing the computational intensity of neural network nodes by representing node input, intermediate, and output values in a different floating-point format with less precision during inference, without retraining, by selecting criteria to reduce the number of layers, nodes, and available numerical formats, thereby reducing the number of iterations needed to determine which values can be represented with less precision.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high precision floating-point format is used for neural network inference, then accuracy is maintained, but computational resource requirements increase
Solution Approach 1:
The patent changes the precision parameter of floating-point representations during inference, allowing dynamic adjustment between high precision (for accuracy) and low precision (for reduced computational resources). This enables the system to adapt precision levels based on specific inference scenarios, resolving the contradiction between maintaining accuracy and reducing resource consumption.
Solution Approach 2:
The system dynamically adjusts the precision of floating-point operations during inference based on input characteristics and required output accuracy. Rather than using a fixed precision level, the system can switch between different precision modes, making the computational resource usage adaptive while maintaining sufficient accuracy for each specific case.
2Measurement precision
If high precision floating-point format is used for all node values, then inference accuracy is maintained, but the number of computations increases
Solution Approach 1:
The patent applies different precision levels to different parts of the neural network computation based on their importance and sensitivity to precision changes. Critical nodes and layers maintain high precision to preserve accuracy, while less sensitive nodes use lower precision to reduce computational overhead, achieving a local optimization strategy.
Solution Approach 2:
Instead of applying uniform high precision to all computations, the system applies high precision only where necessary for maintaining accuracy. This partial application of high precision reduces the overall number of high-cost computations while preserving prediction accuracy through selective precision management.
3Reliability
If full precision is maintained throughout the neural network, then output accuracy is preserved, but hardware resource demands increase
Solution Approach 1:
The system changes the precision parameter of data representations during inference, allowing flexible switching between high and low precision modes. This enables hardware resources to be reduced by using lower precision where acceptable, while maintaining high precision for critical operations that require it for reliability.
Solution Approach 2:
The neural network computation is segmented into different precision zones or stages, where different parts of the computation pipeline use different precision levels. This segmentation allows hardware resources to be optimized by matching precision requirements to actual computational needs at each stage.
Data Source
AI summary
An apparatus and method for efficiently creating less computationally intensive nodes for a neural network. In various implementations, a computing system includes a processor and a memory with circuitry that stores multiple input data values to process during inference of a trained neural network. The processor determines, during inference, which node input values, node intermediate values, and node output values of the trained neural network to represent in a respective one of multiple available floating-point formats with less precision. No retraining is performed, but rather, the updates to the representations occur during inference. The processor uses selection criteria to reduce the amount of computation involved for updating the representations during inference while maintaining accuracy above an accuracy threshold. To do so, the processor uses the selection criteria to reduce the number of layers, the number of nodes within a layer, and the number of weight values per node to inspect.