Variable Precision Weight Management for Neural Network Memory Bandwidth
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Memory bandwidth limitations in neural networks due to the large memory storage and transfer requirements of data and weights, which hinder the full potential throughput of processing apparatuses.
Innovation Solution
Implementing variable precision weight management by separating or intermingling weights of different precisions in memory segments, with metadata to identify precision, allowing for efficient storage and processing using mixed precision weights.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If all weights are stored with full precision to maintain model accuracy, then measurement precision is improved, but memory storage requirements and memory bandwidth consumption increase
Solution Approach 1:
The patent applies local quality by assigning different precision levels to different weights based on their individual importance and contribution to model accuracy. Instead of uniformly storing all weights at full precision, the system analyzes each weight's significance and stores only the necessary precision level, thereby reducing overall memory storage requirements while maintaining model performance.
Solution Approach 2:
The patent changes the precision parameter of weights dynamically based on their importance metrics. By adjusting the precision parameter from full precision to reduced precision for less critical weights, the system optimizes the balance between memory storage efficiency and model accuracy, directly addressing the contradiction between precision and storage requirements.
2Measurement precision
If all weights are transferred with full precision to processing apparatus, then processing accuracy is improved, but memory bandwidth consumption increases
Solution Approach 1:
The patent applies local quality to the weight transfer process by selectively transferring weights at different precision levels according to their importance. Critical weights are transferred with full precision to maintain processing accuracy, while less critical weights are transferred at reduced precision, thereby reducing overall memory bandwidth consumption without significantly impacting model performance.
Solution Approach 2:
The patent dynamically changes the precision parameter during weight transfer based on importance analysis. This allows the system to optimize memory bandwidth utilization by transferring only the necessary precision level for each weight, improving overall productivity while maintaining processing accuracy for critical operations.
3Reliability
If high precision weights are used throughout the network, then model accuracy is improved, but computational resources and memory bandwidth are consumed
Solution Approach 1:
The patent applies local quality by identifying and maintaining high precision only for weights that critically impact model accuracy, while using reduced precision for other weights. This selective approach preserves model reliability for important computations while reducing overall computational resource consumption and energy usage across the neural network.
Solution Approach 2:
The patent changes the precision parameter of weights based on their importance to model accuracy. By dynamically adjusting precision levels, the system optimizes the trade-off between model reliability and computational resource consumption, ensuring high accuracy where needed while reducing energy usage in less critical areas.
Data Source
AI summary
An approach includes identification of a machine learning model for processing and generating an ordered set of weights with varying precisions and metadata that specifies where those values can be found in order to allow the identification of weights needed during processing. In a first embodiment, the variable precision weights are separated into different memory segments where each segment has weights of only a single precision. In a second embodiment, the variable precision weights are provided in a memory where weights of different precisions are intermingled, and those weights are identified using a sequence of pairs of data representing a number of weights with the same precision and the precision of those weights. In some embodiments, both the first and second embodiments are combined, where some segments contain weights with only a single precision and at least one segment stores weights with different precisions within a respective segment.