Graphics Chiplets With Metadata-Driven Training Datatype Adjustment
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network training often faces a trade-off between accuracy and storage/compute costs due to the use of different datatypes, with lower precision weights and activations potentially improving memory bandwidth and compute efficiency but not fully addressed in existing systems.
Innovation Solution
Implementing dynamic datatype adjustment during neural network training to optimize precision based on computational needs, using techniques such as single instruction multiple thread (SIMT) architectures and programmable graphics processors to enhance parallel processing efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If lower precision weights and activations are used in neural network training, then memory bandwidth requirements and compute efficiency are improved, but accuracy may deteriorate
Solution Approach 1:
The system dynamically adjusts the precision of weights and activations during neural network training based on computational needs. Different layers or operations can use different precision levels (e.g., FP16, BF16, INT8, or FP32) rather than fixing precision for the entire network, allowing optimization of compute efficiency while maintaining necessary accuracy where required
Solution Approach 2:
The patent changes the precision parameter of data types used in computations. By selecting appropriate precision levels (such as half-precision floating point, bfloat16, or integer types) for different computational stages, the system improves memory bandwidth utilization and compute efficiency while managing the trade-off with accuracy through careful parameter selection
2Quantity of substance
If lower precision datatypes are used during neural network training, then memory bandwidth requirements are reduced, but storage and compute costs trade off against accuracy
Solution Approach 1:
The system changes the precision parameter of data types from standard FP32 to lower precision formats such as FP16, BF16, or INT8. This reduction in bit-width directly decreases the quantity of data that needs to be stored and transmitted through memory bandwidth, while computational efficiency improves due to smaller data sizes
Solution Approach 2:
The system dynamically selects appropriate precision levels for different operations and layers during training. Rather than uniformly applying low precision throughout, the system adapts precision requirements based on the specific computational needs of each layer, maintaining accuracy where necessary while reducing memory bandwidth requirements where possible
Data Source
AI summary
One embodiment provides a graphics processor comprising a base die including a plurality of chiplet sockets and a plurality of chiplets coupled with the plurality of chiplet sockets. At least one of the plurality of chiplets including first circuitry configured to perform a multi-dimensional matrix multiply accumulate operation to facilitate training for a neural network and second circuitry configured to perform dynamic datatype adjustment during the training for the neural network. The dynamic datatype adjustment is performed based on statistics generated based on output of the multi-dimensional matrix multiply accumulate operation, neural network model metadata, and training metadata associated with training operations to be performed for the neural network.