Adaptive Floating-Point Containers for Low-Memory ANN Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training of artificial neural networks is computationally and memory-intensive due to the need for off-chip memory accesses for storing and retrieving activation and weight tensors, with existing methods relying on static selection of data types and requiring trial-and-error to achieve convergence, leading to suboptimal reductions in data movement.
Innovation Solution
Adapting floating-point containers by dynamically adjusting mantissa and exponent bitlengths using gradient descent and loss functions to optimize data representation for each tensor and layer, allowing the network to learn the minimal bitlengths required for training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If single precision 32-bit floating-point (FP32) is used for training, then accuracy is improved, but memory usage and computational cost increase
Solution Approach 1:
The patent implements dynamic floating-point formats where the mantissa and exponent bitlengths are not fixed but adapt during training. The system determines optimal bitlengths using gradient descent and adjusts the precision of floating-point representations dynamically based on the training progress and tensor characteristics, allowing the model to use fewer bits when high precision is not needed while maintaining accuracy when required
Solution Approach 2:
The patent changes the parameters of the floating-point representation system by treating the mantissa bitlength and exponent bitlength as learnable parameters. Through gradient descent optimization, the system automatically determines the optimal number of bits required for each tensor, transforming the static parameter selection into a dynamic optimization process that balances precision and memory efficiency
2Quantity of substance
If off-chip DRAM accesses are used to store activation tensors, then memory capacity is improved, but execution time and energy consumption increase
Solution Approach 1:
By dynamically adjusting the bitlength parameters of floating-point representations during training, the system reduces the total volume of activation tensors that need to be stored in off-chip memory. This parameter optimization allows more data to fit in faster on-chip memory, reducing reliance on slow DRAM accesses and improving execution time
Solution Approach 2:
The dynamic adaptation of floating-point precision allows the system to optimize memory allocation in real-time during training. As the model learns the minimum required precision for each tensor, it can allocate memory more efficiently, reducing the frequency and volume of slow off-chip memory accesses
3Ease of manufacture
If static data type selection is used, then implementation simplicity is improved, but convergence optimization worsens due to trial-and-error requirements
Solution Approach 1:
The system makes the data type selection process self-service by automatically determining the optimal mantissa and exponent bitlengths through gradient descent optimization. The model itself learns and selects the appropriate precision parameters during training without requiring external manual configuration or trial-and-error experimentation, enabling both simplicity and optimal convergence
4Quantity of substance
If compact data types like BFloat16 or FP16 are used, then memory traffic is reduced, but training accuracy may deteriorate
Solution Approach 1:
Instead of using fixed compact data types like BFloat16 or FP16 throughout training, the system dynamically adjusts the floating-point precision parameters. The mantissa and exponent bitlengths are adapted during training based on the actual precision requirements of different tensors and training stages, allowing the system to use fewer bits when possible while maintaining accuracy when needed
Solution Approach 2:
The patent transforms the static data type selection into a dynamic parameter optimization problem. By treating the number of mantissa bits and exponent bits as adjustable parameters optimized through gradient descent, the system can find the optimal balance between compact representation and training accuracy for each specific tensor and training phase
Data Source
AI summary
Provided is a system and method a computer-implemented method of adapting floating-point containers of training data for training an artificial neural network, the method including: receiving the training data for training the artificial neural network; determining an adapted mantissa bitlength for the training data comprising determining a required number of bits in the mantissas and trimming least significant bits from the mantissas to arrive at the determined number of bits, determining an adapted exponent bitlength for the training data comprising determining a required number of bits in the exponents of the training data and trimming the most significant bits from the exponents to arrive at the determined number of bits, or determining both; and storing the training data with the adapted mantissa bitlengths, the adapted exponent bitlengths, or both. In some cases, the adapted exponents are stored in groups after trimming their bitlengths to fit the value content.