Very Low Precision Floating Point Representation for Deep Learning
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current floating point unit (FPU) technologies are resource-intensive and computationally expensive, leading to long compute times and complex hardware architectures, especially in neural network training, where high precision is desired but not efficiently achieved with existing 16-bit, 32-bit, and 64-bit formats, necessitating a more efficient representation for cognitive computing.
Innovation Solution
Implementing a very low precision floating point representation using less than 16 bits, specifically configuring an FPU with an 8-bit format dividing bits into a sign bit, exponent bits, and mantissa bits to represent numeric values as a multiple of a fraction, allowing for efficient computation with minimal loss of accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional floating point formats (16-bit, 32-bit, 64-bit) are used in FPU, then measurement precision is improved, but device complexity and energy consumption increase
Solution Approach 1:
The patent changes the precision parameter from traditional 16/32/64-bit formats to a customized very low precision format (e.g., 8-bit or lower), adjusting the balance between precision and hardware complexity by redefining the number of bits allocated to significand, exponent, and sign components
Solution Approach 2:
The patent employs lower-precision floating point representations that require simpler, less resource-intensive hardware circuits, effectively using 'cheaper' computational resources that consume less power and occupy less area while still achieving acceptable accuracy for deep learning workloads
2Measurement precision
If high precision floating point computations are performed, then measurement precision is improved, but use of energy increases
Solution Approach 1:
The patent reduces the precision parameter to very low precision format (less than 16 bits), which directly decreases the computational complexity and energy consumption of floating point operations while maintaining sufficient accuracy for neural network training
Solution Approach 2:
The patent uses simplified floating point representations that require less energy-intensive hardware circuits, reducing power consumption by employing 'cheaper' computational resources that are sufficient for the task at hand
3Measurement precision
If traditional floating point formats are used, then measurement precision is improved, but productivity decreases due to longer compute times
Solution Approach 1:
The patent changes the precision parameter to very low precision format, which accelerates computational throughput by reducing the complexity of each floating point operation, thereby improving productivity in neural network training despite reduced precision
Solution Approach 2:
The patent applies partial precision (very low precision rather than full precision) which is sufficient for deep learning workloads, achieving acceptable accuracy while significantly improving computational speed and productivity
Data Source
AI summary
A specialized circuit is configured for floating point computations using numbers represented by a very low precision format (VLP format). The VLP format includes less than sixteen bits and is apportion into a sign bit, exponent bits (e), and mantissa bits (p). The configured specialized circuit is operated to store an approximation of a numeric value in the VLP format, where the approximation is represented as a function of a multiple of a fraction, where the fraction is an inverse of a number of discrete values that can be represented using only the mantissa bits.