Floating-Point MAC Skipping for Lower-Energy AI Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing floating-point arithmetic operations in in-memory computing and ASICs are inefficient, leading to increased data movement and energy consumption, particularly in neural networks, without significantly impacting inference accuracy.
Innovation Solution
Implementing MAC operations with MAC-skipping by excluding product mantissas with delta exponents greater than a predetermined threshold value, determined through algorithm-hardware co-optimization, to reduce computational steps and mantissa multiplications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If traditional floating-point MAC operations are performed in in-memory computing, then computational accuracy is maintained, but energy consumption and data movement increase significantly
Solution Approach 1:
The patent extracts and processes only the most significant mantissa bits (e.g., top 10-15 bits) of floating-point numbers, discarding less significant bits that contribute minimally to inference accuracy. This selective extraction reduces computational workload and energy consumption while maintaining sufficient accuracy for neural network inference tasks
Solution Approach 2:
The patent applies partial action by performing MAC operations only on a subset of mantissa bits rather than the full precision representation. By processing only the essential bits needed for accurate inference, the system reduces energy consumption and data movement requirements while maintaining acceptable computational accuracy
2Reliability
If full precision floating-point MAC operations are executed, then computational accuracy is maintained, but computational efficiency decreases
Solution Approach 1:
The patent extracts only the essential mantissa bits required for accurate inference, eliminating redundant computational steps involving less significant bits. This extraction approach maintains inference accuracy while significantly improving computational efficiency by reducing the number of operations required
Solution Approach 2:
The patent changes the precision parameter of floating-point operations from full precision to reduced precision (e.g., 10-15 significant bits). This parameter modification enables faster computational operations while maintaining sufficient accuracy for neural network inference, thereby improving productivity
3Reliability
If all mantissa bits are processed in MAC operations, then computational completeness is maintained, but the number of computational steps increases
Solution Approach 1:
The patent extracts and processes only the most significant mantissa bits, removing unnecessary computational steps related to less significant bits. This extraction maintains computational completeness for inference purposes while reducing the overall number of computational steps and simplifying the processing architecture
Solution Approach 2:
The patent applies partial action by performing MAC operations only on the essential portion of mantissa bits rather than all bits. This partial processing maintains sufficient computational completeness for accurate inference while significantly reducing the number of computational steps required
Data Source
AI summary
In some embodiments, a computing method includes, for pairs of a first and second floating-point numbers, each having a respective mantissa and exponent, supplying to a respective one of multiply circuits the mantissas of a subset of the pairs of first and second floating-point number, the subset of the plurality of pairs of first and second floating-point numbers each having a respective sum of the exponents of the first and second floating-point numbers, respectively, meeting a predetermined criterion, such as the sum being smaller than a predetermined threshold value; generating, using each of the plurality of multiply circuits, a product of the mantissas of the respective pair of first and second floating-point numbers; accumulating the product mantissas to generate a product mantissa partial sum; combining the product mantissa partial sum and maximum product exponent to generate an output floating point number; and for each of the remaining pairs of first and second floating-point numbers: withholding the mantissas from respective multiply circuits, disabling the respective multiply circuits, or both. A trained AI model can be used to determine the threshold value. Various components for the multiplication and accumulation steps can be disabled for the pairs of numbers not meeting the criterion by a control signal.