Block Floating-Point Conversion for AI Memory Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current methods for training artificial intelligence models, such as deep neural networks, are hindered by the need for extensive computational resources and high memory usage, particularly when using floating-point formats like FP32.
Innovation Solution
The proposed solution involves converting floating-point values into block floating-point values, which share a common exponent and store only the sign and mantissa, reducing memory requirements and enabling efficient processing using hardware accelerators.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If floating-point values (FP32) are used for training AI models, then calculation precision is maintained, but memory usage and power consumption increase significantly
Solution Approach 1:
The patent segments floating-point values into block floating-point values by grouping multiple values with the same exponent together. This segmentation allows the system to store only one exponent for the entire block rather than individual exponents for each value, significantly reducing memory usage while maintaining calculation precision through the block-based organization structure.
Solution Approach 2:
The patent extracts the exponent information from individual floating-point values and stores it separately as a shared exponent for the block. By taking out the exponent from each individual value and consolidating it at the block level, the system eliminates redundant exponent storage and reduces overall memory requirements while preserving the precision needed for calculations.
2Measurement precision
If floating-point values are used for training AI models, then calculation accuracy is maintained, but training time increases due to extensive computational resources required
Solution Approach 1:
The patent merges multiple floating-point values into block floating-point values by combining values with the same exponent into single block representations. This merging reduces the total number of values that need to be processed during training, decreasing computational overhead and training time while maintaining calculation accuracy through the preserved sign and mantissa information.
Solution Approach 2:
The patent changes the parameter representation from individual floating-point values to block-based representations with shared exponents. This parameter change transforms the data structure to reduce memory access operations and computational complexity during training, thereby improving productivity and reducing training time while maintaining the necessary calculation accuracy.
3Quantity of substance
If block floating-point values are used, then memory usage is reduced, but processing complexity increases due to conversion requirements
Solution Approach 1:
The patent applies preliminary action by pre-converting floating-point values to block floating-point values before training operations begin. This pre-processing step organizes the data into blocks with shared exponents in advance, so that during actual training, the system can directly use the optimized block structure without needing to perform complex conversions on-the-fly, thereby reducing processing complexity during execution.
4Use of energy by moving object
If block floating-point format is implemented, then power consumption is reduced, but hardware implementation complexity increases
Solution Approach 1:
The patent uses copying by creating block floating-point representations that replicate the essential information (sign and mantissa) of original floating-point values while using a shared exponent for the block. This copying approach allows the hardware to work with simplified block structures that consume less power, while the copying mechanism manages the complexity of conversion and representation transformation.
Data Source
AI summary
A method of converting a floating-point value into a block floating-point value includes obtaining floating-point values, determining, as a shared exponent, an exponent of at least one first floating-point value having a maximum exponent, from among the floating-point values, storing an index of the at least one first floating-point value in a memory, right-shifting an implicit bit and explicit bits of a mantissa of at least one second floating-point value not having the maximum exponent, from among the floating-point values, by as much as a difference between the shared exponent and an exponent of the at least one second floating-point value, and storing, in the memory, block floating-point values including a sign and a mantissa of the at least one first floating-point value, a sign and the mantissa of the at least one second floating-point value, and the shared exponent.