Mixed-Precision Neural Network Quantization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks executing with floating-point data types are computationally expensive and power-intensive, making them inefficient for devices with limited resources, such as smartphones and tablets, due to the slow speed and high power consumption of floating-point arithmetic.
Innovation Solution
Implementing a mixed-precision approach by converting some floating-point values to smaller integer data types, such as 8-bit or 12-bit integers, while maintaining accuracy by selectively representing weights and BLOBs with differing levels of precision, allowing for efficient execution on hardware that supports multiple integer data types.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If floating-point data types are used to represent neural network weights and BLOBs, then measurement precision is maintained, but productivity decreases and use of energy increases
Solution Approach 1:
The patent changes the precision parameter of data types from uniform floating-point to mixed-precision integer formats (e.g., 8-bit, 16-bit, 32-bit). By adjusting the bit-width parameter for different neural network components, the system achieves faster execution and lower power consumption while maintaining sufficient precision through selective application of different precision levels to different layers or operations.
Solution Approach 2:
The patent applies different precision levels to different parts of the neural network based on their specific requirements. Critical layers that require high accuracy use higher precision (e.g., 32-bit), while less sensitive layers use lower precision (e.g., 8-bit). This local differentiation optimizes overall performance by matching precision to functional needs rather than applying uniform precision throughout.
2Measurement precision
If floating-point data types are used to represent neural network weights and BLOBs, then measurement precision is maintained, but use of energy increases
Solution Approach 1:
The patent changes the precision parameter of data types from uniform floating-point to mixed-precision integer formats. By reducing the bit-width parameter for data representation in less critical operations, the system significantly reduces power consumption while maintaining adequate precision through selective application of different precision levels to different neural network components.
Solution Approach 2:
The patent applies different precision levels to different parts of the neural network based on their specific requirements. Critical layers that require high accuracy use higher precision, while less sensitive layers use lower precision. This local differentiation optimizes power consumption by matching precision to functional needs rather than applying uniform high precision throughout.
3Measurement precision
If floating-point data types are used to represent neural network weights and BLOBs, then measurement precision is maintained, but device complexity increases
Solution Approach 1:
The patent changes the precision parameter of data types from uniform floating-point to mixed-precision integer formats. By adjusting the bit-width parameter for different neural network components, the system reduces computational complexity and resource requirements while maintaining sufficient precision through selective application of different precision levels to different layers or operations.
Solution Approach 2:
The patent applies different precision levels to different parts of the neural network based on their specific requirements. Critical layers that require high accuracy use higher precision, while less sensitive layers use lower precision. This local differentiation reduces overall device complexity by matching precision to functional needs rather than applying uniform high precision throughout.
Data Source
AI summary
Techniques for mixed precision quantization of a machine learning (ML) model. The techniques include receiving a target performance relating to the ML model including objects of a first data type represented by a first number of bits, wherein the target performance relates to changing a first portion of the objects to a second data type represented by a second number of bits and changing a second portion of the objects to a third data type represented by a third number of bits. The techniques further include selecting the first portion and the second portion, based on maintaining a performance relating to the ML model at or below the target performance, and changing the first portion of objects from the first data type to the second data type and the second portion of objects from the first data type to the third data type.