AI Accelerator Quantization for Lower Memory and Power Use
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing artificial intelligence systems face challenges in optimizing computations due to large memory requirements and increased power consumption caused by high precision data handling, leading to decreased performance and increased costs.
Innovation Solution
An accelerator system that includes a quantizer to convert high precision computation results to low precision data, reducing the memory bandwidth and capacity needs, thereby improving performance and reducing power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If high precision data is used for AI computations, then computation accuracy is improved, but memory requirements and power consumption increase
Solution Approach 1:
The patent applies parameter changes by dynamically adjusting the precision of data representation based on computational requirements. The system uses mixed precision computation, where different data types (e.g., FP32, FP16, INT8) are used for different parts of the computation pipeline. This allows the system to maintain high accuracy where needed while reducing power consumption and memory bandwidth usage in less critical operations, directly resolving the contradiction between computation accuracy and power consumption.
2Measurement precision
If high precision data is used for AI computations, then computation accuracy is improved, but memory bandwidth and capacity requirements increase
Solution Approach 1:
The system changes the precision parameter of data stored in memory versus data used in computation. Low precision data types (e.g., INT8, FP16) are used for storing weights and activations in memory, reducing memory capacity and bandwidth requirements. High precision data types (e.g., FP32) are used only when necessary for intermediate computations to maintain accuracy, thereby resolving the contradiction between computation accuracy and memory requirements.
3Use of energy by moving object
If low precision data is used, then memory requirements and power consumption are reduced, but computation accuracy decreases
Solution Approach 1:
The patent segments the computation pipeline into different stages with different precision requirements. The system divides data processing into weight loading, activation processing, and result computation stages, applying appropriate precision levels to each segment. This segmentation allows the system to use low precision for data storage and transmission (reducing power consumption) while maintaining high precision for critical computation operations (preserving accuracy).
Solution Approach 2:
The system dynamically changes precision parameters based on the operational stage and data type. Low precision (INT8, FP16) is used for data in memory and during transmission, reducing power consumption. High precision (FP32) is activated only when needed for intermediate results and final computations, ensuring accuracy is maintained despite using low precision elsewhere in the system.
4Quantity of substance
If low precision data is used, then memory requirements and power consumption are reduced, but computation accuracy decreases
Solution Approach 1:
The patent applies parameter changes by using different precision levels for different data types and computational stages. Low precision data types (INT8, FP16) are used for storing weights and activations in memory, significantly reducing memory capacity and bandwidth requirements. High precision data types (FP32) are used for intermediate computation results and final outputs, ensuring computation accuracy is maintained despite the use of low precision storage formats.
Data Source
AI summary
Disclosed is an accelerator performing an artificial intelligence (AI) computation, which includes a processing element that generates first result data by performing a first computation on first activation data and first weight data loaded from a memory, and a quantizer that generates first output data by performing a quantization on the first result data, and the first activation data, the first weight data, and the first output data are of a low precision type, the first result data is of a high precision type, and the first output data is stored in the memory.