Intention Recognition Attention Quantization for Faster Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing intention recognition models based on neural networks require significant computational resources and suffer from long operation times due to the use of 32-bit floating-point numbers, even after model quantization, leading to inefficient recognition processes.
Innovation Solution
The method involves quantizing the dot product of query and key vectors to a fixed-point target vector of a first bit and determining floating-point attention weights of a second bit through a target mapping relationship, reducing the complexity of calculating attention weights by adjusting the process to finding correspondence relationships.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If model quantization is applied to reduce computation, then model volume is reduced, but recognition efficiency still needs improvement
Solution Approach 1:
The patent changes the precision parameter of attention weight calculation from standard floating-point to a hybrid representation. Specifically, it uses 8-bit fixed-point numbers for the exponent part and 8-bit fixed-point numbers for the mantissa part, totaling 16 bits. This parameter change reduces computational complexity while maintaining sufficient precision for intention recognition tasks, thereby improving recognition efficiency and reducing operation time.
2Measurement precision
If floating-point attention weight calculation is performed, then accuracy is maintained, but computation complexity is high
Solution Approach 1:
The patent segments the attention weight calculation into two separate fixed-point components: an exponent part (8 bits) and a mantissa part (8 bits). This segmentation allows each component to be calculated and stored independently using simpler fixed-point arithmetic operations, reducing the overall calculation complexity compared to standard floating-point operations while preserving the precision needed for accurate attention weight representation.
Data Source
AI summary
The present application relates to an intention recognition method and apparatus, a readable medium, and an electronic device. The method includes: by means of a preset intention recognition quantification model, performing a quantification operation on a dot product of a query vector and a key vector which correspond to each character in a target text, so as to obtain a fixed-point type target vector of a first bit; according to the fixed-point type target vector, determining, by means of a target mapping relationship, a floating-point type attention weight of a second bit corresponding to each character; and according to the floating-point type attention weight, determining a target intention corresponding to the target text, the first bit being smaller than the second bit.