Neural Network Quantization Using Euler Angles for Low-Precision AI
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural networks face challenges in efficiently reducing precision from high-precision formats to low-precision formats without significant loss of accuracy, particularly in resource-constrained devices like smartphones and embedded systems, due to high computation and power consumption.
Innovation Solution
The method involves converting parameter elements of a multidimensional vector neural network into Euler angles, performing quantization on these angles, and determining quantized parameter elements based on the conversion characteristics between multidimensional vectors and Euler angles, specifically using quaternion vectors and spherical surface coordinates to maintain accuracy while reducing precision.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If floating-point representation is lowered from FP32 to increase operation speed and reduce power consumption, then operation speed and power efficiency are improved, but quantization errors increase and accuracy is lost
Solution Approach 1:
The patent changes the parameter representation from Cartesian coordinates to spherical coordinates (Euler angles). By converting weight vectors from (w1, w2, w3, w4) to spherical coordinates (r, θ, φ, ψ), the quantization process operates in a different parameter space where angular quantization better preserves the directional information critical for neural network accuracy while enabling lower precision representation.
Solution Approach 2:
The patent introduces a new dimensional framework by transforming 4D Cartesian weight vectors into spherical coordinate representation. This dimensional change allows the system to separate the magnitude (r) and directional (θ, φ, ψ) components, enabling independent optimization of each component's precision requirements and reducing overall quantization error.
2Use of energy by moving object
If precision is reduced to reduce computation and power consumption in resource-constrained devices, then power consumption and computation cost are reduced, but training and inference performance deteriorates
Solution Approach 1:
By changing to spherical coordinate representation, the patent enables more efficient low-precision computation. The angular parameters (θ, φ, ψ) can be quantized with fewer bits while maintaining the essential directional information needed for accurate neural network operations, thus reducing power consumption without significantly degrading performance.
Solution Approach 2:
The spherical coordinate system acts as an intermediary representation that bridges high-precision FP32 and low-precision quantized formats. This intermediate representation preserves critical information during the transition, allowing accurate training and inference even when using reduced precision arithmetic in resource-constrained devices.
3Manufacturing precision
If quantization is performed directly on parameter elements to reduce precision, then precision is reduced for efficient operation, but quantization errors significantly increase
Solution Approach 1:
The patent applies parameter transformation by converting weight vectors from Cartesian to spherical coordinates before quantization. In the spherical coordinate system, the angular parameters have a more uniform distribution and better condition number, which reduces quantization errors when mapping to discrete low-precision levels.
Solution Approach 2:
By using spherical coordinates (r, θ, φ, ψ) to represent weight vectors, the patent exploits the geometric properties of spherical space. The angular parameters naturally capture the directional relationships in the weight space, and quantizing these angular parameters preserves the geometric structure better than linear quantization of Cartesian coordinates, thereby reducing information loss.
Data Source
AI summary
A processor-implemented method with neural network quantization includes: converting each of parameter elements comprised in a multidimensional parameter vector of a multidimensional vector neural network into a Euler angle; performing quantization on the Euler angles; and determining each of quantized parameter elements based on the quantized Euler angles.