Fixed-Point Quantized Neural Network Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training of artificial neural networks (ANNs) for control devices and embedded systems faces challenges in efficiently using hardware resources due to the need for floating-point operations, which are costly, energy-intensive, and space-consuming, especially when dealing with demanding classification tasks that are difficult to mathematically model.
Innovation Solution
A method for training trainable modules that discretizes parameters to be stored as fixed-point numbers, using a specified number of bits, allowing for efficient implementation on lower-cost, more energy-efficient fixed-point hardware by adjusting parameters to minimize a cost function that balances output reproduction and discretization accuracy, with the number of bits and discrete values optimized for specific applications.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If floating-point operations are used for training neural networks, then calculation precision is maintained, but hardware cost and energy consumption increase significantly
Solution Approach 1:
The patent changes the numerical representation parameter from floating-point to fixed-point format. By quantizing weights and activations to discrete fixed-point values, the system maintains sufficient calculation precision for neural network operations while enabling implementation on simpler, cheaper fixed-point hardware units that consume less energy and occupy less space.
Solution Approach 2:
The patent substitutes the mechanical/floating-point computation system with a fixed-point computation system. Instead of using complex floating-point arithmetic units, the invention employs simpler fixed-point arithmetic operations, replacing the need for expensive floating-point hardware with more efficient fixed-point hardware that achieves comparable performance for the specific application.
2Measurement precision
If floating-point operations are used for training neural networks, then calculation accuracy is maintained, but energy consumption increases
Solution Approach 1:
The patent changes the numerical representation parameter from floating-point to fixed-point format. By quantizing weights and activations to discrete fixed-point values, the system maintains sufficient calculation precision for neural network operations while enabling implementation on simpler, cheaper fixed-point hardware units that consume less energy and occupy less space.
3Productivity
If more GPU working memory is installed to handle floating-point operations, then training capability is improved, but device cost and space requirements increase
Solution Approach 1:
The patent changes the numerical representation parameter from floating-point to fixed-point format. By quantizing weights and activations to discrete fixed-point values, the system maintains sufficient calculation precision for neural network operations while enabling implementation on simpler, cheaper fixed-point hardware units that consume less energy and occupy less space.
4Device complexity
If parameters are discretized to fixed-point numbers, then hardware cost and energy efficiency are reduced, but training accuracy may deteriorate
Solution Approach 1:
The patent changes the numerical representation parameter from floating-point to fixed-point format. By quantizing weights and activations to discrete fixed-point values, the system maintains sufficient calculation precision for neural network operations while enabling implementation on simpler, cheaper fixed-point hardware units that consume less energy and occupy less space.
Solution Approach 2:
The patent incorporates feedback mechanisms during training to adapt to quantization effects. By monitoring training performance and adjusting quantization parameters or applying correction techniques, the system compensates for potential accuracy losses due to discretization, ensuring that fixed-point implementation achieves comparable training accuracy to floating-point methods.
Data Source
AI summary
A method for training a trainable module that maps input variables onto output variables through an internal processing chain. A learning data set is provided including learning values of the input variables and associated learning values of the output variables. A list of discrete values is provided from which the parameters characterizing the internal processing chain are to be selected, the discrete values being selected such that they can be stored without loss of quality. The learning values are mapped by the trainable module onto assessment values of the output variables. A cost function is evaluated that characterizes deviations of the assessment values of the output variables from the learning values and of at least one parameter of the internal processing chain from at least one discrete value in the list. At least one parameter of the internal processing chain is adjusted to improve the value of the cost function.