Reinforcement Learning Circuit With Delayed Activation Quantization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional reinforcement learning devices face significant power consumption due to numerous neural network operations, and applying quantization leads to a substantial reduction in accuracy due to inevitable data loss.
Innovation Solution
A reinforcement learning device that includes a computation circuit which quantizes input activation vectors only after a predetermined quantization delay time has elapsed, maintaining accuracy during neural network operations by using 32-bit fixed-point data until then, and switching to 16-bit fixed-point data after the delay.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Use of energy by moving object
If quantization operation is applied to reduce computation amount, then power consumption is reduced, but accuracy is greatly reduced due to data loss
Solution Approach 1:
The patent applies preliminary action by performing quantization only after a predetermined delay time has elapsed since the start of learning operations. This timing-based approach allows the system to complete initial learning phases with high-precision data before applying quantization, thereby reducing power consumption in later stages while preserving accuracy during critical learning periods.
Solution Approach 2:
The patent implements dynamics by making the quantization operation conditional and time-dependent rather than static. The computation circuit dynamically adjusts whether to quantize input activation vectors based on the elapsed time since learning start, transitioning from high-precision mode to quantized mode as training progresses, thus balancing accuracy and power efficiency throughout the learning process.
2Productivity
If quantization operation is applied early in learning, then computation load is reduced, but data loss occurs and accuracy deteriorates
Solution Approach 1:
The patent uses preliminary action by establishing a delay time mechanism that prevents early quantization. The computation circuit is configured to skip quantization during the initial period, ensuring that high-precision data flows through the network during critical early learning phases, thereby avoiding data loss and accuracy deterioration.
Solution Approach 2:
The patent introduces an intermediary mechanism in the form of a time-based control logic that mediates between the need for fast quantized computation and the requirement for precise unquantized data. This control mechanism acts as a gatekeeper, allowing full-precision computation when needed and enabling quantization when appropriate, thus preventing premature data loss.
Data Source
AI summary
A reinforcement learning device includes a computation circuit configured to perform an operation between a weight matrix and an input activation vector and to apply an activation function on an output of the operation to generate an output activation vector. The computation circuit quantizes the input activation vector when a quantization delay time has elapsed since beginning of a learning operation and does not quantize the input activation vector otherwise.