Dynamic Operator Loading for Neural Network Inference Power Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network inference requires large memory and power consumption, leading to latency and energy inefficiency due to continuous DRAM power usage and hardware accelerators switching between standby and working modes.
Innovation Solution
An apparatus and method that dynamically load neural network operators from DRAM to TCM/SRAM and manage power by using an analyzer and executing scheme to control DRAM power and hardware accelerator modes, optimizing power usage and reducing latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If DRAM is continuously powered on to store neural network operators, then memory availability is improved, but power consumption increases
Solution Approach 1:
The system performs preliminary actions by loading operators from DRAM to TCM/SRAM before they are actually needed for execution. The analyzer predicts future operator requirements and pre-loads them, allowing the DRAM to be powered down earlier while ensuring operators are ready in faster memory when needed.
Solution Approach 2:
The system dynamically adjusts DRAM power states based on real-time execution progress and predicted future needs. The analyzer continuously monitors execution status and dynamically decides when to power DRAM on/off, transitioning between power states rather than maintaining a static state throughout operation.
2Speed
If hardware accelerators are kept in working mode to execute operators, then execution speed is improved, but power consumption increases
Solution Approach 1:
Hardware accelerators are placed in working mode in advance before operators are ready for execution. The analyzer predicts when operators will be available and pre-activates the necessary hardware accelerators, eliminating wait time while allowing them to return to standby mode immediately after use to save power.
Solution Approach 2:
Hardware accelerators alternate between working mode and standby mode in periodic cycles synchronized with operator availability. They are activated just before needed, perform execution, then return to low-power standby, creating an efficient on-off pattern that balances speed and power consumption.
3Speed
If operators are loaded from DRAM to TCM/SRAM during execution, then access speed is improved, but latency increases due to loading time
Solution Approach 1:
Operators are loaded from DRAM to TCM/SRAM in advance of their actual execution needs. The analyzer predicts which operators will be required and initiates loading operations beforehand, so that when execution is needed, operators are already in fast memory ready for immediate access without causing latency.
4Use of energy by moving object
If DRAM is powered off to save energy, then power consumption is reduced, but memory access becomes unavailable
Solution Approach 1:
The memory system is segmented into multiple levels: DRAM for bulk storage, TCM/SRAM for active operators, and execution units for processing. This segmentation allows selective powering of only the necessary memory segments, keeping DRAM off when not needed while maintaining availability of operators in faster memory layers.
Solution Approach 2:
TCM/SRAM acts as an intermediary buffer between DRAM and the execution units. Operators are pre-loaded into this intermediate memory layer, allowing DRAM to be powered down while keeping essential operators readily available in the intermediary TCM/SRAM for immediate access when needed.
Data Source
Figure 1A~2
Figure 3
Figure 4A~5
AI summary
Aspects of the disclosure provide an apparatus for executing a program that involves a plurality of operators. For example, the apparatus can include an executor and an analyzer. The executor can be configured to execute the program with at least a first one of the operators loaded on a second memory from a first memory that stores the operators and to generate a signal based on a progress of the execution of the program with the first operator. The analyzer can be coupled to the executor, the first memory and the second memory, and configured to load at least a second one of the operators of the program next to the first operator stored in the first memory to the second memory before the executor finishes execution of the program with the first operator based on the signal from the executor and an executing scheme stored in the second memory.