Dynamic Operator Loading for Neural Network Inference Power Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network inference requires significant power consumption due to continuous DRAM usage during loading and execution, leading to high energy costs and latency issues, especially when large models are stored in dynamic random access memory (DRAM) and executed across multiple hardware accelerators.
Innovation Solution
An apparatus and method that dynamically load operators from DRAM to a tightly-coupled memory (TCM) or SRAM, managing power by turning on/off DRAM and switching hardware accelerators between standby and working modes based on an executing scheme, reducing latency and energy consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If DRAM is continuously powered on during neural network inference, then data access speed is maintained, but power consumption increases significantly
Solution Approach 1:
The patent applies preliminary action by loading operator data from DRAM to TCM/SRAM memory before the neural network inference execution begins. This pre-loading ensures that frequently accessed operator data is already in faster memory, maintaining data access speed during execution while allowing DRAM to be powered down, thus resolving the contradiction between speed and power consumption.
Solution Approach 2:
The patent implements dynamic power management by adjusting DRAM power states based on execution phase. During active inference, relevant data is kept in TCM/SRAM for fast access, while DRAM is powered down or placed in low-power mode when not actively transferring data. This dynamic approach maintains performance when needed while minimizing power consumption during idle periods.
2Use of energy by moving object
If operators are loaded from DRAM to TCM/SRAM during execution, then power consumption is reduced, but additional latency is introduced
Solution Approach 1:
The patent eliminates loading latency by performing the operator data transfer from DRAM to TCM/SRAM before inference execution begins. The executing scheme identifies which operators will be needed and pre-loads them into faster memory, so that during actual execution, all data is already in place and no additional loading latency occurs.
Solution Approach 2:
The patent segments the neural network operators into different groups based on their access patterns and execution timing. The executing scheme divides operators into those that will be executed sequentially and those that can be pre-loaded, allowing optimized memory management where critical path operators are pre-loaded while others are handled during execution, minimizing overall latency.
3Productivity
If multiple hardware accelerators are used for parallel execution, then processing throughput increases, but power consumption and coordination complexity increase
Solution Approach 1:
The patent applies partial action by activating only the number of hardware accelerators needed for the current execution phase rather than keeping all accelerators continuously active. The executing scheme dynamically determines which accelerators should be active based on the current operator being executed, enabling parallel processing when beneficial while reducing power consumption by idle accelerators.
Solution Approach 2:
The patent implements feedback mechanisms where the executing scheme monitors execution progress and dynamically adjusts accelerator activation. Based on feedback from execution status and data availability, the system activates or deactivates hardware accelerators in real-time, optimizing the balance between throughput and power consumption.
4Speed
If operator data is pre-loaded to TCM/SRAM, then execution speed improves, but memory bandwidth is consumed during loading
Solution Approach 1:
The patent applies local quality by selectively pre-loading only the specific operator data that will be needed for upcoming execution, rather than loading all data uniformly. The executing scheme analyzes the execution plan and identifies which operators require pre-loading, concentrating memory bandwidth resources on critical data transfers while leaving non-critical data for later loading during execution.
Data Source
AI summary
Aspects of the disclosure provide an apparatus for executing a program that involves a plurality of operators. For example, the apparatus can include an executor and an analyzer. The executor can be configured to execute the program with at least a first one of the operators loaded on a second memory from a first memory that stores the operators and to generate a signal based on a progress of the execution of the program with the first operator. The analyzer can be coupled to the executor, the first memory and the second memory, and configured to load at least a second one of the operators of the program next to the first operator stored in the first memory to the second memory before the executor finishes execution of the program with the first operator based on the signal from the executor and an executing scheme stored in the second memory.