Neural Network Memory Segmentation for Power and Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional hardware neurons in artificial neural networks face significant power consumption and latency issues due to the need for frequent memory operations, particularly when loading weight values and bias values from off-chip non-volatile memory.
Innovation Solution
The implementation of a storage device with multiple dedicated storage portions for different data types within the neural network, such as code, data, and weight storage portions, allows for on-chip or near-chip access, reducing the need for off-chip memory access and optimizing storage characteristics for each data type.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If weight values and bias values are stored in off-chip non-volatile memory, then storage capacity is improved, but power consumption and latency increase
Solution Approach 1:
The memory system is segmented into multiple storage portions (first storage portion for weight values, second storage portion for bias values, third storage portion for activation function outputs) with different storage characteristics optimized for their respective data types. This segmentation allows each portion to be tailored for specific access patterns and performance requirements.
Solution Approach 2:
Different storage portions are assigned different storage characteristics (e.g., retention time, access speed, write endurance) according to the specific requirements of the data they store. Weight values require high retention and read efficiency, while activation function outputs may require different characteristics, allowing local optimization rather than uniform storage.
2Productivity
If weight values and bias values are loaded from off-chip memory into on-chip RAM registers, then hardware neuron operations can proceed, but latency increases
Solution Approach 1:
Weight values and bias values are pre-loaded into on-chip storage portions during idle periods or initialization phases, so that when hardware neurons need to perform operations, the required data is already available in fast accessible memory, eliminating the need for real-time off-chip memory access.
Solution Approach 2:
On-chip storage portions serve as an intermediary between off-chip non-volatile memory and hardware neurons. This intermediary layer buffers the data transfer, allowing hardware neurons to access data at high speed from on-chip memory while the bulk storage remains in slower off-chip memory, thus reducing perceived latency.
3Ease of operation
If frequent memory operations are performed for weight values and bias values, then hardware neuron functionality is maintained, but power consumption increases
Solution Approach 1:
The memory access strategy is made dynamic by selectively accessing different storage portions based on operational requirements. During inference, weight and bias values are accessed from optimized storage portions with high read endurance and low power consumption. During training, the system can switch to different access patterns that leverage write-capable storage portions, optimizing power usage for each operational mode.
Data Source
AI summary
An artificial neural network device, including input circuitry configured to provide input data into a neuron, weight operation circuitry configured to input a weight value into the neuron, bias operation circuitry configured to input a bias value into the neuron, activation function circuitry configured to receive an output of the bias operation circuitry and output an activation function output; and a storage device including storage circuitry electrically connected to the weight operation circuitry, the bias operation circuitry, and the activation function circuitry, wherein the input circuitry, the weight operation circuitry, the bias operation circuitry, and the activation function circuitry are operated based on code data, and wherein the storage device includes a plurality of storage portions, each storage portion of the plurality of storage portions configured to store one or more of the code data, the input data, the weight value, the bias value, or the activation function output.