Neural Network Partition Sequencing for Local Memory Reuse
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network (NN) and deep neural network (DNN) computing environments face inefficiencies due to limited local memory, leading to increased latency and power consumption from frequent data transfers between local and main memory, as intermediate layer activations are often stored in main memory, causing unnecessary processing cycles and bandwidth usage.
Innovation Solution
Implementing a 'depth first' and/or 'dynamic depth first' data processing approach that calculates and executes a data apportionment sequence to optimize memory management, allowing for efficient data processing and minimizing main memory reads/writes by utilizing local memory for intermediate activations and storing complete output data in main memory, thereby maximizing local data usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If intermediate layer activations are stored to main memory, then memory capacity is sufficient to hold all data, but data transfer latency and power consumption increase
Solution Approach 1:
The patent segments the memory system into local memory (close to processing units) and main memory (remote storage), and segments the data flow into intermediate activations and final outputs. By keeping intermediate activations in local memory and only transferring final outputs to main memory, the system avoids unnecessary data transfers while maintaining sufficient storage capacity through hierarchical memory architecture.
Solution Approach 2:
The patent introduces local memory as an intermediary buffer between processing units and main memory. This intermediary allows intermediate activations to be stored locally without immediately writing to main memory, thereby reducing data transfer latency and power consumption while still providing access to necessary data for subsequent processing layers.
2Reliability
If intermediate layer activations are stored to main memory, then data is preserved for later use, but bandwidth usage and processing cycles increase
Solution Approach 1:
The patent applies local quality by differentiating the storage location based on data requirements: intermediate activations are stored in local memory where they can be quickly accessed by subsequent processing units, while only final output data is transferred to main memory. This localized storage strategy preserves data reliability for immediate use while avoiding unnecessary bandwidth consumption and processing cycles associated with main memory transfers.
3Device complexity
If local memory size is small, then memory hierarchy is simple, but all intermediate activations must be stored to main memory causing double bandwidth usage
Solution Approach 1:
The patent implements a dynamic memory management strategy where the system adaptively determines which activations to keep in local memory based on the relationship between consecutive layers. When the next layer consumes the current layer's outputs, the system keeps data in local memory to avoid double transfers. This dynamic approach optimizes energy consumption without requiring a complex static memory hierarchy, balancing simplicity with efficiency.
Data Source
AI summary
Optimized memory usage and management is crucial to the overall performance of a neural network (NN) or deep neural network (DNN) computing environment. Using various characteristics of the input data dimension, an apportionment sequence is calculated for the input data to be processed by the NN or DNN that optimizes the efficient use of the local and external memory components. The apportionment sequence can describe how to parcel the input data (and its associated processing parameters—e.g., processing weights) into one or more portions as well as how such portions of input data (and its associated processing parameters) are passed between the local memory, external memory, and processing unit components of the NN or DNN. Additionally, the apportionment sequence can include instructions to store generated output data in the local and/or external memory components so as to optimize the efficient use of the local and/or external memory components.