Neural Network Data Partition Sequencing for Local Memory Bottlenecks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network (NN) and deep neural network (DNN) computing environments face inefficiencies due to limited local memory resources, leading to increased latency and power consumption from frequent data transfers between local and main memory.
Innovation Solution
Implementing a 'depth first' and/or 'dynamic depth first' data processing approach that calculates and executes a data apportionment sequence to optimize memory management and minimize data read/writes to main memory by parceling input data across processing layers and utilizing local memory for intermediate activations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If intermediate layer activations are stored to main memory, then data availability for subsequent layers is ensured, but data transfer latency and power consumption increase
Solution Approach 1:
The patent segments the neural network computation into micro-batches, processing a subset of data through multiple layers before transferring results to main memory. This allows intermediate activations to remain in local memory longer, reducing transfer frequency while ensuring data availability through controlled segmentation of the computation stream.
Solution Approach 2:
The system performs preliminary actions by pre-loading data into local memory and preparing computation pipelines in advance. By organizing data apportionment sequences beforehand and pre-positioning data in local memory buffers, the system minimizes the need for frequent transfers to main memory during actual computation, thereby reducing latency.
2Quantity of substance
If all intermediate activations are stored to main memory, then memory capacity constraints are relieved, but processing speed decreases due to frequent data transfers
Solution Approach 1:
The patent divides the computation into micro-batches that can be processed sequentially in local memory. By segmenting the data flow and processing smaller subsets through multiple layers before main memory access, the system maintains high processing speed while managing limited local memory capacity effectively.
Solution Approach 2:
The system ensures continuity of useful action by maintaining data in local memory throughout the multi-layer processing pipeline. Through data apportionment sequences and micro-batch processing, computations continue uninterrupted in local memory, avoiding the speed penalty of frequent main memory transfers while still respecting capacity constraints.
3Ease of operation
If data is transferred frequently between local and main memory, then data access requirements are met, but power consumption increases
Solution Approach 1:
The system performs preliminary data apportionment and positions required data in local memory buffers before computation begins. By pre-organizing data access patterns and loading necessary data into local memory in advance, the system meets data access requirements while minimizing the frequency and energy cost of transfers to main memory.
Solution Approach 2:
The patent implements self-service mechanisms where the system autonomously manages data apportionment sequences and micro-batch processing to keep required data in local memory. Through intelligent data retention and local processing, the system serves its own data access needs without frequent main memory intervention, thereby reducing power consumption.
4Device complexity
If local memory size is small, then device complexity is reduced, but memory utilization efficiency decreases
Solution Approach 1:
The patent introduces dynamic data apportionment sequences that adaptively manage data in local memory based on computation needs. The system dynamically adjusts which data partitions are retained in local memory and for how long, optimizing memory utilization efficiency without requiring larger or more complex memory architecture.
Solution Approach 2:
The system changes operational parameters by implementing micro-batch processing and adjustable data apportionment strategies. These parameter changes allow the same small local memory to be used more efficiently by controlling data flow patterns, retention durations, and processing batch sizes, thereby improving memory utilization without increasing device complexity.
Data Source
AI summary
Optimized memory usage and management is crucial to the overall performance of a neural network (NN) or deep neural network (DNN) computing environment. Using various characteristics of the input data dimension, an apportionment sequence is calculated for the input data to be processed by the NN or DNN that optimizes the efficient use of the local and external memory components. The apportionment sequence can describe how to parcel the input data (and its associated processing parameters—e.g., processing weights) into one or more portions as well as how such portions of input data (and its associated processing parameters) are passed between the local memory, external memory, and processing unit components of the NN or DNN. Additionally, the apportionment sequence can include instructions to store generated output data in the local and/or external memory components so as to optimize the efficient use of the local and/or external memory components.