Neural Processing System Memory Access Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural processing systems face complexity and high power consumption due to the large number of hardware logic components required for convolutional operations and frequent memory accesses in neural processing units (NPUs), especially when handling various convolutional operation modes like feature map striding, filter dilation, and resampling.
Innovation Solution
A software-hardware mechanism that analyzes data to optimize memory access by determining dimensional and positional characteristic parameters, allowing the control processor to program hardware for efficient convolution operations, reducing the complexity of hardware logic components and minimizing memory accesses.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If hardware logic components are increased to handle various convolutional operation modes, then processing capability is improved, but device complexity and power consumption increase
Solution Approach 1:
The patent introduces a control processor as an intermediary between the hardware logic components and the convolutional operation modes. The control processor dynamically configures the hardware components based on the specific operation mode required, eliminating the need for dedicated hardware for each mode. This mediator approach allows the system to handle various convolutional operations (striding, dilation, resampling) using a unified hardware architecture controlled by software, thereby reducing device complexity while maintaining processing capability.
Solution Approach 2:
The patent implements dynamic configurability in the hardware logic components through control signals from the control processor. The hardware components can dynamically adjust their behavior based on the operation mode being executed, rather than being statically designed for specific modes. This dynamic approach allows the same hardware to adapt to different convolutional requirements, reducing the need for multiple dedicated hardware blocks and thereby reducing overall device complexity.
2Productivity
If hardware logic components are increased to handle various convolutional operation modes, then processing capability is improved, but power consumption increases
Solution Approach 1:
The patent designs the hardware logic components to be universal and multi-functional, capable of executing different convolutional operation modes through dynamic configuration rather than having separate dedicated hardware for each mode. The same hardware block can perform striding, dilation, resampling, and other operations by receiving different control parameters from the control processor. This multi-functionality reduces the total amount of hardware needed, thereby reducing power consumption while maintaining the ability to handle various operation modes.
Solution Approach 2:
The control processor acts as a mediator that manages power consumption by dynamically activating only the necessary hardware components for the current operation mode. Instead of keeping all hardware logic components active simultaneously, the control processor configures and activates only the required components based on the specific convolutional operation being performed. This selective activation reduces overall power consumption while maintaining full processing capability across different modes.
3Measurement precision
If memory accesses are increased for convolutional operations, then processing accuracy is improved, but power consumption and processing efficiency deteriorate
Solution Approach 1:
The patent implements preliminary action by pre-fetching and caching convolutional filter data and input feature map data into on-chip memory buffers before the actual convolution operation. The control processor manages the data loading and caching process, ensuring that required data is available in fast memory during processing. This preliminary data preparation reduces the need for frequent accesses to external memory during the computation phase, thereby reducing power consumption while maintaining processing accuracy through complete data availability.
Solution Approach 2:
The patent merges multiple data access operations into unified memory access patterns. By organizing the convolutional computation to process blocks of data together and by combining filter data and input data loading into coordinated memory access sequences, the system reduces the total number of memory accesses. The control processor orchestrates these merged access patterns to ensure all necessary data is fetched efficiently, maintaining processing accuracy while reducing the frequency of memory accesses and associated power consumption.
4Measurement precision
If memory accesses are increased for convolutional operations, then processing accuracy is improved, but processing efficiency deteriorates
Solution Approach 1:
The control processor performs preliminary data preparation by pre-loading convolutional filter data and input feature map data into on-chip memory buffers before the main computation begins. This preliminary action ensures that all necessary data is available in fast memory during the convolution operation, eliminating the need for repeated external memory accesses during computation. The result is improved processing efficiency while maintaining full processing accuracy through complete data availability.
Solution Approach 2:
The patent segments the convolutional computation into smaller processing blocks that can be executed with data already available in on-chip memory. The control processor divides the overall convolution operation into manageable segments, processing them sequentially with data that has been pre-loaded. This segmentation reduces the frequency of external memory accesses by ensuring each segment has its required data available locally, thereby improving processing efficiency while maintaining accuracy through complete data processing.
Data Source
AI summary
A computer-implemented method, performed in a neural processing system comprising control processor circuitry and arithmetic logic circuitry, of performing a convolution between an input feature map (IFM) and convolutional filter data, resulting in an output feature map (OFM). The method includes, obtaining in the control processor circuitry, dimensional characteristic parameters relating to dimensions of input work batch data arrays and positional characteristic parameters relating to positions of feature map content within the input work batches. The method also includes, in the arithmetic logic circuitry, performing convolutions between the input work batches, generated from the IFM based on the dimensional characteristic parameters and the positional characteristic parameters, and work batch filter data arrays corresponding to the filter to produce a plurality of output work batch data arrays. The plurality of output work batches are combined to generate an OFM.