NPU Modulation Kernels for Fewer CNN Memory Reads
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The processing speed and power consumption of neural processing units (NPUs) are hindered by the frequent and energy-intensive memory reads required to access kernels in convolutional neural networks (CNNs), and kernels in these models often exhibit high similarity, allowing for reduced inference accuracy with minor adjustments.
Innovation Solution
A neural processing unit (NPU) generates modulation kernels using a simple algorithm, reducing memory read operations by utilizing a first kernel and a kernel filter, and employing mathematical functions to create subsequent modulation kernels, thereby minimizing the need to read similar kernels from memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If kernels are read frequently from memory for convolution operations, then the neural network can process data, but power consumption and processing time increase significantly
Solution Approach 1:
The patent pre-calculates and stores kernel filters during the training phase, so that during inference, modulation kernels can be generated quickly using simple mathematical operations instead of reading full kernels from memory repeatedly. This preliminary preparation reduces runtime memory access requirements.
Solution Approach 2:
Instead of storing and reading multiple similar kernels from memory, the patent creates copies of a base kernel through mathematical modulation using stored kernel filters. This reduces memory storage requirements and read operations while maintaining the necessary kernel variations for different channels.
2Ease of operation
If multiple similar kernels are stored in memory for different channels, then convolution operations can be performed, but memory read time and data transfer increase
Solution Approach 1:
The patent merges multiple similar kernels into a single base kernel representation, with variations captured by kernel filters. This consolidation reduces memory storage requirements and enables faster generation of modulation kernels through mathematical operations rather than memory reads.
Solution Approach 2:
The base kernel serves multiple functions by being modulated with different kernel filters to generate kernels for different channels. This universal approach eliminates the need to store and read separate kernels for each channel, improving operation efficiency.
3Loss of energy
If kernel filters and modulation operations are used to reduce memory reads, then power consumption decreases, but device complexity increases
Solution Approach 1:
The patent changes the representation parameters of kernels by storing compact kernel filters instead of full kernels. This parameter transformation enables energy-efficient memory access while the modulation operations, though mathematically complex, are implemented through optimized computational routines.
Data Source
AI summary
A neural processing unit (NPU), a method for driving an artificial neural network (ANN) model, and an ANN driving apparatus are provided. The NPU includes a semiconductor circuit that includes at least one processing element (PE) configured to process an operation of an artificial neural network (ANN) model; and at least one memory configurable to store a first kernel and a first kernel filter. The NPU is configured to generate a first modulation kernel based on the first kernel and the first kernel filter and to generate second modulation kernel based on the first kernel and a second kernel filter generated by applying a mathematical function to the first kernel filter. Power consumption and memory read time are both reduced by decreasing the data size of a kernel read from a separate memory to an artificial neural network processor and/or by decreasing the number of memory read requests.