Segmentable Neural Network Unit with Rotator for Memory Bottleneck Resolution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current neural network processors face inefficiencies in computation due to memory bottlenecks and complex data access patterns, particularly in Convolutional Neural Networks (CNNs), which hinder performance and efficiency.
Innovation Solution
The design incorporates a neural network unit (NNU) with a dynamically configurable neural processing unit (NPU) that employs a flexible memory hierarchy and specialized instructions like MTNN and MFNN to optimize data access and processing, allowing for efficient execution of neural network operations such as convolution and pooling, and supports operations like multiply-accumulate and activation functions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a traditional memory hierarchy is used in neural network processors, then general-purpose computation is supported, but memory bottlenecks occur and computation efficiency decreases
Solution Approach 1:
The memory system is segmented into multiple specialized memory banks (weight memory, activation memory, intermediate memory) that are independently accessible by multiple NPUs simultaneously. This segmentation allows parallel memory access operations, eliminating the bottleneck of sequential memory access in traditional architectures.
Solution Approach 2:
The patent introduces a new dimension of memory organization by creating a multi-level memory hierarchy with direct mapping between NPUs and memory banks. Each NPU has direct access to multiple memory banks through separate interfaces, adding spatial dimensionality to memory access patterns and enabling simultaneous read/write operations across different memory levels.
2Adaptability or versatility
If fixed-precision arithmetic is used in NPUs, then hardware complexity is reduced, but adaptability to different neural network requirements is limited
Solution Approach 1:
The NPU architecture employs dynamic precision selection where the precision of arithmetic operations can be adjusted on-the-fly based on the specific neural network layer and computation requirements. Each NPU can independently configure its multiplier and accumulator precision, allowing adaptation from low-precision (e.g., 8-bit) to high-precision (e.g., 32-bit) operations without hardware redesign.
Solution Approach 2:
The patent implements parameter-based precision control where precision levels are changed through configuration parameters rather than hardware changes. The NPU accepts precision parameters that dynamically adjust the bit-width of multipliers, accumulators, and data paths, enabling the same hardware to efficiently handle different precision requirements of various neural network models.
3Speed
If more memory banks are added to increase bandwidth, then data access speed improves, but memory controller complexity increases
Solution Approach 1:
Each NPU is equipped with self-service memory interface capabilities, including independent address generation, read/write control, and data validation for multiple memory banks. The NPUs autonomously manage their own memory access operations without requiring centralized arbitration or complex controller logic, thereby increasing data access speed while keeping the memory controller relatively simple.
Solution Approach 2:
The memory interface design employs universal control logic that can handle multiple memory banks with the same access patterns. The memory controller uses a standardized interface protocol that works across all memory banks, allowing the same control unit to manage numerous memory banks without increasing complexity proportionally to the number of banks.
4Productivity
If specialized neural network instructions are implemented, then neural network computation performance improves, but instruction set complexity increases
Solution Approach 1:
The patent merges multiple specialized neural network operations (multiply-accumulate, activation functions, bias addition, ReLU) into single composite instructions that can be executed in one clock cycle. For example, a single instruction can perform weighted multiplication, accumulation, activation function application, and bias addition simultaneously, greatly improving throughput while keeping the instruction set relatively compact.
Solution Approach 2:
The instruction set includes preliminary action instructions that prepare data in advance for subsequent neural network operations. These instructions pre-load weights, pre-compute activation functions, and pre-configure NPU parameters before the main computation begins, enabling the subsequent computation stages to proceed at maximum speed without additional setup overhead.
Data Source
Figure 1
Figure 2~3
Figure 4~6A
AI summary
First/second memories hold rows of N weight/data words. Each of N processing units (PU) of index J have a register, an accumulator having an output, an arithmetic unit that performs an operation thereon to accumulate a result, the first input receives the output of the accumulator, the second input receives a respective first memory weight word, the third input receives a respective data word output by the register, and multiplexing logic receives a respective second memory data word and a data word output by the register of PU J-1 and outputs a selected data word to the register. PU J-1 for PU 0 is PU N-1. The multiplexing logic of PU N/4 also receives the data word output by the register of PU (3N/4)-1. The multiplexing logic of PU 3N/4 also receives the data word output by the register of PU (N/4)-1.