Neural Processing Unit Architecture for Fast, Low-Power AI Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional processing units, such as CPUs and GPUs, struggle with slow performance in artificial intelligence and machine learning processes due to the high computational demands and power consumption, which can lead to delays in applications requiring real-time processing, such as automated driving and military targeting systems.
Innovation Solution
The introduction of neural processing units (NPUs) with specialized hardware architectures that mimic the human brain, utilizing digital neurons to perform computations efficiently and locally, allowing for high data-level parallelism and reduced power consumption, with features like synaptic strength value adjustments and boost factors to enhance learning.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If conventional processing units (CPUs and GPUs) are used to execute AI and ML processes, then the system can perform computations, but the processing speed is slow and cannot meet real-time requirements
Solution Approach 1:
The patent segments the processing unit into specialized components: a neural network unit with multiple processing elements organized in arrays, separate weight memory, and dedicated interconnect structures. This segmentation allows parallel processing of neural network operations, achieving speeds 1,200 times faster during training and 10.3 times faster during inference compared to conventional units.
Solution Approach 2:
The patent changes the fundamental operating parameters by using fixed-point arithmetic instead of floating-point, and by organizing data in sparse distributed representations. These parameter changes enable more efficient computation with lower precision requirements, significantly improving processing speed while maintaining accuracy.
2Speed
If computational resources are increased to improve AI and ML processing performance, then processing speed may improve, but power consumption increases
Solution Approach 1:
The neural processing unit performs computations locally using on-chip weight memory and processing elements, eliminating the need to continuously fetch data from external memory. This self-service approach reduces memory access power consumption while maintaining high processing speed for AI workloads.
Solution Approach 2:
The patent replaces conventional von Neumann architecture with a dataflow architecture where computation occurs where data resides. This substitution eliminates the mechanical data movement between separate memory and processing units, reducing power consumption while improving computational efficiency for neural network operations.
3Productivity
If more computational resources are allocated to AI and ML processes, then processing capability improves, but the system becomes impractical for mobile devices with limited resources
Solution Approach 1:
The neural processing unit is designed as a universal accelerator that can handle various neural network architectures and workloads through configuration rather than hardware variation. A single NPU design can service multiple AI applications on mobile devices, providing high processing capability without increasing device complexity or requiring application-specific hardware for each use case.
Data Source
AI summary
Introduced here are integrated circuits (also referred to as “chips”) that can be implemented in a neural processing unit. At a high level, the goal of these chips is to provide higher performance for machine learning algorithms than conventional processing units would. To accomplish this, the neural processing unit can include multiple computing components, each of which is able to independently determine the overlap between encoded data provided as input and values stored in a memory.