NPU Direct External-Memory Pathway for Low-Power Inference
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing electronic devices face challenges with power consumption, heat generation, memory requirements, and cost when embedding artificial neural network model inference technology due to inefficiencies in CPU and GPU performance, and neuromorphic integrated circuits are prone to noise and require large analog-to-digital converters.
Innovation Solution
A standalone, low-power, low-cost neural network processing unit (NPU) with a digital processing element array, SRAM memory, and an NPU scheduler that optimizes resource allocation and minimizes power consumption by reducing bit operations and memory usage based on predefined operation order information.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If CPU or GPU is used for neural network inference, then versatility is maintained, but power consumption and processing efficiency worsen
Solution Approach 1:
The system is segmented into two distinct processing units: a general-purpose CPU/GPU for non-neural network tasks and a specialized NPU for neural network inference. This segmentation allows each unit to be optimized for its specific function, with the NPU consuming significantly less power for neural network operations while the CPU/GPU maintains system versatility.
Solution Approach 2:
An NPU interface unit acts as an intermediary between the CPU/GPU and the NPU, managing data transfer and operation coordination. This intermediary enables the specialized NPU to handle neural network inference efficiently while the general-purpose processor maintains overall system control and versatility.
2Use of energy by moving object
If neuromorphic integrated circuit is used, then power consumption is reduced, but reliability worsens due to noise susceptibility
Solution Approach 1:
The patent replaces the analog mechanical system of neuromorphic circuits with a digital NPU architecture. This substitution eliminates the inherent noise susceptibility of analog circuits while maintaining low power consumption through specialized digital processing elements optimized for neural network operations.
3Reliability
If analog-to-digital converter is added to neuromorphic circuit, then reliability is improved, but device complexity worsens
Solution Approach 1:
The patent extracts the analog-to-digital conversion requirement entirely by designing a fully digital NPU architecture. This eliminates the need for large analog-to-digital converters while maintaining signal accuracy through digital processing, thereby reducing device complexity.
4Adaptability or versatility
If more hidden layers are added to neural network, then intelligence level is improved, but memory requirements and processing time worsen
Solution Approach 1:
The NPU implements local quality optimization by providing dedicated memory structures (weight memory and activation memory) specifically optimized for neural network operations. This localized memory architecture reduces the memory burden on system RAM and enables efficient handling of deep neural networks with multiple hidden layers.
5Measurement precision
If kernel reading frequency is increased, then processing accuracy is maintained, but processing speed worsens due to memory latency
Solution Approach 1:
The NPU implements preliminary action by pre-loading kernel weights into dedicated weight memory before inference operations begin. This pre-positioning of data eliminates repeated memory access latency during processing, maintaining processing accuracy while significantly improving processing speed through reduced memory wait times.
Solution Approach 2:
The NPU architecture enables continuous processing by maintaining kernels in accessible memory states throughout the inference process. This continuity eliminates the stop-start nature of repeated memory reads, allowing sustained high-speed processing without sacrificing accuracy.
Data Source
AI summary
A neural network processing unit (NPU) includes a plurality of processing elements for performing the ANN model computations using weight parameters and input activation data; an NPU internal memory operatively coupled to the plurality of processing elements for storing at least one of the weight parameters or the activation data; and a dedicated external memory interface configured for a direct data pathway to an external main memory system storing ANN model data, the direct data pathway facilitating transfer of a portion of the ANN model data, wherein the NPU is configured to receive the ANN model data via the dedicated external memory interface for processing.