CPU Cache Arbitration for Direct NPU Data Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing electronic devices face delays and overheads in processing neural network computations due to inefficient communication and context switching between central processing units (CPUs) and neural network processing units (NNPUs), particularly in tasks like automatic speech recognition, leading to increased energy consumption and inference delays.
Innovation Solution
The implementation of an arbiter in the CPU to connect a cache memory directly to an interconnect, allowing the NPU to exchange data with the cache memory without going through the memory, thereby reducing delays and overheads by enabling direct data transfer between the CPU and NPU.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If the NPU exchanges data through the memory, then data transfer is possible, but memory access delays increase and energy consumption increases
Solution Approach 1:
The patent introduces an arbiter as an intermediary component that enables direct data transfer between the NPU and cache memory, bypassing the traditional memory path. The arbiter manages the connection and data flow, allowing the NPU to access cache memory directly through the arbiter interface, thus reducing both transfer time and energy consumption compared to going through the memory subsystem.
Solution Approach 2:
The patent segments the memory hierarchy into cache memory directly accessible by NPU and main memory accessible by CPU, with the arbiter managing the cache portion. This segmentation allows the NPU to operate with a dedicated, faster memory access path for frequently used data, reducing the need to access slower main memory and thereby improving speed and reducing energy consumption.
2Productivity
If the CPU and NPU communicate through memory, then communication is established, but context switching overheads increase
Solution Approach 1:
The arbiter serves as a mediator that enables direct communication between the NPU and cache memory, eliminating the need for CPU intervention in data transfer operations. This direct path allows the NPU to perform computations independently without frequent context switching with the CPU, thereby improving computational efficiency and reducing context switching overheads.
Solution Approach 2:
The system pre-loads data into the cache memory that the NPU will need for computation. By having data ready in the cache before the NPU needs it, the system eliminates waiting time and reduces the frequency of context switching between CPU and NPU, thereby improving productivity and reducing time loss.
3Productivity
If the NPU is initialized by the CPU, then the NPU can process neural network computations, but inference delays increase during initialization
Solution Approach 1:
The CPU performs preliminary initialization of the NPU and pre-loads necessary data into the cache memory before the NPU begins its computation tasks. This preliminary action ensures that when the NPU starts processing, all required data is already available in the cache, eliminating initialization delays during actual computation and improving overall throughput.
Solution Approach 2:
The system maintains continuous useful action by keeping the NPU initialized and ready with data in the cache memory. Once initialized, the NPU can continuously process neural network computations without interruption or re-initialization delays, thereby improving productivity while the initial initialization time is amortized over continuous operation.
Data Source
AI summary
An electronic device according to various embodiments may comprise: a memory and a processor operatively connected to the memory. The processor may include: a central processing unit including a cache memory; a neural processing unit; and an interconnector connected to the neural processing unit. The central processing unit may include an arbiter for connecting the cache memory to any one of a processing circuit of the central processing unit and the interconnector. The neural processing unit may exchange data with the cache memory through the arbiter.