Distributed NPU Feature Map Encoding for Lower Inference Data Transfer
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video coding technologies are optimized for human visual perception and do not efficiently address the increased computational load and data transmission requirements of artificial neural networks, particularly in machine inference tasks, leading to network congestion and power consumption issues.
Innovation Solution
A neural processing unit (NPU) is designed to distribute the processing of artificial neural networks across heterogeneous processors based on the size of output feature maps, minimizing data transfer and optimizing computational load and power consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional video coding technologies are used for machine inference, then video compression efficiency is improved, but computational load and data transmission requirements increase
Solution Approach 1:
The neural network processing is divided into multiple stages, with early stages processed on edge-type NPUs and later stages processed on server-type NPUs. This segmentation allows efficient utilization of computational resources by matching the processing complexity of each stage with the appropriate hardware platform, thereby reducing overall computational load while maintaining compression efficiency.
Solution Approach 2:
Feature maps serve as intermediaries between different processing stages and hardware platforms. The encoder generates feature maps that can be transferred between edge-type and server-type NPUs, enabling coordinated processing across heterogeneous devices. This intermediary mechanism allows the system to leverage the strengths of different hardware types while reducing the computational burden on any single platform.
2Power
If distributed processing across heterogeneous processors is implemented, then computational load is optimized, but device complexity increases
Solution Approach 1:
The encoder is designed with multi-functionality to operate with both edge-type and server-type NPUs, generating feature maps that can be processed across different hardware platforms. This universal design simplifies the overall system architecture by using a single encoder that can adapt to different processing scenarios, rather than requiring separate encoders for each hardware type.
Solution Approach 2:
The system dynamically adjusts processing parameters based on the type of NPU being used (edge-type or server-type). By changing parameters such as processing precision, feature map resolution, and computational operations according to the hardware capabilities, the system achieves optimized computational load distribution without requiring complex reconfiguration of the entire architecture.
3Productivity
If feature map data is transferred between processors, then computational tasks are distributed, but data transmission requirements increase
Solution Approach 1:
The system extracts and processes only the necessary feature map data at each processing stage, rather than transferring all intermediate data. By selectively extracting and processing only the essential features at early stages on edge-type NPUs, the system reduces the volume of data that needs to be transmitted to server-type NPUs, thereby decreasing data transmission requirements while maintaining effective computational task distribution.
Data Source
AI summary
According to an example of the present disclosure, a neural processing unit (NPU) capable of encoding is provided. The NPU comprises one or more processing elements (PEs) which perform operations for a plurality of layers of an artificial neural network and generate a plurality of output feature maps. The NPU also comprises an encoder which encodes at least one particular output feature map among a plurality of output feature maps into a bitstream and then transmits thereof.