NPU Bitstream Decoding for Scalable Feature Map Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current technologies lack an effective method for machine-based image analysis, particularly in handling high-resolution and high-quality video data, and there is a need for improved processing units to handle advanced video coding standards like VVC and AI-driven tasks.
Innovation Solution
A neural processing unit (NPU) is developed to perform inference using artificial neural networks for decoding and encoding video or feature maps, incorporating processing elements that handle bitstreams with base and enhancement layers, enabling efficient image analysis and processing for machine tasks.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If VVC (H.266) video coding technology is adopted to achieve more than twice the compression efficiency of HEVC, then compression rate and processing capability are improved, but device complexity and requirement for advanced processing units increase
Solution Approach 1:
The processing unit is divided into multiple processing elements (PEs), each responsible for specific operations such as decoding base layer data, decoding enhancement layer data, or performing AI inference. This segmentation allows the system to handle VVC's complex processing requirements through distributed computation, improving compression efficiency while managing device complexity through modular architecture.
2Adaptability or versatility
If neural processing units with AI inference capabilities are integrated to enable machine-based image analysis, then image analysis capability is improved, but device complexity and integration difficulty increase
Solution Approach 1:
The neural processing unit is designed with multi-functionality, integrating AI inference capabilities alongside traditional VVC decoding functions. The processing elements can perform diverse tasks including base layer decoding, enhancement layer decoding, and AI-based image analysis, enabling the system to handle various applications from standard video processing to advanced machine learning tasks without requiring separate dedicated hardware for each function.
3Adaptability or versatility
If feature maps are processed in bitstream format with base and enhancement layers to enable scalable decoding, then adaptability to different quality levels is improved, but processing complexity and data handling difficulty increase
Solution Approach 1:
The bitstream is segmented into base layer data and enhancement layer data, with corresponding processing elements dedicated to decoding each layer. The base layer processing elements handle fundamental video data while enhancement layer processing elements add additional quality information. This segmentation enables scalable decoding where systems can process only the base layer for low-quality requirements or combine both layers for high-quality output, managing data processing complexity through structured organization.
Solution Approach 2:
The system performs preliminary decoding of the base layer before processing enhancement layers, establishing a foundation that simplifies subsequent processing. By preparing the base layer data first, the system reduces the complexity of handling enhancement layer data, as it can leverage already-decoded base layer information to efficiently process and integrate enhancement layer features.
Data Source
AI summary
A neural processing unit (NPU) for decoding video or feature map is provided. The NPU may comprise at least one processing element (PE) to perform an inference using an artificial neural network. The at least one PE may be configured to receive and decode data included in a bitstream. The data included in the bitstream may comprise data of a base layer. Alternatively, the data included in the bitstream may comprise data of the base layer and data of at least one enhancement layer. The data of the base layer included in the bitstream may include a first feature map. The data of the at least one enhancement layer included in the bitstream may include a second feature map.