Feature Map Encoding With Selective Video Tools for CNN Tensors
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video compression technologies face challenges in efficiently encoding and decoding tensors from convolutional neural networks (CNNs), particularly in distributed systems, due to high computational complexity and memory requirements, which are exacerbated by the need for high processing power in cloud implementations and limited flexibility in edge devices.
Innovation Solution
A method and apparatus for generating and decoding encoded data from CNNs, utilizing a feature map encoding and decoding system that selectively applies different functions based on the presence of feature maps, optimizing the use of VVC tools for efficient packing and quantization of feature maps into planar frames, reducing overhead and ensuring resilient task performance across varying bitrates.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If regular video encoding functions are used for encoding CNN feature maps, then encoding flexibility is maintained, but computational complexity and processing time increase
Solution Approach 1:
The patent segments the encoding process into two distinct paths: one optimized for regular video data and another specialized for CNN feature maps. The system divides feature maps into planar frames and applies selective encoding functions based on data type, allowing simplified processing for feature maps while maintaining full functionality for video data.
Solution Approach 2:
The patent implements dynamic function selection where the encoding approach changes based on the input data type. The system dynamically determines whether to apply full video encoding functions or simplified feature map encoding functions, optimizing computational resources while maintaining encoding flexibility when needed.
2Power
If cloud-based CNN processing is used, then processing power is sufficient, but memory requirements and costs increase
Solution Approach 1:
The patent extracts the CNN inference process from the cloud environment and implements it at the edge device. By processing feature maps locally and only transmitting encoded results, the system eliminates the need for large cloud-based memory allocations while maintaining adequate processing power through optimized local computation.
Solution Approach 2:
The patent transforms the processing architecture from a centralized cloud-based model to a distributed edge-based model. This dimensional shift in processing location reduces memory requirements by eliminating the need for large cloud memory allocations while maintaining processing capabilities through local inference.
3Reliability
If feature maps are encoded using all video encoding functions, then encoding completeness is ensured, but overhead and processing time increase
Solution Approach 1:
The patent applies partial encoding action by selecting only the necessary encoding functions for feature maps rather than applying all video encoding functions. The system identifies and applies only the subset of functions required for feature map encoding, reducing processing time while maintaining encoding completeness for the specific data type.
4Productivity
If CNN inference is performed on entire video frames, then batch processing efficiency is improved, but memory consumption increases
Solution Approach 1:
The patent segments the video frame processing into individual feature map planes that can be processed independently. By dividing the batch processing into manageable planar segments, the system maintains batch processing efficiency while reducing peak memory consumption through progressive processing of smaller data units.
Data Source
AI summary
An apparatus for generating first encoded data and second encoded data. The apparatus comprises a determining unit for determining whether the apparatus generates encoded data including encoded data of a feature map based on a neural network. The apparatus also comprise an encoding unit for generating the first encoded data using a plurality of functions for encoding video data, in a case where the apparatus generates the first encoded data in a form of encoded video data not including the encoded data of the feature map. The encoding unit generates the encoded data of the feature map using a first part of the plurality of functions but not using a second part of the plurality of functions, in a case where the apparatus generates the second encoded data including the encoded data of the feature map.