Feature Tensor Temporal Resampling for Selective Picture-Level Upsampling
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video codecs optimized for human consumption are inefficient for compressing large 3D feature tensors generated by machine vision algorithms, leading to suboptimal performance in remote analysis and privacy concerns due to the need for transmitting raw data.
Innovation Solution
Implementing a method for selective temporal resampling and upsampling of feature tensors using learned shallow neural networks and explicit signaling flags to manage the decoding process, ensuring efficient compression and privacy protection in split inference scenarios.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional video codecs are used for compressing feature tensors, then compression is performed, but compression efficiency is suboptimal for machine vision data
Solution Approach 1:
The patent changes the temporal sampling rate parameter of feature tensors, transmitting only a subset of frames (e.g., every other frame) and reconstructing intermediate frames through temporal upsampling at the decoder. This parameter modification optimizes compression efficiency while maintaining sufficient accuracy for machine vision tasks.
Solution Approach 2:
The patent creates reconstructed copies of intermediate feature tensors through temporal upsampling at the decoder side. Instead of transmitting all original frames, the system transmits a subset and generates copies of intermediate temporal states through interpolation, reducing transmission data while maintaining functional equivalence.
2Quantity of substance
If temporal resampling is applied to reduce data size, then transmission efficiency improves, but decoding complexity increases
Solution Approach 1:
The patent inverts the traditional approach by performing upsampling at the decoder rather than the encoder. The encoder transmits downsampled feature tensors with minimal processing, while the decoder performs temporal upsampling to reconstruct the full temporal resolution, shifting complexity from encoding to decoding.
Solution Approach 2:
The patent introduces an intermediary temporal upsampling process that acts as a mediator between the transmitted downsampled frames and the required full-resolution frames. This intermediary reconstruction process uses learned shallow neural networks to generate intermediate temporal states without requiring complex transformations at either endpoint.
3Measurement precision
If feature tensors are transmitted without compression, then accuracy is maintained, but transmission bandwidth is excessive
Solution Approach 1:
The patent maintains continuity of useful information through temporal upsampling that reconstructs intermediate feature tensors between transmitted frames. This continuous reconstruction ensures that no critical temporal information is lost while significantly reducing the volume of data that needs to be transmitted across the network.
Data Source
AI summary
In various implementations, method and devices are disclosed that encode or decode a set of feature tensors used in Video Coding for Machine as a sequence of images as addressed in Features Coding Machine. For instance, the decoding method comprises obtaining an indication for enabling of a temporal resampling of a set of feature tensors at sequence level; obtaining an indication for estimating an upsampled set of feature tensors at picture level; and decoding the sequence of images by selectively activating/deactivating upsampling a set of feature tensors based on the indications. According to different variant, the indication for estimating an upsampled set of feature tensors at picture level may be derived or parsed from a syntax element fpps_inactive_upsampling_flag signaled in a Feature Picture Parameter Set (FPPS).