Video Bitstream Decoding With Selective Neural Coding Offload
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural-network-based coding tools require high computational power, which devices with limited GPU capabilities, such as smartphones, cannot handle, leading to inefficiencies in video decoding.
Innovation Solution
Implement a flexible signaling mechanism to enable or disable neural-network-based coding tools at various levels or layers of the video coding scheme, allowing devices with varying computational power to decode video bitstreams effectively by utilizing cloud-assisted decoding.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If neural-network-based coding tools are used during video encoding, then video coding efficiency is improved, but computational power requirements increase beyond what mobile devices can provide
Solution Approach 1:
The patent introduces a cloud server as an intermediary between the video encoder and decoder. The cloud server performs the computationally intensive neural-network-based decoding operations, while the mobile device only handles lightweight tasks like receiving encoded video data and displaying decoded output. This mediator approach allows advanced coding tools to be used without burdening the mobile device's limited computational resources.
Solution Approach 2:
The patent shifts the computational workload from the spatial dimension (local processing on mobile device) to the network dimension (remote processing on cloud server). By utilizing cloud-based resources accessible through network connectivity, the system can employ sophisticated neural-network-based coding tools that would be impossible to run locally on resource-constrained mobile devices.
2Manufacturing precision
If neural-network-based coding tools are enabled, then decoding quality is improved, but device complexity increases
Solution Approach 1:
The patent extracts the complex neural-network-based decoding functionality from the mobile device and relocates it to a cloud server. The mobile device retains only essential decoding components for basic video format handling and display output. This extraction allows high decoding quality to be achieved through cloud-based neural networks while keeping the mobile device's complexity minimal.
3Productivity
If cloud-assisted decoding is implemented, then video decoding efficiency on mobile devices is improved, but network bandwidth consumption increases
Solution Approach 1:
The patent implements partial cloud-assisted decoding where only specific portions of the video processing pipeline (neural-network-based prediction and filtering stages) are offloaded to the cloud server. The mobile device continues to handle other decoding stages locally. This partial approach balances the trade-off by achieving improved decoding efficiency through cloud assistance while limiting network bandwidth consumption to only the necessary data exchanges for the offloaded functions.
Data Source
AI summary
A method for decoding a video from a video bitstream encoded using a video encoder, a system for decoding a video bitstream, and a non-transitory computer-readable medium are provided. A syntax element indicating a use of a neural-network-based coding tool during encoding a portion of the video bitstream is identified from the video bitstream. A value of the syntax element is determined. If the value of the syntax element indicates the neural-network-based coding tool is enabled during the encoding of the portion of the video bitstream, the portion of the video bitstream is decoded into pictures by using the neural-network-based coding tool. If the value of the syntax element indicates the neural-network-based coding tool is disabled during the encoding of the portion of the video bitstream, the portion of the video bitstream into pictures is decoded without using the neural-network-based coding tool. The decoded pictures are outputted for display.