Warping-Based Neural Video Codec for Mobile Full HD Decoding
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network-based video compression techniques face challenges in computational efficiency, particularly on battery-powered devices like smartphones, due to large floating-point networks and pixel-dense warping operations, which are too intensive for mobile deployment.
Innovation Solution
A neural video coder architecture optimized for mobile deployment, utilizing predictive modeling, block-based warping, and parallel entropy coding on a GPU, with efficient quantization of weights and activations, enabling real-time full HD video decoding on mobile devices.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If large floating-point networks and pixel-dense warping operations are used, then video compression performance is improved, but computational complexity and power consumption increase making mobile deployment infeasible
Solution Approach 1:
The patent segments the continuous pixel-dense warping operation into discrete block-based warping operations. Instead of processing every pixel individually with dense operations, the video frame is divided into blocks that undergo warping transformations. This segmentation reduces the computational complexity from O(N²) for pixel-dense operations to O(N×B) for block-based operations where B is the block size, making mobile deployment feasible while preserving essential motion compensation functionality.
Solution Approach 2:
The patent changes the precision parameter from floating-point to integer arithmetic throughout the neural network pipeline. By quantizing weights and activations to lower precision integer formats (e.g., INT8), the computational complexity is reduced and power consumption is lowered, enabling mobile deployment. This parameter change maintains adequate compression performance while making the system suitable for battery-powered devices.
2Manufacturing precision
If large floating-point networks are used, then video compression performance is improved, but power consumption increases making battery-powered device deployment infeasible
Solution Approach 1:
The patent systematically changes the precision parameter from floating-point to integer arithmetic throughout the neural network pipeline. By quantizing weights and activations to lower precision integer formats (e.g., INT8), the power consumption is reduced by a factor of 4-8× compared to floating-point operations. This parameter change enables deployment on battery-powered mobile devices while maintaining adequate video compression performance through careful quantization-aware training.
3Productivity
If quantization is applied to neural network variables, then speed and power consumption are improved, but compression efficiency deteriorates due to information loss in entropy coding variables
Solution Approach 1:
The patent applies quantization-aware training as a preliminary action before actual deployment. During the training phase, the quantization process is simulated and its effects on the neural network variables are pre-compensated for. This preliminary action allows the model to learn robust representations that are resilient to quantization effects, thereby minimizing information loss in entropy coding variables while maintaining the speed and power consumption benefits of integer arithmetic in the deployed system.
Data Source
AI summary
An example computing device may include memory and one or more processors. The one or more processors may be configured to parallel entropy decode encoded video data from a received bitstream to generate entropy decoded data. The one or more processors may be configured to predict a motion vector based on the entropy decoded data. The one or more processors may be configured to decode a motion vector residual from the entropy decoded data. The one or more processors may be configured to add the motion vector residual and motion vector. The one or more processors may be configured to warp previous reconstructed video data with an overlapped block-based warp function using the motion vector to generate predicted current video data. The one or more processors may be configured to sum the predicted current video data with a residual block to generate current reconstructed video data.