Neural Network Video Encoding Syntax Generation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current video processing technologies, such as standard codecs, face inefficiencies in encoding and decoding due to reliance on brute-force search for rate-distortion optimization and are limited by quantization parameters, which affect the size and partitioning of coding, prediction, and transform units.
Innovation Solution
A neural network-based apparatus and method that processes video frames by generating a syntax element using a quantization parameter, pre-decoded reference images, and multiple frames, performing entropy encoding, and adjusting quantization based on adaptive instance normalization, allowing for intra- and inter-prediction, motion estimation, and compensation, while being compatible with standard codecs.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If brute-force search is used for rate-distortion optimization in standard codecs, then encoding accuracy is improved, but computational complexity and processing time increase significantly
Solution Approach 1:
The patent replaces the mechanical brute-force search mechanism with a neural network-based system. The neural network is trained to predict optimal encoding parameters (CU partition, PU partition, prediction mode, TU partition) directly, substituting the iterative computational search process with a learned prediction model that provides similar or better accuracy with significantly reduced computational complexity
Solution Approach 2:
The neural network performs preliminary learning during the training phase on large datasets of video sequences and ground truth encoding parameters. This preliminary action allows the network to encode optimal decisions in advance, eliminating the need for complex real-time searches during actual video encoding operations
2Ease of operation
If quantization parameters are used to determine CU, PU, and TU sizes in standard codecs, then encoding process is simplified, but compression efficiency and video quality are limited
Solution Approach 1:
The patent introduces dynamic parameter adjustment where the neural network adaptively determines encoding parameters based on local image characteristics. Instead of using fixed quantization parameter-based sizing, the system dynamically selects CU partition, PU partition, prediction mode, and TU partition sizes according to the actual content being encoded, achieving both simplicity and high compression efficiency
Solution Approach 2:
The system changes the fundamental parameter used for decision-making from quantization parameter (QP) to neural network predicted parameters. The neural network outputs optimal values for CU partition size, PU partition size, prediction mode, and TU partition size based on learned patterns, replacing the traditional QP-driven parameter selection and achieving superior compression efficiency while maintaining operational simplicity
3Productivity
If neural network is used for video compression, then compression efficiency and video quality are improved, but compatibility with standard codecs must be maintained
Solution Approach 1:
The neural network acts as an intermediary that generates syntax elements compatible with standard codec formats. The network outputs predictions for CU partition, PU partition, prediction mode, and TU partition that can be directly integrated into existing HEVC or other standard codec frameworks, serving as a bridge between advanced AI-based compression and traditional standardized decoding
Solution Approach 2:
The neural network is designed to be universally applicable across different video sequences and compatible with multiple standard codec formats. The same network architecture and training approach can be applied to enhance HEVC, AVC, or other codecs, providing a universal solution that maintains broad adaptability and compatibility while improving compression efficiency
Data Source
Figure 1
Figure 2
Figure 3
AI summary
An apparatus with video processing includes: one or more processors configured to: generate a syntax element processable by a target standard codec by inputting a quantization parameter, a predecoded reference image, and a plurality of frames comprised in a video to a neural network and compressing the plurality of frames, and generate a bitstream by performing entropy encoding on the syntax element.