Neural Network Video Encoding Using Virtual Frames
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current image encoding/decoding technologies face challenges in efficiently handling high-resolution and high-definition video data, particularly in predicting pixel values for accurate encoding and decoding, especially with the increasing demand for UHD content.
Innovation Solution
The implementation of a neural network-based method for video processing, utilizing convolutional neural networks, Long Short Term Memory (LSTM) networks, and deconvolutional networks for generating feature vectors and prediction frames, along with the concept of virtual frames for inter prediction, to enhance prediction accuracy and efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If traditional inter-prediction and intra-prediction technologies are used for high-resolution video encoding, then encoding can be performed on UHD content, but prediction accuracy deteriorates due to the limitations of conventional methods in handling high-definition data
Solution Approach 1:
The patent replaces traditional mechanical prediction methods (inter-prediction and intra-prediction algorithms) with a neural network-based system. The neural network automatically learns prediction patterns from training data, substituting the manual rule-based approaches with an adaptive intelligent system that achieves higher prediction accuracy for high-definition video content.
Solution Approach 2:
The patent transforms the prediction problem by changing from direct pixel value prediction to feature vector prediction. The neural network processes extracted features (motion vectors, gradient information, texture characteristics) and generates predicted feature vectors, which are then transformed back to pixel values. This parameter transformation enables more effective handling of high-resolution data.
2Measurement precision
If neural network-based prediction is implemented, then prediction accuracy improves for high-resolution videos, but computational complexity and processing time increase
Solution Approach 1:
The patent performs feature extraction and neural network training in advance before actual video encoding. The neural network is pre-trained on large datasets of video content, so during real-time encoding, only inference is required rather than full training. This preliminary preparation significantly reduces processing time during actual video compression operations.
Solution Approach 2:
The patent divides the video processing into separate stages: feature extraction, neural network prediction, and residual calculation. By segmenting the processing pipeline, the system can optimize each stage independently and parallelize operations, reducing overall processing time while maintaining high prediction accuracy.
3Productivity
If feature vectors and virtual frames are generated using neural networks, then encoding efficiency improves, but device complexity and memory requirements increase
Solution Approach 1:
The patent introduces feature vectors as an intermediary representation between the original video frames and the prediction process. Instead of directly predicting pixel values, the system extracts meaningful features (motion, gradients, textures) and operates on these simplified representations. This intermediary layer reduces the complexity of direct frame-to-frame prediction while improving encoding efficiency.
Data Source
AI summary
Disclosed herein are a video decoding method and apparatus and a video encoding method and apparatus. A virtual frame is generated by a video generation network including a generation encoder and a generation decoder. The virtual frame is used as a reference frame in inter prediction for a target. Further, a video generation network for inter prediction may be selected from among multiple video generation networks, and inter prediction that uses the selected video generation network may be performed.