Generative Video Encoding Temporal Consistency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing video encoding and decoding methods using generative adversarial networks suffer from visual flickering and floating artifacts due to the lack of consideration for temporal similarity between reconstructed and to-be-encoded video frames.
Innovation Solution
A model training method that involves acquiring reference and to-be-encoded sample frames, deforming the reference frames to generate reconstructed frames, and using both spatial and temporal discriminators to assess authenticity and generate an adversarial loss value for training the generative model.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If only spatial domain similarity is considered during authenticity identification, then the identification process is simple, but the reconstructed video frame sequences exhibit visual flickering and floating artifacts with poor video reconstruction quality
Solution Approach 1:
The discriminator is segmented into two distinct components: a spatial discriminator that evaluates spatial domain similarity between individual reconstructed frames and to-be-encoded frames, and a temporal discriminator that evaluates temporal domain similarity by comparing spliced sequences of frames. This segmentation allows each discriminator to specialize in one aspect of quality assessment, resolving the contradiction by improving reliability through comprehensive evaluation while managing complexity through functional division
Solution Approach 2:
The solution transitions from evaluating only the spatial dimension (single frame similarity) to incorporating the temporal dimension (frame sequence consistency). By splicing multiple consecutive frames into a video clip and using the temporal discriminator to evaluate the spliced reconstructed video clip against the spliced to-be-encoded video clip, the system adds temporal domain assessment, thereby improving video reconstruction quality and eliminating flickering artifacts
2Reliability
If temporal domain similarity is considered by splicing frames and using a temporal discriminator, then video reconstruction quality improves, but the computational complexity and processing time increase
Solution Approach 1:
The method performs preliminary splicing of multiple consecutive frames into video clips before temporal domain evaluation. By pre-assembling the frame sequences in chronological order and preparing the spliced video clips in advance, the system reduces the computational burden during the actual temporal discrimination process, thereby managing training time while maintaining temporal consistency evaluation
Solution Approach 2:
The spatial and temporal discrimination processes are merged into a unified adversarial training framework where both discriminators operate simultaneously but independently on their respective domains. The loss functions from both discriminators are combined to train the generator, allowing efficient joint optimization that improves temporal consistency without excessive time penalty
3Productivity
If only single frame authenticity identification is performed, then the processing speed is fast, but the reconstructed video sequences suffer from flickering and floating artifacts
Solution Approach 1:
The system performs preliminary splicing of multiple consecutive frames into video clips before temporal domain evaluation. By pre-assembling the frame sequences in chronological order, the method enables efficient batch processing that maintains processing speed while allowing the temporal discriminator to evaluate overall sequence consistency, thereby eliminating flickering artifacts without significantly sacrificing productivity
Solution Approach 2:
The spatial discriminator's approach is copied and adapted for the temporal domain: just as the spatial discriminator compares individual reconstructed frames with to-be-encoded frames, the temporal discriminator compares spliced sequences (copies) of frames. This copying of the discrimination methodology across different domains allows fast processing to be maintained while extending evaluation to the temporal dimension to eliminate artifacts
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Embodiments of the present application provide a model training method, a video encoding method, and a decoding method. The model training method comprises: acquiring a reference sample frame and a plurality of consecutive to-be-encoded sample frames; deforming the reference sample frame through a generator in an initial generative model to generate reconstructed sample frames; inputting each reconstructed sample frame and the corresponding to-be-encoded sample frame into a first discriminator in the initial generative model to obtain a first identification result; splicing the to-be-encoded sample frames in a timestamp order to obtain a spliced to-be-encoded sample frame, and splicing the reconstructed sample frames to obtain a spliced reconstructed sample frame; inputting the spliced to-be-encoded sample frame and the spliced reconstructed sample frame into a second discriminator in the initial generative model to obtain a second identification result; obtaining an adversarial loss value based on the first identification result and the second identification result; and training the initial generative model based on the adversarial loss value. The present application enables that the reconstructed video frame sequence and the to-be-encoded video frame sequence are kept consistent in time domain, thereby improving the reconstruction quality.