Temporally Consistent Video Transformer for Flicker Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional transformer models used for video processing often result in temporally inconsistent regions, leading to noticeable flickering and artifacts in processed videos.
Innovation Solution
A TempFormer model is introduced, which incorporates a wavelet transform, spatial-temporal transformer blocks, and a recurrent architecture with an overlap loss term to enforce temporal consistency, reducing inconsistencies and flickering between frames.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If conventional transformer models are used to process video frames, then object detection and classification accuracy is improved, but temporal consistency deteriorates causing flickering and artifacts
Solution Approach 1:
The video processing is segmented into multiple passes: a first pass processes frames independently for accurate object detection, while a second pass refines the output by comparing consecutive frames to eliminate temporal inconsistencies. This segmentation allows each pass to optimize for its specific goal without compromising the other.
Solution Approach 2:
The system implements feedback by comparing processed frames with their adjacent neighbors and using the detected inconsistencies to guide corrections. The loss function incorporates temporal consistency terms that provide feedback signals to adjust the processed frames, reducing flickering while preserving detection accuracy.
2Productivity
If transformer models process each frame independently, then processing speed is improved, but temporal consistency deteriorates leading to noticeable flickering
Solution Approach 1:
The processing is divided into an initial independent frame processing stage that maintains high speed, followed by a correction stage that operates on the differences between consecutive frames. This segmentation preserves processing efficiency while adding temporal consistency enforcement.
Solution Approach 2:
Rather than processing all frames with full temporal analysis (which would reduce speed), the system applies temporal consistency corrections only where needed - specifically at frame boundaries and in regions detected to have inconsistencies. This partial action maintains speed while improving temporal consistency.
Data Source
AI summary
Techniques are disclosed for enhancing videos using a machine learning model that is a temporally-consistent transformer model. The machine learning model processes blocks of frames of a video in which the temporally first input video frame of each block of frames is a temporally second to last output video frame of a previous block of frames. After the machine learning model is trained, blocks of video frames, or features extracted from the video frames, can be warped using an optical flow technique and transformed using a wavelet transform technique. The transformed video frames are concatenated along a channel dimension and input into the machine learning model that generates corresponding processed video frames.