Video Deblurring via 3D Convolution and GAN
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current video deblurring methods using 2D convolutional neural networks are limited in extracting spatio-temporal information, resulting in insufficiently real deblurred videos due to the lack of temporal feature extraction and reliance on pixel-based reconstruction loss functions.
Innovation Solution
A video deblurring method employing a generative adversarial network with 3D convolution processing on continuous image frames to extract spatio-temporal information, including spatial and temporal features, for enhanced deblurring processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If 2D convolution kernel is used for video deblurring, then the processing can be completed with simpler computational structure, but only spatial information in single frame can be extracted and temporal information between frames is lost
Solution Approach 1:
The patent transitions from 2D convolution to 3D convolution, adding a temporal dimension to the feature extraction process. The 3D convolution kernel operates across three dimensions (height, width, and time), enabling simultaneous extraction of spatial features within frames and temporal features across consecutive frames, thereby resolving the information loss without significantly increasing model complexity
2Ease of manufacture
If pixel-based reconstruction loss function is used for supervised information, then the training process is simpler, but the deblurred video appears insufficiently real
Solution Approach 1:
The patent changes the loss function parameter from traditional pixel-based reconstruction loss to perceptual loss based on deep learning features. This parameter change transforms the optimization objective to preserve higher-level semantic information and perceptual quality, making the deblurred videos appear more realistic while maintaining a relatively simple training framework
Data Source
AI summary
A method of video deblurring by an electronic device is described. The processing circuitry of the electronic device acquires N continuous image frames from a video clip the N being a positive integer, and the N continuous image frames including a blurry image frame to be processed. The processing circuitry of the electronic device performs three-dimensional (3D) convolution processing on the N continuous image frames with a generative adversarial network model, to acquire spatio-temporal information corresponding to the blurry image frame. The spatio-temporal information includes spatial feature information of the blurry image frame, and temporal feature information between the blurry image frame and a neighboring image frame of the N continuous image frames The processing circuitry of the electronic device performs deblurring processing on the blurry image frame by using the spatio-temporal information corresponding to the blurry image frame through the generative adversarial network model, to output a sharp image frame.