Video super-resolution method and system, and electronic device and storage medium

Abstract

Disclosed in the present invention are a video super-resolution method and system, and an electronic device and a storage medium. The method comprises: establishing a video super-resolution model in a complex motion scenario; and training and testing the model. The model comprises a feature extraction sub-module, a cross-frame alignment sub-module, a feature enhancement sub-module and an up-sampling reconstruction sub-module, wherein the feature extraction sub-module is used for extracting a shallow feature from input consecutive video frames; the cross-frame alignment sub-module uses deformable convolution to perform feature-level alignment on different frames, so as to realize cross-frame spatio-temporal information fusion and motion compensation; the feature enhancement sub-module performs enhancement on features; and the up-sampling reconstruction sub-module performs resolution upscaling on enhanced features, which are output by the feature enhancement sub-module, and reconstructs the enhanced features, which have been subjected to resolution upscaling, into video frames. By means of the present invention, the super-resolution effect of a video in a complex motion scenario is improved, and the quality and visual effect of the video are enhanced.

Description

Video super-resolution methods, systems, electronic devices and storage media Technical Field

[0001] This invention belongs to the field of image technology, specifically relating to a video super-resolution method, system, electronic device, and storage medium. Background Technology

[0002] With the rise of audio-visual entertainment software, such as the popularity of short video platforms, people often desire to access and watch high-resolution videos. High-resolution video means a high pixel density in video frames, providing more detail that is indispensable in many practical applications. For example, high-resolution medical imaging helps doctors make more accurate diagnoses; high-resolution surveillance video provides more effective security; and high-resolution satellite imagery makes it easy to distinguish similar objects. However, high-resolution digital cameras, camcorders, or smartphones are usually expensive because acquiring high-resolution video requires higher-precision optical and image sensors. These high-precision optical and image sensors are costly and present technological barriers. As a result, a large amount of low-resolution, low-quality video exists in our real world, while the demand for high-definition video continues to expand.

[0003] Video super-resolution technology is a new technique to increase the spatial resolution of videos, thereby overcoming the limitations of sensor and optical manufacturing technologies. Video super-resolution (or simply video super-resolution) aims to restore low-resolution, noisy, low-quality videos to high-resolution, high-quality videos. Video super-resolution technology is closely related to our lives and is playing a crucial role in various industries such as video surveillance, virtual reality image enhancement, and medical image enhancement. Furthermore, it can also AI-restore old, low-resolution films and television programs (upgrading them to high-definition or ultra-high-definition video), which is strong evidence of the application of video super-resolution technology in commercial fields.

[0004] Traditional video super-resolution methods mostly employ optical flow to estimate the motion of adjacent and reference frames for motion compensation and alignment; or they use deformable convolution-based methods to align adjacent and reference frames. However, optical flow methods have low alignment accuracy, while deformable convolution-based methods typically use one or more ordinary convolutional layers to uniformly learn the motion offsets between different adjacent and reference frames. These methods lack the ability to learn offsets and have a small receptive field, resulting in low robustness to complex scenes, especially those with large motion. Furthermore, our low-resolution videos often contain complex and diverse motion scenes, and most existing super-resolution technologies cannot reconstruct high-quality video data, failing to meet users' visual needs. Summary of the Invention

[0005] To address the aforementioned problems, this invention provides a video super-resolution method, system, electronic device, and storage medium, aiming to improve the super-resolution effect of videos in complex motion scenes and enhance video quality and visual effects.

[0006] According to a first aspect of the present disclosure, a video super-resolution method is provided, the method comprising the following steps:

[0007] A video dataset containing multiple different scenes is acquired and preprocessed. The preprocessed video dataset is then divided into a training dataset and a test dataset.

[0008] A video super-resolution model is established, comprising a feature extraction submodule, a cross-frame alignment submodule, a feature enhancement submodule, and an upsampling reconstruction submodule. The feature extraction submodule extracts shallow features from consecutive input video frames. The cross-frame alignment submodule uses deformable convolution to perform feature-level alignment on different frames based on the shallow features output by the feature extraction submodule, achieving spatiotemporal information fusion and motion compensation across frames. The feature enhancement submodule enhances the spatiotemporal fusion features output by the cross-frame alignment submodule. The upsampling reconstruction submodule amplifies the resolution of the enhanced features output by the feature enhancement submodule and reconstructs the amplified enhanced features into video frames.

[0009] The video super-resolution model is trained and tested using the training dataset and the test dataset, respectively.

[0010] Video super-resolution is achieved using the video super-resolution model that has been trained and tested.

[0011] In some embodiments, the cross-frame alignment submodule performs motion compensation based on different frame rate groups, including: aligning the features of adjacent frames and reference frames in a decoupled manner based on adjacent frames containing different time series in different groups, fusing the aligned features of adjacent frames with the features of reference frames through a convolutional layer, and inputting the fused features into the feature enhancement submodule.

[0012] In some embodiments, the cross-frame alignment submodule is specifically implemented by including the following steps:

[0013] Reference frames and adjacent frames are determined based on the feature sequences of the input video frames;

[0014] The feature sequences are grouped according to different frame rates. This is because the distance of adjacent frames from the reference frame in the time series (i.e., frame rate) and the motion offset of adjacent frames relative to the reference frame are both proportional and inversely proportional to the similarity between adjacent frames and the reference frame. Therefore, more efficient grouping and alignment of adjacent frames with different frame rates can be achieved by decoupling.

[0015] In each group, a set of cascaded dilated convolutions and one layer of ordinary convolutions are used to learn the offsets of adjacent frames and reference frames in the feature layer;

[0016] The offset of each group and the features of adjacent frames are input into a deformable convolutional neural network to obtain the aligned features.

[0017] The Concatenation operation is used to concatenate the reference frame features and the aligned features on the channel. Then, a simple convolutional layer is used to effectively aggregate the concatenated features to obtain the output features of the cross-frame alignment submodule.

[0018] In some embodiments, the video super-resolution model is trained and tested using the training dataset and the test dataset, respectively, specifically including:

[0019] The training datasets from various different scenarios are sequentially input into the video super-resolution model;

[0020] During iterative training, the loss value between the video frames generated by the video super-resolution model and the true values ​​is calculated;

[0021] The video super-resolution model is trained with the goal of reducing the loss value, and the network weights are updated and optimized simultaneously until convergence.

[0022] The reconstruction performance of the video super-resolution model was evaluated using the test dataset.

[0023] In some embodiments, the loss value is calculated based on the Charbonnier loss function.

[0024] In some embodiments, the feature enhancement submodule includes no fewer than three feature enhancement blocks, each of which consists of a convolutional kernel of size 3. A standard convolutional layer of size 3 and a convolutional kernel of size 3 It consists of 3 grouped convolutional layers.

[0025] In some embodiments, the cascaded dilated convolution includes three dilated convolutions with a kernel size of 3.

[0026] According to a second aspect of the present disclosure, a video super-resolution system is provided, the system comprising:

[0027] The dataset acquisition module is used to acquire video datasets containing multiple different scenes and preprocess them, dividing the preprocessed video datasets into training datasets and test datasets.

[0028] A model building module is used to build a video super-resolution model. The video super-resolution model includes a feature extraction submodule, a cross-frame alignment submodule, a feature enhancement submodule, and an upsampling reconstruction submodule. The feature extraction submodule extracts shallow features from consecutive input video frames. The cross-frame alignment submodule uses deformable convolution to perform feature-level alignment on different frames based on the shallow features output by the feature extraction submodule, achieving cross-frame spatiotemporal information fusion and motion compensation. The feature enhancement submodule enhances the spatiotemporal fusion features output by the cross-frame alignment submodule. The upsampling reconstruction submodule amplifies the resolution of the enhanced features output by the feature enhancement submodule and reconstructs the amplified enhanced features into video frames.

[0029] The model training and testing module is used to train and test the video super-resolution model using the training dataset and the test dataset, respectively.

[0030] A video super-resolution implementation module is used to implement video super-resolution using the video super-resolution model that has been trained and tested.

[0031] According to a third aspect of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the video super-resolution method described above.

[0032] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, the storage medium storing computer instructions that, when executed by a processor, implement the steps of the video super-resolution method described above.

[0033] This disclosure provides a video super-resolution method, system, electronic device, and storage medium. The method includes establishing a video super-resolution model. A feature extraction submodule extracts features from consecutive input video frames using one or more cascaded convolutional layers, laying the foundation for efficient motion compensation at the feature level. A cross-frame alignment submodule uses a group alignment mechanism to perform fine-grained alignment and fusion of adjacent frames at different frame rates, decoupling adjacent frames from a temporal sequence perspective. Different alignment submodules are used for different groups, making alignment more flexible. For example, the offset learned by the high-frame-rate group with greater motion can be adjusted accordingly. Based on this, the present invention can continuously manage the spatiotemporal relationships between frames, achieving efficient inter-frame alignment and motion compensation, and outputting aligned depth features. Furthermore, the cross-frame submodule based on dilated convolutional groups can quickly learn high-precision offsets. The alignment submodule method avoids the inaccuracies of single optical flow alignment and solves the limitations of traditional deformable convolutional methods (insufficient alignment capability in large motion and complex scenes), while also avoiding the computational cost of 3D convolution. In other words, it achieves efficient and accurate cross-frame motion compensation with lower computational cost. The feature enhancement submodule consists of six efficient feature enhancement blocks, each consisting of a convolutional kernel of size 3. A standard convolutional layer of size 3 and a convolutional kernel of size 3 The model consists of 3 group convolutional layers. The group convolutional layer adopts the idea of ​​group convolution, which has a lower computational cost compared to ordinary convolution. Ordinary convolution is used to solve the problem of difficult interaction between channels of different groups caused by group convolution. The efficient feature enhancement block in this invention integrates the advantages of ordinary convolution and group convolution, while effectively reducing the computational complexity of the model, which helps to achieve an effective balance between model performance and computational cost.

[0034] In summary, the video super-resolution method, system, electronic device, and storage medium proposed in this invention for complex motion scenes intelligently divide the input continuous video frames temporally through a grouping alignment mechanism, improving compatibility with complex motion scenes (especially large motion scenes). Secondly, by rapidly processing the features of different video frames, efficient spatiotemporal information fusion is achieved, enhancing information extraction capabilities. Furthermore, through a novel feature enhancement mechanism, the details and textures of the video frames are fully preserved, which is beneficial to the final reconstruction result and results in better visual effects. Thanks to these advantages, the video super-resolution method, system, electronic device, and storage medium of this invention can achieve high super-resolution performance with a relatively low number of parameters, and can be widely promoted and applied in practical industrial scenarios.

[0035] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0036] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0037] Figure 1 is a flowchart of the video super-resolution method in an embodiment of the present invention;

[0038] Figure 2 is a block diagram of a video super-resolution model in a complex motion scene according to an embodiment of the present invention;

[0039] Figure 3 is a flowchart of the cross-frame alignment submodule in an embodiment of the present invention;

[0040] Figure 4 is a schematic diagram of the video super-resolution system structure in an embodiment of the present invention;

[0041] Figure 5 is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0042] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present invention are shown in the drawings, not the entire structure.

[0043] Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of these steps can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the steps can be rearranged. The process can be terminated when its operation is complete, but may also have additional steps not included in the figures. The process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0044] Video super-resolution differs from image super-resolution. In video, consecutive frames are significantly correlated in time and space. While the shooting environment and content of the same segment or scene are similar, there can be motion and offset between different frames. Therefore, when performing video super-resolution, a key consideration is how to achieve frame alignment to better extract and fuse the spatiotemporal information from different frames.

[0045] Given these constraints, efficient inter-frame alignment of consecutive video frames from different input scenes is a key problem that video super-resolution methods urgently need to solve.

[0046] Inter-frame alignment, also known as motion compensation, involves dividing consecutive video frames into adjacent frames and a reference frame that needs to be processed. Inter-frame alignment refers to calculating the pixel motion offset between adjacent frames and the reference frame, and then aligning the adjacent frames to the reference frame based on the obtained offset. Currently, the main inter-frame alignment methods are optical flow and deformable convolution.

[0047] Both motion compensation and deformable convolution methods aim to extract inter-frame motion information. Optical flow calculates the spatiotemporal correlation and motion changes between adjacent frames and a reference frame, thus warping adjacent frames to align them with the reference frame. Deformable convolution adds an offset to a standard convolution kernel, resulting in a variable-shape kernel, which is then convolved with input features. Therefore, input video frames can be fused across frames using deformable convolution. Optical flow is simple and straightforward, but its alignment accuracy is not high. Deformable convolution methods typically use one or more convolutional neural network layers to learn motion offsets, providing diverse biases, strong robustness, and relatively good alignment results. However, the learning cost for offsets is high, and the receptive field of several stacked convolutions is limited. For scenes with large motion, its alignment accuracy needs further improvement, especially since the learned offsets are often too small, resulting in weak motion compensation capabilities in large-motion scenes.

[0048] Furthermore, 3D convolution and recurrent convolutional neural networks (RCNNs) can also be used to implicitly fuse spatiotemporal information between adjacent frames and reference frames. That is, without explicitly performing inter-frame alignment of the video, the excellent spatiotemporal modeling performance of 3D convolution or RCNNs can be leveraged to implicitly extract and fuse motion information between different temporal frames while simultaneously extracting or enhancing features. This eliminates the need for a motion compensation module and reduces computational costs, but it lacks robustness in complex scenes.

[0049] Based on this, embodiments of the present invention provide the following embodiments for a video super-resolution method, system, electronic device, and storage medium:

[0050] One embodiment illustrates a video super-resolution method, as shown in Figure 1. The method includes the following steps:

[0051] S1. Obtain a video dataset containing multiple different scenes and preprocess it. Divide the preprocessed video dataset into a training dataset and a test dataset.

[0052] In the specific implementation process, a video dataset containing at least 50 complex motion scenes is acquired. After preprocessing, the dataset is further divided into a training dataset and a test dataset. The dataset consists of consecutive video frames containing adjacent frames and reference frames.

[0053] Specifically, video datasets for training and testing can be collected through various means. For example, searching for keywords related to the scene using search engines to download various free video datasets; obtaining video data stored in public database servers; or recording video data using devices such as mobile phones and cameras. After data preprocessing (such as removing abnormal video frames and standardizing the resolution of different frames), the dataset is randomly divided into training and testing datasets. The training dataset accounts for 85% and is used to train the video super-resolution model, while the testing dataset accounts for 15% and is used to test the super-resolution performance of the generated model. By inputting low-resolution, low-quality videos, high-resolution, high-quality videos generated by the embodiments of this invention can be obtained.

[0054] S2. Establish a video super-resolution model, which includes a feature extraction submodule, a cross-frame alignment submodule, a feature enhancement submodule, and an upsampling reconstruction submodule. The feature extraction submodule extracts shallow features from consecutive input video frames. The cross-frame alignment submodule uses deformable convolution to perform feature-level alignment on different frames based on the shallow features output by the feature extraction submodule, achieving cross-frame spatiotemporal information fusion and motion compensation. The feature enhancement submodule enhances the spatiotemporal information fused features output by the cross-frame alignment submodule. The upsampling reconstruction submodule amplifies the resolution of the enhanced features output by the feature enhancement submodule and reconstructs the amplified enhanced features into video frames.

[0055] S3. Train and test the video super-resolution model using the training dataset and the test dataset respectively;

[0056] S4. Implement video super-resolution using the video super-resolution model completed through training and testing.

[0057] In the specific implementation process, as shown in Figure 2, the video super-resolution model for complex motion scenes consists of a feature extraction submodule, a cross-frame alignment submodule, a feature enhancement submodule, and an upsampling reconstruction submodule, among which...

[0058] The feature extraction submodule is responsible for extracting shallow features from consecutive video frames (including adjacent frames and reference frames) in complex input scenes, laying the foundation for subsequent cross-frame alignment and fusion. Specifically, the feature extraction submodule uses one or several cascaded convolutional layers to fully extract features from the consecutive input video frames, laying the foundation for further efficient motion compensation at the feature level.

[0059] The cross-frame alignment submodule performs motion compensation based on different frame rates, including: aligning the features of adjacent frames and the reference frame in a decoupled manner based on adjacent frames containing different time series in different groups, fusing the aligned adjacent frames with the reference frame through a convolutional layer, and inputting the fused features into the feature enhancement submodule.

[0060] In the specific implementation process, as shown in Figure 3, the cross-frame alignment submodule uses deformable convolution to perform feature-level fast alignment of different frames, thereby realizing cross-frame spatiotemporal information fusion and motion compensation. The cross-frame alignment submodule can fully and adaptively learn the spatiotemporal relationship between consecutive frames, complete motion compensation and align features, realize efficient fusion of cross-frame information, and improve compatibility with complex motion scenes, especially large motion scenes. The cross-frame alignment submodule is implemented in the following steps: First, a reference frame and adjacent frames are determined based on the feature sequences of the input video frames. Second, the feature sequences are grouped according to different frame rates. Considering the frame rate, i.e., the distance between adjacent frames and the reference frame in the time series, which is proportional to the motion offset of the adjacent frame relative to the reference frame and inversely proportional to the similarity between the adjacent and reference frames, adjacent frames with the same frame rate are assigned to the same group and then aligned, achieving efficient motion compensation in a decoupled manner. Third, a set of cascaded dilated convolutions and one ordinary convolution layer are used in each group to learn the offsets between adjacent and reference frames at the feature layer. Fourth, the offsets of each group and the features of adjacent frames are input into a deformable convolutional neural network to obtain aligned features. Fifth, the reference frame features and aligned features are concatenated on the channel using a concatenation operation, and then a convolutional layer is used to fuse the concatenated features to obtain the output features of the cross-frame alignment submodule. Specifically, the cross-frame alignment submodule groups the features of the input consecutive frames according to different frame rates. It uses a set of cascaded dilated convolutions and a single ordinary convolution to learn the motion offsets between adjacent frames and the reference frame. This increases the receptive field of the convolution without increasing computation, improving compatibility with complex motion scenes (especially large motion scenes). Based on the offsets learned in each group, deformable convolutions are used to align adjacent frames with the reference frame, achieving efficient and adaptive cross-frame alignment and motion compensation in a decoupled manner. Finally, a convolutional layer is used to effectively aggregate the aligned features of each group, deeply exploring the spatiotemporal correlation between adjacent frames and the reference frame, and fully extracting and fusing information from adjacent frames that is beneficial for super-resolution reconstruction of the reference frame.

[0061] In one embodiment, the cross-frame alignment submodule can fully explore the spatiotemporal relationship between adjacent frames and the reference frame, thereby achieving efficient motion compensation and obtaining better video super-resolution results. Specifically, assuming the feature sequence of the input video frames is {tn, ..., t, ..., t+n}, there are a total of 2n+1 video frame features. The features of frame t are the features of the reference frame, and the features of the remaining 2n frames are the features of adjacent frames. The frame alignment submodule first adopts a grouping strategy, that is, according to the temporal relationship (frame rate), dividing the frame into n groups. For example, the features of frames tn, t, and t+n are in one group; the features of frames t-n+1, ​​t, and t+n-1 are in another group, and so on, until finally the features of frames t-1, t, and t+1 are in the last group. The frame rates of different groups are different. For example, the frame rate in the feature group of frames tn, t, and t+n is n, meaning that the features of frames tn and t+n are n times further away from the reference frame in the time series. A higher frame rate indicates that adjacent frames are farther from the reference frame in the time series, meaning a larger motion offset relative to the reference frame, and thus a lower similarity. Within each group, a cascaded set of dilated convolutions and a single layer of regular convolutions are used to learn the offsets between adjacent and reference frames at the feature level. The dilated convolution group contains three dilated convolutions with a kernel size of 3, and dilation rates of 1, 2, and 3, respectively. A higher dilation rate indicates a larger receptive field. Compared to regular convolutions, dilated convolutions can expand the receptive field, enhancing representational ability when learning offsets and avoiding the possibility of underestimating offsets due to a small receptive field. Simultaneously, dilated convolutions do not increase computation, maintaining model efficiency. Based on this, the cross-frame submodule based on dilated convolution groups can quickly learn high-precision offsets. Then, within each group, the features of adjacent frames are aligned with the reference frame using the learned offsets and a deformable convolutional network. Note that the aforementioned ordinary convolutional layer is contained within a deformable convolutional layer, used to map the intermediate features learned by the dilated convolutional group for generating offsets into the final pixel offsets. After all groups are aligned, all group features are concatenated through a concatenation operation, and then fused with the reference frame through a simple convolutional layer.

[0062] In some preferred embodiments, downsampling methods can be used to scale video frames before alignment, thereby indirectly reducing motion, improving alignment accuracy, and enhancing the super-resolution effect for videos with large motion scenes.

[0063] Unlike standard deformable convolutional methods that use only one or a few layers of ordinary convolutions to learn the offsets of all adjacent frames at once before alignment, the cross-frame alignment submodule in this invention uses a group alignment mechanism to perform fine-grained alignment and fusion of adjacent frames at different frame rates. This decouples adjacent frames from their temporal relationships, and different alignment submodules are used for different groups, making the alignment more flexible. For example, high-frame-rate groups with greater motion tend to learn larger offsets. Based on this, this invention can continuously analyze the spatiotemporal relationships between frames, achieve efficient inter-frame alignment and motion compensation, and output aligned depth features.

[0064] The alignment method mentioned in this invention avoids the inaccuracy of single optical flow alignment, solves the limitations of traditional deformable convolution methods (insufficient alignment capability in large motion and complex scenes), and avoids the computational cost of 3D convolution. In other words, it achieves efficient and accurate cross-frame motion compensation with lower computational cost.

[0065] The embodiment includes a cross-frame alignment submodule to achieve inter-frame alignment and motion. First, based on the features of the continuous video frames input by the feature extraction submodule, a grouping strategy is used to efficiently extract the motion offset between adjacent frames and the reference frame. Then, motion compensation is performed, and finally, the aligned features are fused to achieve a decoupled inter-frame alignment.

[0066] Specifically, the input to the cross-frame alignment submodule can be consecutive video frames from any scene (assuming a total of 2n+1 frames of features are input, including features from 2n adjacent frames and features from 1 reference frame). In one specific embodiment, an efficient temporal grouping strategy is used to group the features of the input 2n+1 frames, where the features of the t-th (t = n+1)-th frame are the features of the reference frame, and the features of the remaining 2n frames are the features of adjacent frames. The frame alignment submodule first divides the features into n groups based on the time series (i.e., frame rate). For example, the features of the tn-th, t-th, and t+n-th frames are in one group, the features of the t-n+1-th, t-th, and t+n-1-th frames are in another group, and so on, until finally the features of the t-1-th, t-th, and t+1-th frames are in a third group. Assume the feature set of the input frames is: , Let the features of the i-th video frame be represented as follows:

[0067] in This indicates a grouping operation. Note that t=n+1, meaning there are 2n+1 frames of features. This refers to the characteristics of the first frame, and so on. The frame rates are different in different groups, for example... In this group, the frame rate is n. A higher frame rate means that adjacent frames are farther from the reference frame, which also represents a larger motion offset of adjacent frames in the group and a decrease in similarity with the reference frame. In each group, a set of cascaded dilated convolutions and a single layer of normal convolutions are independently used to learn the motion offsets of adjacent frames and the reference frame at the feature level. The dilated convolution group contains three dilated convolutions with a kernel size of 3, and the dilation rates are set to 1, 2, and 3, respectively. Compared with ordinary convolutions, dilated convolutions can expand the receptive field and enhance the representation ability without increasing the computational cost, which can effectively solve the problem that the learned offsets may be too small due to a small receptive field. The above-mentioned cascaded dilated convolution group is used to obtain intermediate features for generating the final offset, and a single layer of normal convolution is used to map the intermediate features into the final offset. It should be noted that in Figure 3, the single-layer convolution is integrated into the deformable convolution alignment and is not explicitly drawn. Furthermore, this invention inputs the offsets and the features of adjacent frames into the deformable convolutional neural network to obtain the final aligned features.

[0068] In this embodiment, as shown in Figure 3, it is assumed that after grouping and alignment, the features of the aligned 2n adjacent frames are output, and the features of the aligned 2n adjacent frames are assumed to be... ,in To obtain the aligned features of frame tn (i.e., the first frame), the process is repeated sequentially. Then, the aligned features are concatenated with the features of frame t (the (n+1)th frame). Specifically, a concatenation operation is used to concatenate the features along the channels. Following this, a convolutional kernel of size 1 is applied. The convolutional network layer 1 performs a depthwise fusion operation on the concatenated features. This operation can be explained by the following formula:

[0069]

[0070] in, Indicates 1 1. Convolution function, This represents the concatenation operation of features along the channel dimension. The result of fusing the features of the reference frame with the aligned features of the adjacent frames is the output of the entire cross-frame alignment submodule.

[0071] In this embodiment, the cross-frame alignment submodule integrates spatiotemporal grouping strategies, dilated convolution, and deformable convolution, achieving fine-grained alignment and motion compensation in a way that decouples motion complexity. This avoids the problem of small learning offsets caused by the small receptive field of conventional deformable convolution. The larger receptive field ensures the accuracy of motion offset learning, enabling this invention to adapt to complex motion scenarios, such as high-motion scenarios. Furthermore, since dilated convolution does not increase computational cost compared to ordinary convolution, the cross-frame alignment submodule effectively balances computational load. This submodule ultimately achieves a novel spatiotemporal alignment, efficiently modeling the motion relationship of the same pixel between the reference frame and adjacent frames, effectively extracting useful information from adjacent frames, thereby enhancing video reconstruction performance.

[0072] The feature enhancement submodule continuously enhances the fused features to improve the model's expressive power. Specifically, it extracts and enhances useful information from the fused features, such as texture and details—high-frequency information beneficial for super-resolution reconstruction—thereby improving the model's performance. The upsampling and reconstruction submodule amplifies the resolution of the enhanced features and reconstructs them into higher-resolution, higher-quality video frames. Specifically, the feature enhancement submodule includes at least three feature enhancement blocks, each consisting of a convolutional kernel of size 3. A standard convolutional layer of size 3 and a convolutional kernel of size 3 It consists of a group of 3 convolutional layers. The cascaded dilated convolution includes three cascaded dilated convolutional layers with a kernel size of 3.

[0073] In the specific implementation process, the feature enhancement submodule is used to extract the high-frequency information of the aligned and fused features to achieve further depth enhancement, fully preserving information such as object texture and details, which is then used in the subsequent upsampling reconstruction submodule to amplify the spatial resolution.

[0074] In a preferred embodiment, the feature enhancement submodule consists of 6 efficient feature enhancement blocks, each of which is composed of a convolutional kernel of size 3. A standard convolutional layer of size 3 and a convolutional kernel of size 3 The system consists of 3 group convolutional layers. These group convolutional layers employ the concept of grouped convolution, resulting in lower computational cost compared to regular convolution. Regular convolution addresses the issue of difficult interaction between channels in different groups caused by group convolution. Therefore, the efficient feature enhancement block in this invention integrates the advantages of both regular and group convolution, while effectively reducing the model's computational complexity, thus achieving a good balance between model performance and computational cost.

[0075] Leveraging a deep network architecture, the feature enhancement submodule can further refine and enhance the fused features, extracting information beneficial for super-resolution reconstruction (such as high-frequency features like details and textures). Notably, the inventors used residual techniques within the feature enhancement submodule; that is, the input features of the feature enhancement submodule are element-wise added to the outputs of six efficient feature enhancement blocks to obtain the final output of the feature enhancement submodule. Furthermore, similar residual techniques are employed within the efficient feature enhancement blocks, which will not be elaborated upon here. The application of residual techniques can effectively improve training efficiency, avoid gradient explosion problems, and enhance the model's representational capabilities. It should be noted that the number of efficient feature enhancement blocks can be adjusted according to the actual usage of the network model; for example, it can be set to any one of 4, 8, or 10.

[0076] The upsampling reconstruction submodule uses a CNN-based upsampling method to amplify the resolution of the enhanced features and restore them to a high-definition reference frame before outputting them. This enables the restoration of low-quality, low-resolution videos of different scenes into high-resolution, high-definition videos.

[0077] In the specific implementation process, the upsampling reconstruction submodule, facing the feature map output by the feature enhancement submodule, first uses a CNN-based upsampling method to amplify the resolution, that is, it uses the pixel shuffle technique (sub-pixel convolution operation) to first amplify the resolution of the feature map to achieve the upsampling function; then, it uses a single convolutional layer to reconstruct the upsampled features into high-definition, noise-free, high-resolution video frames and output them, achieving video super-resolution in complex motion scenes and ensuring the processing effect of the super-resolution method on video frames in different scenes. Finally, this invention successfully constructs a video super-resolution network model for complex motion scenes.

[0078] It should be noted that, for the sake of clarity and conciseness, the feature extraction submodule, feature enhancement submodule, and upsampling reconstruction submodule are not described in detail with diagrams and formulas in this invention. However, this invention has clearly described the technologies employed in these three submodules; therefore, those skilled in the art should be able to recognize the functions and effects of these submodules.

[0079] In S3, the video super-resolution model is trained and tested using training and testing datasets, respectively. Specifically, this includes: sequentially inputting the training datasets from various different scenarios into the video super-resolution model; calculating the loss value between the video frames generated by the model and the ground truth during iterative training; training the model with the loss value reduction as the optimization objective, and simultaneously updating and optimizing the network weights until convergence; and evaluating the reconstruction performance of the model using the testing dataset. The loss value is calculated based on the Charbonnier loss function. This implementation improves the super-resolution effect on videos in complex motion scenes, enhancing video quality and visual effects.

[0080] In the specific implementation process, a pre-built video super-resolution model based on neural networks for complex motion scenarios is initialized, and the hyperparameters of the neural network are set, including the number of neural network layers, the number of feature channels, the learning rate, and the number of iterations.

[0081] From the preprocessed training dataset, consecutive video frames from different scenes will be selected and sequentially input into the constructed deep learning-based video super-resolution network model for training. The input consecutive video frames (i.e., the training dataset) should contain more than 50 scenes, and each input should include at least 5 consecutive video frames (including the reference frame and adjacent frames).

[0082] The training process involves iteratively selecting consecutive video frames from different scenes within the training set. Each iteration selects a batch of consecutive video frames, including low-resolution, low-quality inputs and corresponding high-resolution video frames as ground truth for supervised training. The final loss value is calculated based on the ground truth, the model's output, and the loss function. This loss value is continuously optimized, and the network parameters of the video super-resolution model are constantly updated. During iteration, changes in the learning rate and the decreasing pattern of the loss value are observed to determine if parameter retuning and retraining are necessary. After training, the loss value tends to converge.

[0083] During each iteration of training, the video super-resolution network selects multiple batches of consecutive video frames from the training dataset and inputs them into the deep learning-based video super-resolution network model. The loss value of the video super-resolution network is calculated based on the loss function pre-set for the video super-resolution network.

[0084] The video super-resolution model for complex motion scenes employs the robust Charbonnier function as its loss function. Compared to L1 and L2 losses, the Charbonnier loss function is less likely to cause overly smooth images, resulting in more realistic reconstructed videos. Furthermore, compared to the commonly used L2 loss function, the proposed loss function offers better compatibility with outliers, thereby improving reconstruction performance.

[0085] To perform global optimization for super-resolution tasks, this invention requires inputting the continuous video frames to be super-resolution enhanced into the super-resolution model for complex motion scenes based on deep learning. After the model outputs the predicted video frames, the predicted video frames (i.e., the super-resolution reconstruction results of the reference frames) and the corresponding ground truth video frames (i.e., the high-resolution ground truth of the reference frames) are input into the proposed Charbonnier loss function to obtain the final global loss value.

[0086] The global loss value of the video super-resolution network is calculated. In an optional implementation of this embodiment, the loss value of the video super-resolution model under complex motion scenes is calculated, which is the final global loss value. Specifically, it is assumed that... , These are the k-th video frame after super-resolution enhancement (i.e., the result of super-resolution reconstruction) and its corresponding ground truth value, respectively, where K is the number of video frames requiring super-resolution enhancement. This is the final global loss function. The Charbonnier loss function calculation formula mentioned in this invention is as follows: in, The parameter is a fixed value, usually a small constant. This invention sets it to 1e-6, which can improve the ability to handle abnormal data, make the training process of the network model more stable, and thus enhance the performance of the model.

[0087] Based on a pre-set learning rate, this invention sets the learning rate to 1e-4, then uses the gradient descent algorithm for continuous iterative training, updating the weight parameters of each sub-module of the network model while continuously reducing the loss value until it converges. When the loss value converges, the selected network parameters are used as the final optimized model parameters to generate a video super-resolution model for complex motion scenes.

[0088] Based on a pre-defined dataset, consecutive video frames from the test dataset are selected. The video super-resolution model for complex motion scenes generated when the loss value converges is tested to verify the model's actual super-resolution reconstruction performance. After testing and verification, it can be used as the final video super-resolution model for complex motion scenes.

[0089] Another embodiment illustrates a video super-resolution system. Referring to Figure 4, the system 400 includes:

[0090] The dataset acquisition module 410 is used to acquire video datasets containing multiple different scenes and preprocess them, and divide the preprocessed video datasets into training datasets and test datasets.

[0091] The model building module 420 is used to build a video super-resolution model. The video super-resolution model includes a feature extraction submodule, a cross-frame alignment submodule, a feature enhancement submodule, and an upsampling reconstruction submodule. The feature extraction submodule extracts shallow features from consecutive input video frames. The cross-frame alignment submodule uses deformable convolution to perform feature-level alignment on different frames based on the shallow features output by the feature extraction submodule, achieving cross-frame spatiotemporal information fusion and motion compensation. The feature enhancement submodule enhances the spatiotemporal information fused features output by the cross-frame alignment submodule. The upsampling reconstruction submodule amplifies the resolution of the enhanced features output by the feature enhancement submodule and reconstructs the amplified enhanced features into video frames.

[0092] The model training and testing module 430 is used to train and test the video super-resolution model using the training dataset and the test dataset, respectively.

[0093] The video super-resolution implementation module 440 is used to implement video super-resolution using the video super-resolution model that has been trained and tested.

[0094] In addition to the modules described above, the system 400 may also include other components; however, since these components are not relevant to the embodiments of this disclosure, their illustrations and descriptions are omitted here.

[0095] Other specific working processes of the video super-resolution system 400 are described in the above-described video super-resolution method embodiments and will not be repeated here.

[0096] Another embodiment illustrates that the system of the present invention can also be implemented using the architecture of the computing device shown in FIG. 5. FIG. 5 shows the architecture of the computing device. As shown in FIG. 5, the system includes a computer system 510, a system bus 530, one or more CPUs 540, input / output 520, and a memory 550. The memory 550 can store various data or files used for computer processing and / or communication, as well as program instructions executed by the CPU, including the video super-resolution method of the embodiment. The architecture shown in FIG. 5 is merely exemplary; when implementing different devices, one or more components in FIG. 5 may be adjusted according to actual needs. The memory 550, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the video super-resolution method in the embodiments of the present invention (e.g., the dataset acquisition module 410, model building module 420, model training and testing module 430, and video super-resolution implementation module 440 in the video super-resolution system 400). One or more CPUs 540 execute various functional applications and data processing of the system of the present invention by running the software programs, instructions, and modules stored in the memory 550, i.e., implementing the above-described video super-resolution method, which includes the following steps:

[0097] A video dataset containing multiple different scenes is acquired and preprocessed. The preprocessed video dataset is then divided into a training dataset and a test dataset.

[0098] A video super-resolution model is established, comprising a feature extraction submodule, a cross-frame alignment submodule, a feature enhancement submodule, and an upsampling reconstruction submodule. The feature extraction submodule extracts shallow features from consecutive input video frames. The cross-frame alignment submodule uses deformable convolution to perform feature-level alignment on different frames based on the shallow features output by the feature extraction submodule, achieving spatiotemporal information fusion and motion compensation across frames. The feature enhancement submodule enhances the spatiotemporal fusion features output by the cross-frame alignment submodule. The upsampling reconstruction submodule amplifies the resolution of the enhanced features output by the feature enhancement submodule and reconstructs the amplified enhanced features into video frames.

[0099] The video super-resolution model is trained and tested using the training dataset and the test dataset, respectively.

[0100] Video super-resolution is achieved using the video super-resolution model that has been trained and tested.

[0101] Of course, the processor of the server provided in the embodiments of the present invention is not limited to performing the method operations described above, but can also perform related operations in the video super-resolution method provided in any embodiment of the present invention.

[0102] The memory 550 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 550 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory 550 may further include memory remotely configured relative to one or more CPUs 540, which can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0103] Input / output 520 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Input / output 520 may also include a display device such as a display screen.

[0104] This invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, this computer program implements the video super-resolution method described in the above embodiments. The computer-readable storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0105] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0106] The program code contained on the storage medium can be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0107] Furthermore, other specific operational processes of a non-transitory computer-readable storage medium are described in the above-described embodiments of the video super-resolution method, and will not be repeated here.

[0108] Based on the technical solutions provided in the above embodiments, a video super-resolution method, system, electronic device, and storage medium are disclosed. The method includes establishing a video super-resolution model. A feature extraction submodule extracts features from consecutive input video frames using one or more cascaded convolutional layers, laying the foundation for efficient motion compensation at the feature level. A cross-frame alignment submodule performs fine-grained alignment and fusion of adjacent frames at different frame rates through a group alignment mechanism, decoupling adjacent frames from a temporal sequence perspective. Different alignment submodules are used for different groups, making the alignment more flexible. For example, the high-frame-rate group with higher motion intensity learns the bias... The offset is often large. Based on this, this invention can continuously determine the spatiotemporal relationship between frames to achieve efficient inter-frame alignment and motion compensation, and output the aligned depth features. In addition, the cross-frame sub-module based on dilated convolution groups can quickly learn high-precision offsets. The alignment sub-module method avoids the inaccuracy of single optical flow alignment and solves the limitations of traditional deformable convolution methods (insufficient alignment capability in large motion and complex scenes), while also avoiding the computational cost of 3D convolution. That is, it achieves efficient and accurate cross-frame motion compensation with low computational cost. The feature enhancement sub-module consists of 6 efficient feature enhancement blocks, each of which is composed of a convolution kernel of size 3. A standard convolutional layer of size 3 and a convolutional kernel of size 3 The model consists of 3 group convolutional layers. The group convolutional layer adopts the idea of ​​group convolution, which has a lower computational cost compared to ordinary convolution. Ordinary convolution is used to solve the problem of difficult interaction between channels of different groups caused by group convolution. The efficient feature enhancement block in this invention integrates the advantages of ordinary convolution and group convolution, while effectively reducing the computational complexity of the model, which helps to achieve an effective balance between model performance and computational cost.

[0109] In summary, the video super-resolution method, system, electronic device, and storage medium proposed in this invention for complex motion scenes intelligently divide the input continuous video frames temporally through a grouping alignment mechanism, improving compatibility with complex motion scenes (especially large motion scenes). Secondly, by rapidly processing the features of different video frames, efficient spatiotemporal information fusion is achieved, enhancing information extraction capabilities. Furthermore, through a novel feature enhancement mechanism, the details and textures of the video frames are fully preserved, which is beneficial to the final reconstruction result and results in better visual effects. Thanks to these advantages, the video super-resolution method, system, electronic device, and storage medium of this invention can achieve high super-resolution performance with a relatively low number of parameters, and can be widely promoted and applied in practical industrial scenarios.

[0110] In this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a step or method that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such a step or method.

[0111] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. A video super-resolution method, characterized in that, The method includes the following steps: A video dataset containing multiple different scenes is acquired and preprocessed. The preprocessed video dataset is then divided into a training dataset and a test dataset. A video super-resolution model is established, comprising a feature extraction submodule, a cross-frame alignment submodule, a feature enhancement submodule, and an upsampling reconstruction submodule. The feature extraction submodule extracts shallow features from consecutive input video frames. The cross-frame alignment submodule uses deformable convolution to perform feature-level alignment on different frames based on the shallow features output by the feature extraction submodule, achieving spatiotemporal information fusion and motion compensation across frames. The feature enhancement submodule enhances the spatiotemporal fusion features output by the cross-frame alignment submodule. The upsampling reconstruction submodule amplifies the resolution of the enhanced features output by the feature enhancement submodule and reconstructs the amplified enhanced features into video frames. The video super-resolution model is trained and tested using the training dataset and the test dataset, respectively. Video super-resolution is achieved using the video super-resolution model that has been trained and tested.

2. The video super-resolution method according to claim 1, characterized in that, The cross-frame alignment submodule performs motion compensation based on different frame rate groups, including: aligning the features of adjacent frames and reference frames in a decoupled manner based on adjacent frames containing different time series in different groups, fusing the aligned adjacent frames with the reference frame through a convolutional layer, and inputting the fused features into the feature enhancement submodule.

3. The video super-resolution method according to claim 2, characterized in that, The cross-frame alignment submodule is specifically implemented by the following steps: Reference frames and adjacent frames are determined based on the feature sequences of the input video frames; Based on the fact that the motion offset of adjacent frames relative to the reference frame is directly proportional to the frame rate and inversely proportional to the similarity between adjacent frames and the reference frame, the feature sequences are grouped according to different frame rates, and adjacent frames with different frame rates are aligned by decoupling. In each group, a set of cascaded dilated convolutions and one layer of ordinary convolutions are used to effectively learn the motion offset between adjacent frames and reference frames at the feature level. The offset of each group and the features of adjacent frames are input into a deformable convolutional neural network to obtain the aligned features. The Concatenation operation is used to concatenate the reference frame features and the aligned features on the channel, and then a convolutional layer is used to perform feature fusion to obtain the output features of the cross-frame alignment submodule.

4. The video super-resolution method according to claim 1, characterized in that, The video super-resolution model is trained and tested using the training dataset and the test dataset, respectively, specifically including: The training datasets from various different scenarios are sequentially input into the video super-resolution model; During iterative training, the loss value between the video frames generated by the video super-resolution model and the true values ​​is calculated; With the goal of reducing the loss value, the video super-resolution model is trained, and the network model weights are updated and optimized simultaneously until convergence. The reconstruction performance of the video super-resolution model was evaluated using the test dataset.

5. The video super-resolution method according to claim 4, characterized in that, The loss value is calculated based on the Charbonnier loss function.

6. The video super-resolution method according to claim 1, characterized in that, The feature enhancement submodule includes no fewer than three feature enhancement blocks, each consisting of a convolutional kernel of size 3. A standard convolutional layer of size 3 and a convolutional kernel of size 3 It consists of 3 grouped convolutional layers.

7. The video super-resolution method according to claim 3, characterized in that, The set of cascaded dilated convolutions includes three cascaded dilated convolutions with a kernel size of 3.

8. A video super-resolution system, characterized in that, The system includes: The dataset acquisition module is used to acquire video datasets containing multiple different scenes and preprocess them, dividing the preprocessed video datasets into training datasets and test datasets. A model building module is used to build a video super-resolution model. The video super-resolution model includes a feature extraction submodule, a cross-frame alignment submodule, a feature enhancement submodule, and an upsampling reconstruction submodule. The feature extraction submodule extracts shallow features from consecutive input video frames. The cross-frame alignment submodule uses deformable convolution to perform feature-level alignment on different frames based on the shallow features output by the feature extraction submodule, achieving cross-frame spatiotemporal information fusion and motion compensation. The feature enhancement submodule enhances the spatiotemporal fusion features output by the cross-frame alignment submodule. The upsampling reconstruction submodule amplifies the resolution of the enhanced features output by the feature enhancement submodule and reconstructs the amplified enhanced features into video frames. The model training and testing module is used to train and test the video super-resolution model using the training dataset and the test dataset, respectively. A video super-resolution implementation module is used to implement video super-resolution using the video super-resolution model that has been trained and tested.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the video super-resolution method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium, wherein computer instructions are stored on the storage medium, characterized in that, When the instructions are executed by the processor, they implement the steps of the video super-resolution method as described in any one of claims 1 to 7.

Images