Interpolation method and apparatus for video, and electronic device and medium
The Transformer module, with its bidirectional gated loop unit and hierarchical multi-scale window mechanism, addresses the performance limitations of existing video interpolation models on small sample data, achieving efficient and accurate video interpolation results suitable for scenarios such as autonomous driving and game rendering.
Patent Information
- Application Number
- PCT/CN2025/098914
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-09
- Filing Date
- 2025-06-04
- Publication Date
- 2026-11-12
AI Technical Summary
Existing video interpolation models have weak generalization performance when training data is insufficient. The receptive field of traditional convolution operations is limited, which restricts the performance of video interpolation. Furthermore, there is a lack of efficient methods for generating high-quality frames with small sample data.
The Transformer module employs a bidirectional gated recurrent unit combined with a hierarchical multi-scale window mechanism. It models the historical and future information of image frames through bidirectional information flow, and uses the multi-scale spatiotemporal feature gated recurrent unit to extract local details and global semantic features on small sample data to construct a video interpolation model.
It improves the accuracy and robustness of video interpolation with small sample data, and can efficiently generate high-quality interpolated frames, which are suitable for scenarios such as autonomous driving and game rendering.
Smart Images

Figure CN2025098914_12112026_PF_FP_ABST
Abstract
Description
A video interpolation method, apparatus, electronic device and medium Technical Field
[0001] This invention relates to the field of computer vision technology, and more specifically, to a video interpolation method, apparatus, electronic device, and medium. Background Technology
[0002] Video interpolation aims to restore the integrity and smoothness of a video by repairing or generating missing or corrupted frames using algorithms. This technology is of great significance in the field of video processing. For example, the restoration of old movies and low-quality videos can significantly improve the overall quality and viewing experience; video interpolation technology in video surveillance systems can fill in dropped frames caused by network bandwidth limitations or equipment failures, ensuring the integrity and continuity of surveillance video; in the field of medical imaging, video interpolation can help generate more intermediate frames, improve the temporal resolution of dynamic images, and enable doctors to observe changes in lesion areas more accurately, improving diagnostic accuracy, which is particularly important in dynamic medical imaging such as ultrasound scanning.
[0003] Currently, mainstream interpolation models are primarily based on recurrent neural network (RNN) architectures. Due to the temporal and spatial complexity of video data, these models need to capture the dependencies between temporal and spatial dimensions simultaneously. In recent years, researchers have proposed various hybrid models combining convolutional neural networks (CNNs) and RNNs, such as ConvLSTM and E3D-LSTM, to better model the spatiotemporal features in videos. However, traditional convolutional operations mainly focus on local spatial dependencies. Although the receptive field can be expanded by stacking convolutional layers or increasing the kernel size, the actual effective receptive field is often much smaller than the theoretical value, which limits the video interpolation performance of the model to some extent. To address this issue, Video Visual Transformer (ViViT) has been proposed, which models global spatiotemporal information through self-attention mechanisms, temporal encoding, and positional encoding. However, because the Transformer architecture lacks the inductive bias inherent in CNNs, its generalization performance is weak, typically requiring large-scale training data to achieve ideal results. Therefore, there is a need to find a video frame interpolation method that requires less training data, is fast, and generates high-quality images. Summary of the Invention
[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a video interpolation method, apparatus, electronic device, and medium.
[0005] The objective of this invention is achieved through the following technical solution: a video interpolation method, comprising:
[0006] Obtain the video to be interpolated and its missing segments, and construct an image dataset using the video to be interpolated and its missing segments as a set of data.
[0007] Obtain the position information of the first frame of the missing segment in the video to be interpolated, and locate the interpolation range of the missing segment;
[0008] The image dataset is used to train the interpolation frame model. During the model training process, the video to be interpolated is interpolated based on the position information of the first frame of the missing segment in the video to be interpolated and the interpolation range of the missing segment.
[0009] The trained interpolation frame model is used for video interpolation.
[0010] Furthermore, an image dataset is constructed using the video to be interpolated and its missing segments as a set of data, including:
[0011] Maintain a one-to-one correspondence between the video to be interpolated and its missing segments;
[0012] The paired video data is decomposed frame by frame according to the time series, converted into a continuous sequence of image frames, and saved as an image file;
[0013] The image is denoised, then cropped to the same resolution, and the image data is enhanced.
[0014] Furthermore, the position information of the first frame of the missing segment in the video to be interpolated is obtained, and the interpolation range of the missing segment is located, including:
[0015] Obtain the first frame image file of the missing segment and use it as the reference frame; by calculating the structural similarity index between the reference frame and each frame in the video image frame sequence to be interpolated, determine the frame with the highest similarity, and use the next frame of the reference frame as the interpolation starting point, that is, the position of the first frame of the missing segment in the video to be interpolated.
[0016] The interpolation range of the missing segment is determined based on the interpolation start point and the length of the missing segment.
[0017] Furthermore, the interpolation frame model is a bidirectional gated loop unit.
[0018] The bidirectional gated loop unit includes a forward gated loop unit, a backward gated loop unit, and a dual-stream feature fusion layer; the dual-stream feature fusion layer uses a summation and averaging operation to fuse the image features of the bidirectional information streams;
[0019] Both the forward-gated recurrent unit and the backward-gated recurrent unit include image block encoding, a first multi-scale spatiotemporal feature gated recurrent unit, image block fusion, a second multi-scale spatiotemporal feature gated recurrent unit, image block expansion, and an image reconstruction layer, which are connected in sequence.
[0020] The image block encoding, image block fusion and image block expansion are respectively used for image vectorization processing, feature dimension reduction aggregation and feature up-sampling; the image reconstruction layer adopts two-dimensional deconvolution operation to restore the spatial resolution and detail information of the image;
[0021] The first multi-scale spatio-temporal feature gated recurrent unit and the second multi-scale spatio-temporal feature gated recurrent unit each comprise a gated recurrent unit.
[0022] Further, the gated recurrent unit is nested with a linear projection module and at least one Transformer module comprising a hierarchical multi-scale window mechanism; an input of the gated recurrent unit is spliced with a hidden state at a previous time to serve as an input of the linear projection module, and an output of the linear projection module serves as an input of the Transformer module comprising the hierarchical multi-scale window mechanism; an output of the Transformer module comprising the hierarchical multi-scale window mechanism is subjected to a key equation to become an output of the gated recurrent unit, that is, an output of the multi-scale spatio-temporal feature gated recurrent unit; the key equation is:
[0023] Update gate Z t =σ(W xz X t +W hz H t-1 +b z )
[0024] Reset gate R t =σ(W xR X t +W hR H t-1 +b R )
[0025] Candidate hidden state
[0026] Hidden state
[0027] Wherein, σ represents a Sigmoid activation function; X t represents an image frame input at a current time; H t-1 represents a hidden state at a previous time; tanh is a hyperbolic tangent function; W xz , W hz , W xR , W hR , W xh and W rh are weight parameters of the model; b z , b R and b h are biases of the model.
[0028] Further, if the gated recurrent unit is nested with multiple Transformer modules comprising the hierarchical multi-scale window mechanism, the Transformer modules comprising the hierarchical multi-scale window mechanism are connected in series.
[0029] Further, the method further comprises: only performing the forward gated recurrent unit to predict the next image frame.
[0030] The application further provides a video interpolation device based on a bidirectional gated recurrent unit, comprising:
[0031] A data set construction module is configured to obtain a to-be-interpolated video and a missing segment thereof to construct an image data set by taking the to-be-interpolated video and the missing segment thereof as a group of data.
[0032] An interpolation position and range acquisition module is configured to acquire position information of a first frame of the missing segment in the to-be-interpolated video and locate an interpolation range of the missing segment.
[0033] A model training module is configured to train an interpolation frame model by using the image data set, and interpolate the to-be-interpolated video based on the position information of the first frame of the missing segment in the to-be-interpolated video and the interpolation range of the missing segment during the model training process.
[0034] An interpolation module is configured to input the to-be-processed missing video into the trained interpolation frame model to perform interpolation and output a complete video.
[0035] The application further provides an electronic device comprising a memory and a processor, wherein the memory is coupled to the processor, the memory is configured to store program data, and the processor is configured to execute the program data to implement the above-mentioned video interpolation method based on a bidirectional gated recurrent unit.
[0036] The application further provides a computer-readable storage medium having a computer program stored thereon, wherein the program is executed by a processor to implement the above-mentioned video interpolation method based on a bidirectional gated recurrent unit.
[0037] Compared with the prior art, the application has the following beneficial effects:
[0038] 1. The bidirectional structure fully captures the time dependence in the video sequence by simultaneously modeling the historical and future information of the image frame, thereby improving the accuracy and continuity of the interpolated frame.
[0039] 2. The multi-scale spatio-temporal feature gated recurrent unit combined with the Transformer of the gating cycle unit and the hierarchical multi-scale window mechanism can efficiently extract local details and global semantic features of an image under a small sample data set, thereby realizing high-quality interpolation of a missing video segment.
[0040] 3. The method can only perform forward gated recurrent unit prediction of a next image frame, and is suitable for understanding a dynamic environment around an automatic driving system, reducing game rendering delay and the like. BRIEF DESCRIPTION OF DRAWINGS
[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.
[0042] Fig. 1 is a flowchart of a video interpolation method based on a bidirectional gated recurrent unit according to an embodiment of the present application.
[0043] Fig. 2 is a structure diagram of a bidirectional gated recurrent unit according to an embodiment of the present application.
[0044] Fig. 3 is a structure diagram of a forward / backward gated recurrent unit according to an embodiment of the present application.
[0045] Fig. 4 is a structure diagram of a multi-scale spatio-temporal feature gated recurrent unit according to an embodiment of the present application, wherein the activation function a is a Sigmoid activation function, and the activation function b is a hyperbolic tangent function.
[0046] Fig. 5 is a structure diagram of an electronic device corresponding to Fig. 1 according to the present application. DETAILED DESCRIPTION
[0047] The present application will be described in detail below with reference to the drawings. The features in the following embodiments and implementation manners can be combined with each other without conflict.
[0048] It should be clear that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments in the present application without creative labor are within the scope of protection of the present application.
[0049] The terminology used in the embodiments of the present application is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used in the description of the embodiments of the present application and the appended claims, the singular forms "a," "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
[0050] FIG. 1 is a flowchart of a video interpolation method based on a bidirectional gated recurrent unit according to an embodiment of the present application, comprising the following steps:
[0051] S1, establishing an image dataset;
[0052] Specifically, the to-be-interpolated video and the missing segment thereof need to be strictly paired to ensure the accuracy and integrity of the data. The paired video data is sequentially decomposed frame by frame, converted into a continuous image frame sequence, and saved as an image file. A Gaussian filter is used to reduce the noise of the image, and then the high-quality image is cropped to the same resolution, and the image data is enhanced using a common data enhancement method in deep learning.
[0053] S2, locating the interpolation range of the missing segment by obtaining the position information of the first frame of the missing segment in the to-be-interpolated video;
[0054] Specifically, the first frame image file of the missing segment is obtained and used as a reference frame. By calculating the structural similarity index (SSIM) of the reference frame and each frame in the to-be-interpolated video image frame sequence, the frame with the highest similarity is located. The next frame of the frame is used as the starting point of interpolation, and the subsequent frame sequence is the interpolation range of the missing segment, and the interpolation length is consistent with the image frame sequence length of the missing segment.
[0055] S3, constructing a bidirectional gated recurrent unit;
[0056] As shown in FIG. 2, the bidirectional gated recurrent unit comprises a forward gated recurrent unit, a backward gated recurrent unit, and a double-flow feature fusion layer.
[0057] FIG. 3 is a structural diagram of the forward / backward gated recurrent unit, comprising the following structures: image block encoding, two multi-scale spatiotemporal feature gated recurrent units, image block fusion, image block expansion, and an image reconstruction layer. One multi-scale spatiotemporal feature gated recurrent unit is arranged between the image block encoding and the image block fusion, and the other is arranged between the image block fusion and the image block expansion.
[0058] Image patch encoding, image patch fusion, and image patch expansion are core technologies in the general Transformer architecture used for image vectorization, feature dimensionality reduction and aggregation, and feature upsampling, respectively. The image reconstruction layer uses two-dimensional deconvolution operations to restore the spatial resolution and detail information of the image.
[0059] The dual-stream feature fusion layer uses an additive operation to fuse image features from both bidirectional information streams.
[0060] As shown in Figure 4, the multi-scale spatiotemporal feature gated loop unit includes: a gated loop unit, a linear projection module, and a Transformer containing a hierarchical multi-scale window mechanism.
[0061] The multi-scale spatiotemporal feature gated recurrent unit is an innovative architecture improved on the gated recurrent unit. That is, the gated recurrent unit embeds a linear projection module and a Transformer containing a hierarchical multi-scale window mechanism. One or more Transformers containing the hierarchical multi-scale window mechanism can be embedded. If multiple are embedded, they are connected in series. That is, the output of the first Transformer containing the hierarchical multi-scale window mechanism is the input of the second Transformer containing the hierarchical multi-scale window mechanism, the output of the second Transformer containing the hierarchical multi-scale window mechanism is the input of the third Transformer containing the hierarchical multi-scale window mechanism, and so on. The hierarchical multi-scale window mechanism can capture local detailed features and global semantic information at the same time, thereby significantly enhancing the model's ability to model long-distance dependencies. In the calculation process of each time step, the gated recurrent unit mainly consists of the following four core components: (1) Update gate Z t , used to control the degree of update of the hidden state at the current moment, and to decide whether to incorporate new information into the hidden state; (2) Reset gate R t (3) Candidate hidden state H′ t Based on the current input and the result of the reset gate, new memory content is generated as a potential state update candidate; (4) Hidden state H t By updating the hidden state and candidate hidden states from the previous time step through a weighted fusion, the final state output is determined, thereby achieving dynamic updating and preservation of information. As shown in Figure 3, the key equation of the multi-scale spatiotemporal feature-gated recurrent unit is: Z t =σ(W xz X t +W hz H t-1 +b z R t =σ(W xR Xt +W hR H t-1 +b R )
[0062] Where σ represents the Sigmoid activation function; X t H represents the current image frame input; t-1 Represents the hidden state at the previous time step; tanh is the hyperbolic tangent function, compressing the result to [-1, 1]; W xz W hz W xR W hR W xh and W rh All of these are the model's weight parameters; b z b R and b h All of these are biases of the model; This is the Hadamard product operator.
[0063] The hierarchical multi-scale window mechanism decomposes the original global self-attention mechanism into the computation of multiple local windows of different scales, and performs feature extraction at different resolutions through a hierarchical processing method.
[0064] S4. The model evaluation uses mean squared error (MSE), peak signal-to-noise ratio (PSNR), and structural similarity index (SSIM) as evaluation metrics to comprehensively measure the quality of the generated image frames and their similarity to the missing segments.
[0065] Example 1: A method for imputing missing fragments in ovarian cancer ultrasound images;
[0066] S1, Create an image dataset;
[0067] Specifically, a high-resolution ultrasound device suitable for abdominal or pelvic scanning is selected to scan the lesion sites of ovarian cancer patients. Ultrasound images are obtained by receiving and processing echoes carrying characteristic information of the lesion sites. Radiologists annotate video segments containing tumor sites. The acquired ultrasound and tumor site video segment data are in DICOM format. The acquired ultrasound images are the videos to be interpolated, and the video segments of the tumor sites are the missing segments, maintaining a one-to-one correspondence between the videos to be interpolated and their missing segments. The videos to be interpolated and their missing segments are decomposed frame-by-frame according to time series, converted into image frame sequences, and denoised using a Gaussian filter. After cropping to a resolution of 320*240, the data is augmented using common deep learning data augmentation methods such as rotation, flipping, and scaling. In this example, the ovarian cancer ultrasound image dataset contains 68 ultrasound examination results, and each video to be interpolated consists of 50 frames, with the labeled missing segments being 10 frames long. To ensure the effectiveness of model training and evaluation, the dataset is divided into training, validation, and test sets in a 6:2:2 ratio, which are used for model training, hyperparameter tuning, and performance evaluation, respectively.
[0068] S2, by obtaining the position information of the first frame of the missing segment in the video to be interpolated, the interpolation range of the missing segment is located;
[0069] Specifically, for each set of ultrasound data, the first frame of the missing segment is extracted as the reference frame. By calculating the structural similarity index between the reference frame and each frame in the video image frame sequence to be interpolated, the frame with the highest similarity to the reference frame is located, and the index position of its next frame is recorded. During model training, this index is used as the starting point for interpolation of the missing segment, and the missing content is predicted and generated according to a fixed interpolation length (10 frames).
[0070] S3, construct a bidirectional gated loop unit;
[0071] Specifically, each forward-gated recurrent unit and backward-gated recurrent unit is connected to two multi-scale spatiotemporal feature-gated recurrent units before and after the image patch fusion operation, respectively, to enhance the model's ability to extract local and global features.
[0072] Each multi-scale spatiotemporal feature-gated recurrent unit integrates two sets of Transformer modules containing a hierarchical multi-scale window mechanism (in this embodiment, two sets of Transformer modules containing a hierarchical multi-scale window mechanism are integrated; the number of these modules is not limited, and one, three, or more sets can be integrated). Therefore, each bidirectional gated recurrent unit contains a total of eight sets of Transformer modules with a hierarchical multi-scale window mechanism, which are embedded into the forward and backward gated recurrent units respectively. During model training, the batch size is set to 2, the Adam optimizer is used, the initial learning rate is set to 0.0001, and the mean squared error loss function is selected as the optimization objective to minimize the pixel-level difference between the generated interpolated frames and the real frames.
[0073] S4. The model evaluation uses mean squared error (MSE), peak signal-to-noise ratio (PSNR), and structural similarity index (SSIM) as evaluation metrics, as detailed in Table 1.
[0074] Table 1: Evaluation Indicators
[0075] In this embodiment 1, when the model's performance on both the training and validation sets reaches convergence, the complete network model structure and its corresponding hyperparameter configuration are saved. In practical applications, when calling this neural network algorithm, the saved model file can be loaded, and predictions can be made directly on the input data based on the stored hyperparameters, thereby ensuring model consistency and the reliability of prediction results.
[0076] This specification also provides a video interpolation device based on a bidirectional gated loop unit, comprising:
[0077] The dataset construction module is used to obtain the video to be interpolated and its missing segments, and to construct an image dataset using the video to be interpolated and its missing segments as a set of data.
[0078] The interpolation position and range acquisition module is used to obtain the position information of the first frame of the missing segment in the video to be interpolated, and to locate the interpolation range of the missing segment;
[0079] The model training module is used to train the interpolation frame model using the image dataset. During the model training process, the video to be interpolated is interpolated based on the position information of the first frame of the missing segment in the video to be interpolated and the interpolation range of the missing segment.
[0080] The interpolation module is used to input the missing video to be processed into the trained interpolation frame model for interpolation and output the complete video.
[0081] It should be noted that the device embodiment shown in this embodiment matches the content of the above method embodiment, and the content of the above method embodiment can be referred to, and will not be repeated here.
[0082] This specification also provides a schematic structural diagram of an electronic device corresponding to Figure 1, as shown in Figure 5. As shown in Figure 5, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, memory, and non-volatile memory, and may also include other hardware required for other services. The processor reads the corresponding computer program from the non-volatile memory into memory and then runs it to implement the video interpolation method described in Figure 1.
[0083] This specification also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described video interpolation method based on a bidirectional gated loop unit.
[0084] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0085] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more flowchart illustrations and / or one or more block diagrams.
[0086] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.
[0087] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.
[0088] The above embodiments are only used to illustrate the design concept and features of the present invention, and their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. The protection scope of the present invention is not limited to the above embodiments. Therefore, all equivalent changes or modifications made based on the principles and design ideas disclosed in the present invention are within the protection scope of the present invention.
Claims
1. A video interpolation method based on a bidirectional gated loop unit, characterized in that, include: Obtain the video to be interpolated and its missing segments, and construct an image dataset using the video to be interpolated and its missing segments as a set of data. Obtain the position information of the first frame of the missing segment in the video to be interpolated, and locate the interpolation range of the missing segment; The image dataset is used to train the interpolation frame model. During the model training process, the video to be interpolated is interpolated based on the position information of the first frame of the missing segment in the video to be interpolated and the interpolation range of the missing segment. The trained interpolation frame model is used for video interpolation.
2. The video interpolation method based on a bidirectional gated loop unit according to claim 1, characterized in that, An image dataset is constructed using the video to be interpolated and its missing segments as a set of data, including: Maintain a one-to-one correspondence between the video to be interpolated and its missing segments; The paired video data is decomposed frame by frame according to the time series, converted into a continuous sequence of image frames, and saved as an image file; The image is denoised, then cropped to the same resolution, and the image data is enhanced.
3. The video interpolation method based on a bidirectional gated loop unit according to claim 1, characterized in that, Obtain the position information of the first frame of the missing segment in the video to be interpolated, and locate the interpolation range of the missing segment, including: Obtain the first frame image file of the missing segment and use it as the reference frame; by calculating the structural similarity index between the reference frame and each frame in the video image frame sequence to be interpolated, determine the frame with the highest similarity, and use the next frame of the reference frame as the interpolation starting point, that is, the position of the first frame of the missing segment in the video to be interpolated. The interpolation range of the missing segment is determined based on the interpolation start point and the length of the missing segment.
4. The video interpolation method based on a bidirectional gated loop unit according to claim 1, characterized in that, The interpolation frame model is a bidirectional gated loop unit; The bidirectional gated loop unit includes a forward gated loop unit, a backward gated loop unit, and a dual-stream feature fusion layer; the dual-stream feature fusion layer uses a summation and averaging operation to fuse the image features of the bidirectional information streams; Both the forward-gated recurrent unit and the backward-gated recurrent unit include image block encoding, a first multi-scale spatiotemporal feature gated recurrent unit, image block fusion, a second multi-scale spatiotemporal feature gated recurrent unit, image block expansion, and an image reconstruction layer, which are connected in sequence. The image block encoding, image block fusion, and image block expansion are used for image vectorization processing, feature dimensionality reduction aggregation, and feature upsampling, respectively; the image reconstruction layer uses two-dimensional deconvolution operation to restore the spatial resolution and detail information of the image. Both the first multi-scale spatiotemporal feature gated loop unit and the second multi-scale spatiotemporal feature gated loop unit include a gated loop unit.
5. The video interpolation method based on a bidirectional gated loop unit according to claim 4, characterized in that, The gated recurrent unit nests a linear projection module and at least one Transformer module containing a hierarchical multi-scale window mechanism. The input of the gated recurrent unit is concatenated with the hidden state from the previous time step as the input of the linear projection module, and the output of the linear projection module is used as the input of the Transformer module containing the hierarchical multi-scale window mechanism. The output of the Transformer module containing the hierarchical multi-scale window mechanism, after passing through a key equation, becomes the output of the gated recurrent unit. The key equation is: Update gate Z... t =σ(W xz X t +W hz H t-1 +b z Reset door R t =σ(W xR X t +W hR H t-1 +b R Candidate hidden state Hidden state Where σ represents the Sigmoid activation function; X t H represents the current image frame input; t-1 Represents the hidden state at the previous time step; tanh is the hyperbolic tangent function; W xz W hz W xR W hR W xh and W rh All of these are the model's weight parameters; b z b R and b h These are all biases of the model.
6. The video interpolation method based on a bidirectional gated loop unit according to claim 5, characterized in that, If a gated loop unit nests multiple Transformer modules that contain a hierarchical multi-scale window mechanism, the Transformer modules containing the hierarchical multi-scale window mechanism are connected in series.
7. A video interpolation method based on a bidirectional gated loop unit according to any one of claims 4-6, characterized in that, Also includes: Only the forward-gated loop unit is executed to predict the next image frame.
8. A video interpolation device based on a bidirectional gated loop unit, characterized in that, include: The dataset construction module is used to obtain the video to be interpolated and its missing segments, and to construct an image dataset using the video to be interpolated and its missing segments as a set of data. The interpolation position and range acquisition module is used to obtain the position information of the first frame of the missing segment in the video to be interpolated, and to locate the interpolation range of the missing segment; The model training module is used to train the interpolation frame model using the image dataset. During the model training process, the video to be interpolated is interpolated based on the position information of the first frame of the missing segment in the video to be interpolated and the interpolation range of the missing segment. The interpolation module is used to input the missing video to be processed into the trained interpolation frame model for interpolation and output the complete video.
9. An electronic device comprising a memory and a processor, characterized in that, The memory is coupled to the processor; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the video interpolation method based on a bidirectional gated loop unit as described in any one of claims 1-7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements a video interpolation method based on a bidirectional gated loop unit as described in any one of claims 1-7.
