Video frame interpolation method and apparatus based on optical flow, and electronic device and storage medium

Abstract

Provided in the embodiments of the present disclosure are a video frame interpolation method and apparatus based on an optical flow, and a device, a medium and a program product. In the method, by using as reference frames a plurality of forward video frames for a video frame to be subjected to prediction, a forward optical-flow estimation module captures an optical flow warping feature of the reference frames by means of optical flow estimation, and by means of a plurality of rounds of cyclic processing, a real value of said video frame is gradually approached, such that a target predicted video frame is finally generated. In the method, the real value of the video frame to be subjected to prediction is gradually approached by means of a multi-round cyclic optical flow estimation method, thereby improving the accuracy of prediction and estimation. In addition, only forward video frames are used as reference frames, such that the impact of introducing backward reference frames on the coding efficiency is avoided, thereby meeting the requirements of a low-delay time-domain coding mode.

Description

Video frame interpolation method and device based on optical flow, electronic device and storage medium

[0001] Cross-reference to Related Applications

[0002] The present disclosure claims priority to Chinese Patent Application No. 202411537885.9, filed October 30, 2024, entitled “Video encoding method and device based on optical flow, electronic device and storage medium,” the entire contents of which are incorporated herein by reference in their entirety. TECHNICAL FIELD

[0003] The present disclosure relates to the technical field of video coding, and in particular, to a video frame interpolation method and device based on optical flow, an electronic device, a storage medium, and a computer program product. BACKGROUND

[0004] The Video Coding for Machines (VCM) working group was established by the video coding standard organization to study video coding for machine processing. At present, the temporal frame interpolation network adopted by the working group is a method of predicting and generating intermediate frames based on forward and backward reference frames. This method performs well in frame interpolation effect and can effectively improve the frame rate and smoothness of the video, and is suitable for various video processing scenarios. However, this two-way reference frame method has certain limitations, especially in the Low Delay (LD) mode, as it needs to rely on future frames as reference, which cannot meet the requirements of real-time application scenarios. SUMMARY

[0005] The present disclosure provides a video frame interpolation method and device based on optical flow, an electronic device, a storage medium, and a computer program product.

[0006] Other characteristics and advantages of the present disclosure will become apparent from the following detailed description, or will be learned by practice of the present disclosure.

[0007] According to one aspect of this disclosure, a video frame interpolation method based on optical flow is provided. The method includes: stitching together a first video frame, a second video frame, a first modified frame obtained in the (i-1)th round, a second modified frame, and a mask to obtain a first feature; wherein the first modified frame and the second modified frame are obtained by modifying the first video frame and the second video frame, respectively; and based on the scaling factor in the i-th round, performing optical flow estimation on the optical flow in the (i-1)th round and the first feature through a forward optical flow estimation module to obtain the optical flow and residual information of the mask in the i-th round. The optical flow and mask obtained in round i-1 are fused with the residual information of the optical flow and mask in round i to obtain the optical flow and mask in round i. Based on the optical flow in round i, the first video frame and the second video frame are transformed to obtain the first transformed frame and the second transformed frame in round i. In response to the round number i not being equal to the preset round number N, the round number i is incremented by 1, and the above steps are repeated. In response to the round number i being equal to the preset round number N, the target prediction video frame is generated based on the first transformed frame, the second transformed frame, and the mask obtained in round i.

[0008] In an exemplary embodiment, the first video frame, the second video frame, and the target predicted video frame are time-continuous video frames.

[0009] In an exemplary embodiment, the scaling factor is associated with the number of rounds i.

[0010] In an exemplary embodiment, generating the target prediction video frame based on the first changed frame, the second changed frame, and the mask obtained in the i-th round includes:

[0011] The target prediction video frame is generated by weighted summation of the first changed frame, the second changed frame, and the mask obtained in the i-th round, as shown in the following formula:

[0012] in, This represents the first changed frame obtained in the i-th round. I2 represents the second changed frame obtained in the i-th round, and M represents the target predicted video frame. i This represents the mask obtained in the i-th round.

[0013] In an exemplary embodiment, the step of estimating the optical flow of the (i-1)th round and the first feature based on the scaling factor of the i-th round, and obtaining the optical flow and mask residual information of the i-th round, includes: estimating the optical flow variation of the (i-1)th round and the first feature based on the scaling factor of the i-th round using an optical flow variation module to obtain optical flow variation features; estimating the spatial variation of the (i-1)th round and the first feature using a spatial variation module to obtain spatial variation features; concatenating the optical flow variation features and the spatial variation features to obtain optical flow estimation features; and performing a deconvolution operation on the optical flow estimation features to obtain the optical flow and mask residual information of the i-th round.

[0014] In an exemplary embodiment, the step of estimating the optical flow change of the optical flow and the first feature in the (i-1)th round based on the scaling factor of the i-th round to obtain the optical flow change feature includes: concatenating the optical flow and the first feature in the (i-1)th round to obtain the second feature; performing channel alignment on the second feature to obtain the third feature; and performing optical flow feature learning on the third feature through the residual feature learning module to obtain the optical flow change feature.

[0015] In an exemplary embodiment, concatenating the optical flow and the first feature in the (i-1)th round to obtain the second feature includes: performing a downsampling operation on the optical flow and the first feature in the (i-1)th round to obtain the downsampled optical flow and the first feature; concatenating the downsampled optical flow and the first feature to obtain the second feature; and performing optical flow feature learning on the third feature through the residual feature learning module to obtain the optical flow variation feature includes: performing optical flow feature learning on the third feature through the residual feature learning module to obtain the fourth feature; performing a channel alignment operation on the fourth feature to obtain the fifth feature; and performing an upsampling operation on the fifth feature to obtain the optical flow variation feature; wherein the sampling ratios of the downsampling and upsampling are associated with the scaling factor.

[0016] In an exemplary embodiment, the step of performing channel alignment on the second feature to obtain the third feature includes: performing at least one convolution operation on the second feature to obtain the third feature aligned with a preset number of channels.

[0017] In an exemplary embodiment, the step of estimating the spatial variation of the optical flow and the first feature in the (i-1)th round using the spatial variation module to obtain the spatial variation feature includes: concatenating the optical flow and the first feature in the (i-1)th round to obtain a sixth feature; performing channel alignment on the sixth feature to obtain a seventh feature; performing spatial attention learning on the seventh feature using a group-based spatial attention module (GSA) to obtain the spatial attention feature; and performing feature learning on the spatial attention feature to obtain the spatial variation feature.

[0018] In an exemplary embodiment, the step of performing spatial attention learning on the seventh feature through a grouped spatial attention module to obtain the spatial attention feature includes: performing a first dilated convolution operation on the seventh feature to obtain an eighth feature; dividing the eighth feature into at least one channel feature according to channels; performing dilated convolution operations on each of the at least one channel feature to obtain at least one channel spatial feature; concatenating the at least one channel spatial feature to obtain a ninth feature; performing feature learning on the seventh and ninth features respectively to obtain a tenth and thirteenth features; performing activation processing on the tenth and thirteenth features, and multiplying the activated tenth and thirteenth features by a dot product to obtain an eleventh feature; performing feature learning on the eleventh feature to obtain a twelfth feature; and adding the seventh and twelfth features to obtain the spatial attention feature.

[0019] In an exemplary embodiment, dividing the eighth feature into at least one channel feature according to the number of channels includes: dividing the eighth feature into at least one channel feature according to the number of channels according to a preset channel number ratio.

[0020] In an exemplary embodiment, the step of performing dilated convolution operations on the at least one channel feature to obtain at least one channel spatial feature further includes: the convolution kernel size of the dilated convolution operation corresponding to different channel features is different.

[0021] In an exemplary embodiment, the method further includes: determining whether the number of generated target predicted video frames is greater than the number of remaining frames in the video to be encoded; and discarding the target predicted video frames based on the number of remaining frames in response to the number of target predicted video frames being greater than the number of remaining frames.

[0022] According to another aspect of this disclosure, an optical flow-based video coding apparatus is provided, comprising: a video frame stitching module configured to stitch together a first video frame, a second video frame, a first modified frame obtained in the (i-1)th round, a second modified frame, and a mask to obtain a first feature; wherein the first modified frame and the second modified frame are obtained based on the first video frame and the second video frame, respectively; a forward optical flow estimation module configured to estimate the optical flow of the (i-1)th round and the first feature based on a scaling factor in the i-th round, to obtain the optical flow and residual information of the mask in the i-th round; and a fusion module. A module is configured to fuse the optical flow and mask obtained in the (i-1)th round with the residual information of the optical flow and mask in the i-th round to obtain the optical flow and mask in the i-th round; a change processing module is configured to perform change processing on the first video frame and the second video frame based on the optical flow in the i-th round to obtain the first changed frame and the second changed frame in the i-th round; a prediction frame module is configured to increment the number of rounds i by 1 in response to the number of rounds i not being equal to the preset number of rounds N, and repeat the aforementioned steps; and generate a target prediction video frame based on the first changed frame, the second changed frame, and the mask obtained in the i-th round in response to the number of rounds i being equal to the preset number of rounds N.

[0023] According to another aspect of this disclosure, an electronic device is provided, comprising: one or more processors; and a storage device configured to store one or more programs that, when executed by the one or more processors, cause the one or more processors to implement the optical flow-based video interpolation method as described in the above embodiments.

[0024] According to another aspect of this disclosure, a computer-readable storage medium is provided that stores a computer program, which, when executed by a processor, implements the optical flow-based video frame interpolation method as described in the above embodiments.

[0025] According to another aspect of this disclosure, a computer program product is provided, including a computer program / signaling, characterized in that, when the computer program / signaling is executed by a processor, it implements the optical flow-based video frame interpolation method as described in the above embodiments.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0027] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure, and are not intended to unduly limit this disclosure.

[0028] Figure 1 shows a flowchart of a video frame interpolation method based on optical flow according to an embodiment of the present disclosure;

[0029] Figure 2 shows a schematic diagram of the optical flow-based video coding process according to an embodiment of the present disclosure;

[0030] Figure 3 shows a flowchart of the optical flow estimation process according to an embodiment of the present disclosure;

[0031] Figure 4 shows a flowchart of the optical flow variation estimation process according to an embodiment of the present disclosure;

[0032] Figure 5 shows a flowchart of the spatial variation estimation process according to an embodiment of the present disclosure;

[0033] Figure 6 shows a schematic diagram of the structure of the grouped spatial attention module according to an embodiment of the present disclosure;

[0034] Figure 7 shows a schematic diagram of the results of the forward optical flow estimation module according to an embodiment of the present disclosure;

[0035] Figure 8 shows a schematic diagram of the structure of a video coding device based on optical flow according to an embodiment of the present disclosure;

[0036] Figure 9 shows a schematic diagram of the structure of an electronic device suitable for implementing exemplary embodiments of the present disclosure. Detailed Implementation

[0037] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0038] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0039] It should be noted that the ordinal numbers such as "first" and "second" mentioned in the embodiments of this disclosure are used to distinguish multiple objects, and are not used to limit the order, timing, priority or importance of multiple objects. Furthermore, the descriptions of "first" and "second" do not limit the objects to necessarily being different.

[0040] To address the aforementioned issues, this disclosure proposes an optical flow-based video frame interpolation method. By improving existing frame interpolation algorithms based on optical flow estimation, it avoids dependence on backward reference frames and meets the low-latency mode requirements in video coding. This method significantly reduces processing latency without sacrificing frame prediction quality.

[0041] Figure 1 shows a flowchart of a video interpolation method based on optical flow according to an embodiment of the present disclosure. Figure 2 shows a schematic diagram of a video encoding process based on optical flow according to an embodiment of the present disclosure. As shown in Figures 1 and 2, the video interpolation method based on optical flow may include the following steps.

[0042] In step S110, the first video frame, the second video frame, the first change frame and the second change frame obtained in the (i-1)th round, and the mask are stitched together to obtain the first feature; wherein, the first change frame and the second change frame are obtained based on the change processing of the first video frame and the second video frame, respectively.

[0043] In this embodiment of the disclosure, the video frame interpolation method predicts future video frames based on optical flow estimation of existing video frames, thereby encoding the inserted video frames. Optical flow video frame prediction is a technique for predicting future frames based on the optical flow field between adjacent frames. Optical flow refers to the pixel-level motion vector of an object, surface, or edge in a continuous sequence of image frames. By calculating optical flow, changes between frames can be captured, which can then be used to predict the content of subsequent frames. This method is typically used to predict one frame over several consecutive frames, and multiple frames are generated through rolling prediction, featuring high computational efficiency and good prediction accuracy in the short term.

[0044] In this embodiment of the disclosure, the optical flow video frame prediction method employs a stepwise approximation approach. By iteratively estimating the optical flow of existing video frames, the prediction of future video frames is gradually approximated. Therefore, steps S110 to S140 in this scheme constitute a stepwise approximation cyclic estimation process. The process proceeds stepwise according to a pre-defined cyclic estimation procedure until a preset loop exit condition is met.

[0045] In this embodiment of the disclosure, as shown in FIG2, the first video frame I0, the second video frame I1, and the first change frame obtained in the (i-1)th round are combined. Second changing frame and mask M i-1 The features are then concatenated to obtain the first feature X1.

[0046] Among them, the first video frame I0 and the second video frame I1 are video frames that existed before the video frame to be predicted.

[0047] The first change frame Second changing frame These are the first and second modified frames obtained in the (i-1)th round of the aforementioned loop. The first and second modified frames are obtained by warping the first and second video frames, respectively. Warping is an optical flow transformation technique in computer vision and image processing. It mainly refers to transforming one image to match the position and viewpoint of another image based on a given optical flow field. Specifically, warping compensates for the displacement between images by moving pixels, enabling one image to align with another.

[0048] The mask M i-1 It is a mask image representing changes in optical flow obtained through optical flow estimation.

[0049] In an exemplary embodiment, the first video frame I0, the second video frame I1, and the target predicted video frame are adjacent video frames that are consecutive in the temporal domain.

[0050] It should be noted that the exemplary embodiments given in this disclosure predict one video frame to be predicted based on two existing video frames. In practical applications, multiple video frames to be predicted can also be predicted based on multiple existing video frames as needed. For example, two video frames to be predicted can be predicted based on three existing video frames.

[0051] In an exemplary embodiment, the simultaneous prediction of multiple video frames to be predicted using multiple existing video frames can be further enhanced by a rolling prediction method. Rolling prediction refers to the process of sequentially predicting a series of consecutive frames, starting from the current frame and predicting future frames one by one in chronological order, achieving long-sequence frame prediction by continuously updating reference frames. During rolling prediction, the reference frame can be an original video frame or a previously predicted frame. Rolling prediction is performed forward using the already obtained predicted frames.

[0052] In step S120, based on the scaling factor of the i-th round, the optical flow of the (i-1)-th round and the first feature are estimated by the forward optical flow estimation module to obtain the optical flow of the i-th round and the residual information of the mask.

[0053] In this embodiment of the present disclosure, as shown in FIG2, the first feature X1 obtained by splicing together the aforementioned step S110 and the optical flow F of the (i-1)th round are combined. i-1 Input the values ​​into the Forward Flow Estimation (FFE) module. The optical flow F... i-1This is the optical flow information (Flow) obtained through optical flow estimation. The forward optical flow estimation module is a technique for estimating pixel motion vectors from the current video frame to the next video frame. This module predicts the optical flow field by analyzing changes between adjacent frames, thereby supporting video frame interpolation or prediction. A scaling factor is pre-set in this forward optical flow estimation module. This scaling factor is used to adjust the scaling ratio of the relevant input features. Based on the pre-set scaling factor, the forward optical flow estimation module estimates the optical flow F of the first input feature X1 and the optical flow F in the (i-1)th round. i-1 Optical flow estimation is performed to obtain the optical flow in the i-th round and the residual information ΔF of the mask. i ΔM i Residual information refers to the difference between the actual value and the predicted value in signal processing, used to represent information or errors that were not captured.

[0054] In an exemplary embodiment, the scaling factor is associated with the current round number i, meaning the scaling factor changes as the number of rounds increases, thereby gradually approximating the actual value. For example, the scaling factor is updated every three rounds. The scaling factor decreases as the number of rounds increases.

[0055] In an exemplary embodiment, the scaling factor is not a pre-set fixed value, but is calculated and adjusted according to the parameter values ​​obtained in the previous iterations during the loop process. This disclosure does not specifically limit how to select the parameters of the previous iterations or how to design the relevant calculation formulas.

[0056] In step S130, the optical flow and mask obtained in the (i-1)th round and the residual information of the optical flow and mask in the i-th round are fused to obtain the optical flow and mask in the i-th round.

[0057] In this embodiment of the disclosure, the optical flow F obtained in the (i-1)th round is... i-1 Compared with the optical flow residual information ΔF obtained in the i-th round mentioned above i The optical flow F in the i-th round is obtained by fusion through the fusion module. i Similarly, the mask M obtained in the (i-1)th round... i-1 Compared with the mask residual information ΔM obtained in the i-th round mentioned above i The fusion is performed by the fusion module to obtain the mask M in the i-th round. i For details, please refer to the following formula: F i =F i-1 +ΔF i M i =M i-1 +ΔM i

[0058] In step S140, the first video frame and the second video frame are modified based on the optical flow of the i-th round to obtain the first modified frame and the second modified frame of the i-th round.

[0059] In this embodiment of the disclosure, in order to perform the next round of cyclic processing, based on the optical flow F of the aforementioned i-th round... i Warping is performed on the first video frame I0 and the second video frame I1 to obtain the first warped frame in the i-th round. Second changing frame Through the above-described transformation process, the first transformed frame required for feature stitching in step S110 of the next round is obtained. Second changing frame

[0060] In this embodiment of the disclosure, steps S110 to S140 constitute a cyclic process. Through this cyclic process, the first changed frame is obtained based on the first video frame I0, the second video frame I1, and the (i-1)th round. Second changing frame Mask M i-1 and optical flow F i-1 The optical flow can be estimated by the forward optical flow estimation module, and finally the optical flow F of the i-th round can be obtained. i and mask M i and the first change frame of the i-th round. Second changing frame This process is repeated, and as the number of rounds i increases, the value of the video frame to be predicted gradually approaches the true value.

[0061] In step S150, in response to the fact that the number of rounds i is not equal to the preset number of rounds N, the number of rounds i is incremented by 1, and the aforementioned steps are repeated.

[0062] In this embodiment of the disclosure, the exit condition for the loop process is set to a preset number of rounds N. If the current round number i is not equal to the preset number of rounds N, i.e., the exit condition is not met, the loop process continues. Based on this loop process, the round number i is incremented by 1, and the aforementioned steps S110 to S140 are repeated.

[0063] In step S160, in response to the round number i being equal to the preset round number N, a target prediction video frame is generated based on the first change frame, the second change frame, and the mask obtained in the i-th round.

[0064] In this embodiment of the disclosure, the exit condition for the loop process is set to a preset number of rounds N. In response to the current round number i equaling the preset number of rounds N, i.e., the exit condition is met, the loop process exits. The first change frame is obtained based on the i-th round. Second changing frame and mask Mi Generate target prediction video frame I2.

[0065] In an exemplary embodiment, the target prediction video can be based on the first change frame obtained in the i-th round. Second changing frame and mask M i The weighted sum is then obtained. See the following formula for details:

[0066] in, This represents the first changed frame obtained in the i-th round. I2 represents the second changed frame obtained in the i-th round, and M represents the target predicted video frame. i This represents the mask obtained in the i-th round.

[0067] In an exemplary embodiment, before generating the target prediction video frame based on the first and second changed frames obtained in the i-th round, an activation (sigmoid) operation needs to be performed on the first and second changed frames. The target prediction video frame is then generated based on the activated first and second changed frames and the mask.

[0068] In an exemplary embodiment, during video encoding, the total number of video frames after insertion should remain consistent with the total number of video frames in the original video. Therefore, when multiple insertion video frames are predicted backwards based on existing video frames using the scheme provided in this disclosure, redundancy in the insertion video frames may occur, resulting in a discrepancy between the total number of insertion video frames and the total number of video frames in the original video. Therefore, special processing is needed for the remaining frames at the end of the video frames to ensure that the total number of encoded video frames remains consistent. Specifically, this may include the following steps.

[0069] Determine whether the number of generated target prediction video frames is greater than the number of remaining frames in the video to be encoded.

[0070] In response to the number of target predicted video frames being greater than the number of remaining frames, the target predicted video frames are discarded based on the number of remaining frames.

[0071] After generating the target predicted video frames based on step S160, the number of generated target predicted video frames is compared with the remaining number of frames in the current video to be encoded. When the number of target predicted video frames is less than or equal to the remaining number of frames, the prediction of inserted video frames is performed normally according to the method provided in this disclosure. When the number of target predicted video frames is greater than the remaining number of frames, it indicates that there is redundancy in the generated target predicted video frames, and some of the generated target predicted video frames are discarded based on the remaining number of frames. For example, two target predicted video frames are generated based on three existing video frames, while the remaining number of frames in the current video to be encoded is one. Based on this, one of the two generated target predicted video frames is discarded based on the remaining number of frames to keep the total number of encoded video frames consistent.

[0072] The optical flow-based video frame interpolation method provided in this disclosure uses a forward optical flow estimation module to capture the optical flow variation characteristics of the video frame to be predicted by using multiple video frames preceding the frame to be predicted as reference frames. Through multiple rounds of iterative processing, the method gradually approximates the true value of the video frame to be predicted, ultimately generating the target predicted video frame. This method improves the accuracy of prediction by gradually approximating the true value of the video frame to be predicted through multi-round cyclic optical flow estimation. Furthermore, by using only forward video frames as reference frames, it avoids the impact of introducing backward reference frames on coding efficiency, thus meeting the requirements of low-latency temporal coding modes.

[0073] Figure 3 shows a flowchart of the optical flow estimation process according to an embodiment of the present disclosure. Figure 7 shows a schematic diagram of the results of the forward optical flow estimation module according to an embodiment of the present disclosure. As shown in Figures 3 and 7, based on the optical flow-based video frame interpolation method shown in Figure 1, step S120 may include the following steps.

[0074] In step S310, based on the scaling factor of the i-th round, the optical flow of the (i-1)-th round and the first feature are estimated by the optical flow variation module to obtain the optical flow variation feature.

[0075] In this embodiment of the present disclosure, as shown in FIG7, the forward optical flow estimation module includes an optical flow variation module and a spatial variation module. The optical flow variation module, based on the scaling factor of the i-th round, modifies the optical flow F input to the (i-1)-th round of the forward optical flow estimation module. i-1 Optical flow variation estimation is performed using the first feature X1 to obtain the optical flow variation feature X. f This optical flow variation estimation predicts changes in the optical flow field by analyzing the optical flow field and feature map of the previous round, given the scaling factor of the current round. Optical flow variation feature X f It is used to characterize the correlation features of optical flow changes between consecutive video frames.

[0076] In step S320, the spatial variation of the optical flow and the first feature in the (i-1)th round is estimated by the spatial variation module to obtain the spatial variation feature.

[0077] In this embodiment of the present disclosure, as shown in FIG7, the forward optical flow estimation module includes an optical flow variation module and a spatial variation module. The spatial variation module, based on the scaling factor of the i-th round, modifies the optical flow F input to the (i-1)-th round of the forward optical flow estimation module. i-1 Spatial variation estimation is performed using the first feature X1 to obtain the spatial variation feature X. s This spatial variation estimation predicts the spatial variations of an image by analyzing the optical flow and feature maps from the previous round, given the scaling factor of the current round. Spatial variation feature X s , used to characterize the relevant features of spatial changes between video frames.

[0078] In step S330, the optical flow variation features and spatial variation features are spliced ​​together to obtain optical flow estimation features.

[0079] In this embodiment of the disclosure, the optical flow variation feature X obtained in the aforementioned steps S310 and S320 is... f and spatial variation characteristics X s By stitching the images together, we obtain the optical flow estimation feature X. ffe .

[0080] In an exemplary embodiment, the optical flow variation feature X f and spatial variation characteristics X s Video frames are stitched together based on feature channels. Feature channels are used to represent feature information in different dimensions within a video frame.

[0081] In step S340, the optical flow estimation features are deconvolved to obtain the optical flow and mask residual information of the i-th round.

[0082] In this embodiment of the disclosure, the optical flow estimation feature X ffe Perform a deconvolution operation to obtain the optical flow and mask residual information ΔF in the i-th round. i ΔM i As mentioned earlier, the optical flow estimation feature X ffe It is generated based on feature channel stitching. Therefore, the optical flow estimation feature X... ffe After performing the deconvolution operation, the residual information ΔF of the optical flow can be extracted based on the correspondence of the feature channels. i and the residual information ΔM of the mask i .

[0083] Figure 4 shows a flowchart of the optical flow variation estimation process according to an embodiment of the present disclosure. Figure 7 shows a schematic diagram of the results of the forward optical flow estimation module according to an embodiment of the present disclosure. As shown in Figures 4 and 7, based on the optical flow-based video frame interpolation method shown in Figure 3, step S310 may include the following steps.

[0084] In step S410, the optical flow of the (i-1)th round and the first feature are spliced ​​together to obtain the second feature.

[0085] In this embodiment of the present disclosure, as shown in FIG7, the optical flow F of the (i-1)th round input to the forward optical flow estimation module is... i-1 The first feature X1 is concatenated with the second feature X2 to obtain the second feature X2.

[0086] In an exemplary embodiment, as shown in FIG7, in order to improve the efficiency of data processing, the optical flow F is further subjected to the above feature stitching before the feature stitching is performed. i-1 The optical flow and the first feature X1 are downsampled to obtain the downsampled optical flow and the first feature. The downsampled optical flow and the first feature are then concatenated to obtain the second feature X2.

[0087] In an exemplary embodiment, the sampling factor of the downsampling process is Scale_list[i]. The sampling factor of the downsampling process is associated with the scaling factor of the current round, and the sampling factor is adjusted according to the scaling factor.

[0088] In step S420, the second feature is subjected to channel alignment to obtain the third feature.

[0089] In this embodiment of the disclosure, as shown in FIG7, a channel alignment operation is performed on the second feature X2 to obtain the third feature X3. The channel alignment operation refers to adjusting the number of feature channels of the feature to align with the target number of channels. The target number of channels may vary depending on the needs of subsequent processing. No specific target number of channels is limited here. Furthermore, this channel alignment operation is a commonly used technique in the art and will not be further described here.

[0090] In an exemplary embodiment, at least one convolution operation is performed on the second feature to obtain a third feature aligned with a preset number of channels. In this example, channel alignment is achieved through multiple convolution operations. For example, where stride = 2 and M is the preset number of feature channels. Therefore, performing two convolution operations on the second feature yields a third feature aligned with the preset number of feature channels M. After the first convolution operation, an M / 2 channel feature is output. After the second convolution operation, an M channel feature is output.

[0091] In step S430, the optical flow feature is learned by the residual feature learning module to obtain the optical flow change feature.

[0092] In this embodiment of the disclosure, as shown in FIG7, the optical flow feature is learned by the residual feature learning module to obtain the optical flow change feature X3. f The residual feature learning module is a network module used to learn residual information from input features, aiming to capture the difference between input features and target features. For example, the ResBlock network. Optical flow feature learning refers to the process of estimating pixel-level optical flow motion vectors by analyzing image sequences.

[0093] In an exemplary embodiment, as shown in FIG7, corresponding to the downsampling operation in step S410 above, the optical flow feature learning is performed on the third feature through the residual feature learning module to obtain the optical flow change feature, including:

[0094] The fourth feature is obtained by performing optical flow feature learning on the third feature through the residual feature learning module;

[0095] Perform channel alignment on the fourth feature to obtain the fifth feature;

[0096] The fifth feature is upsampled to obtain the optical flow variation feature.

[0097] In an exemplary embodiment, the optical flow feature is learned from the third feature using the residual feature learning module to first obtain the fourth feature X4. Then, channel alignment is performed on the fourth feature X4 to obtain the fifth feature X5. Finally, an upsampling operation is performed on the fifth feature X5 to obtain the optical flow variation feature X. f This upsampling process is the reverse of the downsampling process in step S410. Therefore, the sampling ratio of this upsampling process is the same as that of the downsampling process, which is Scale_list[i]. Similarly, the sampling ratio of this upsampling process is also related to the scaling factor of the current round, and the sampling ratio is adjusted according to the scaling factor.

[0098] Figure 5 shows a flowchart of the spatial variation estimation process according to an embodiment of the present disclosure. Figure 7 shows a schematic diagram of the results of the forward optical flow estimation module according to an embodiment of the present disclosure. As shown in Figures 5 and 7, based on the optical flow-based video frame interpolation method shown in Figure 3, step S320 may include the following steps.

[0099] In step S510, the optical flow of the (i-1)th round and the first feature are spliced ​​together to obtain the sixth feature.

[0100] In this embodiment of the present disclosure, as shown in FIG7, the optical flow F of the (i-1)th round input to the forward optical flow estimation module is... i-1The sixth feature X6 is obtained by concatenating it with the first feature X1. This sixth feature X6 may be the same as the aforementioned second feature X2, or it may be different from the second feature X2 due to different concatenation methods. This is not limited here.

[0101] In step S520, the sixth feature is subjected to channel alignment to obtain the seventh feature.

[0102] In this embodiment of the disclosure, based on the channel alignment requirement, the sixth feature X6 is subjected to channel alignment operation to obtain the seventh feature X7.

[0103] In step S530, the seventh feature is spatial attention learned by the Group-based Spatial Attention (GSA) module to obtain the spatial attention feature.

[0104] In this embodiment of the disclosure, spatial attention feature X7 is obtained by performing spatial attention learning on the seventh feature X7 through a pre-designed grouped spatial attention module. A .

[0105] This spatial attention module is designed to extract attention features from video frames at different scales in the spatial domain. It applies dilated convolutions of varying sizes to extract features at different scales from the video frames. This extraction of features at different scales enhances the ability to capture different levels of detail, thus better capturing the spatial variation features of the video frames. The obtained spatial attention feature X... A It is used to characterize the spatial variation features of video frames.

[0106] In an exemplary embodiment, different weights can be assigned to the grouped spatial attention modules in different rounds according to the aforementioned iterative process. These weights are associated with the aforementioned scaling factor. Grouped spatial attention modules with the same scaling factor have the same weight, while grouped spatial attention modules with different scaling factors have different weights, depending on the scaling factor corresponding to the round.

[0107] In step S540, feature learning is performed on the spatial attention features to obtain the spatial change features.

[0108] In this embodiment of the disclosure, the spatial attention feature X is... A Feature learning is performed to obtain the spatial variation feature X. s .

[0109] In an exemplary embodiment, the spatial attention feature X is applied through at least one convolutional operation. A Feature learning is then performed. Specifically, this can be achieved using a 1x1 convolutional kernel to learn the spatial attention feature X.A Perform convolution operations to obtain the spatial variation features X. s .

[0110] Figure 6 shows a schematic diagram of the structure of the grouped spatial attention module according to an embodiment of the present disclosure. As shown in Figure 6, based on the optical flow-based video frame interpolation method shown in Figure 5, step S530 may include the following steps.

[0111] Step 1: Perform a first dilated convolution operation on the seventh feature to obtain the eighth feature.

[0112] In this embodiment of the disclosure, the seventh feature X7 is subjected to a first dilated convolution operation to obtain the eighth feature X8.

[0113] Dilated convolution is a convolutional approach proposed to address the issue of reduced image resolution and information loss caused by downsampling in image semantic segmentation. It introduces a new parameter called the "dilation rate," which defines the spacing between values ​​in the convolution kernel when processing data. The dilation rate determines the interval between elements in the convolution kernel, thereby increasing its receptive field (i.e., the area that the convolution kernel can cover) while keeping the kernel size constant.

[0114] In an exemplary embodiment, the dilated convolution operation can be adjusted using multiple parameters. The adjustment of the dilated convolution operation can be represented by DWConv(a,b,c,d), where a represents the number of input feature channels, b represents the number of output feature channels, c represents the kernel size, and d represents the stride. Additionally, Dilation represents the dilation rate of the dilated convolution. As shown in Figure 6, in this example, the first dilated convolution is set to DWConv(C,C,3,1), with Dilation = 1.

[0115] Step 2: Divide the eighth feature into at least one channel feature according to the channel; perform dilated convolution operation on each of the at least one channel feature to obtain at least one channel spatial feature.

[0116] In this embodiment of the disclosure, the eighth feature X8 is divided into multiple different channel features X according to the feature channels. C For the multiple channel features X that have been divided C Then perform dilated convolution operations separately to obtain the features X of each channel. C Corresponding multiple channel spatial features X DW .

[0117] In an exemplary embodiment, during the channel feature segmentation process, the eighth feature X8 can be divided into at least one channel feature X according to a pre-set channel number ratio.C As shown in Figure 6, in this example, the eighth feature X8 is divided into four channel features X1 according to the number of channels. C1 X C2 X C3 X C4 For example, if the eighth feature X8 includes 16 feature channels, then the features of channels 0-3 are divided into channel features X. C1 The features of channels 4-7 are divided into channel features X. C2 The features of channels 8-11 are divided into channel features X. C3 The features of channels 12-15 are divided into channel features X. C4 Of course, depending on the actual application needs, the number of channels for each feature can not be evenly divided, but rather according to a pre-set channel ratio. For example, the eighth feature X8 includes 16 feature channels, which can be divided into three channel features in a 2:1:1 ratio. Then, the features of channels 0-7 are divided into channel features X. C1 The features of channels 8-11 are divided into channel features X. c2 The features of channels 12-15 are divided into channel features X. C3 .

[0118] In an exemplary embodiment, as described above, the dilated convolution operation can be adjusted using multiple parameters. Based on this, in this example, the kernel size for the dilated convolution operation varies depending on the channel feature, thus performing dilated convolution operations at different scales. As shown in Figure 6, in this example, for channel feature X... C1 Without performing dilated convolution, this feature is preserved, resulting in the channel spatial feature X. DW For channel feature X C2 The kernel size for the dilated convolution operation is 3x3, and Dilation = 3, yielding the channel spatial features X. DW For channel feature X C3 The kernel size for the dilated convolution operation is 5x5, and the dilation is 3, resulting in the channel spatial feature X. DW3 For channel feature X C4 The kernel size for the dilated convolution operation is 7x7, and dilation = 3, yielding the channel spatial features X. DW By applying dilated convolutions of different sizes, features at different scales can be extracted from video frames in the spatial domain, enhancing the ability to capture different levels of detail and thus better capturing the spatial variation features of video frames.

[0119] It should be pointed out that not performing dilated convolution on channel features to preserve the original features should also be regarded as a special dilated convolution operation, as one of the different dilated convolution operations that can be selected.

[0120] Step 3: Segment the spatial features of at least one channel to obtain the ninth feature.

[0121] In this embodiment of the disclosure, the aforementioned multiple channel spatial features X DW By splicing the features together, we obtain the ninth feature, X9.

[0122] Step 4: Perform feature learning on the seventh and ninth features respectively to obtain the tenth and thirteenth features.

[0123] In this embodiment of the disclosure, feature learning is performed on the seventh feature X7 and the ninth feature X9 respectively to obtain the tenth feature X. 10 and the thirteenth feature X 13 As shown in Figure 6, this feature can be learned using a single convolutional layer. This convolutional operation can be set to Conv(C,C,3,1).

[0124] Step 5: Activate the tenth and thirteenth features, and multiply the activated tenth and thirteenth features by a dot product to obtain the eleventh feature.

[0125] In this embodiment of the disclosure, the tenth feature X is... 10 and the thirteenth feature X 13 Perform activation processing, and then apply the activated tenth feature X. 10 and the thirteenth feature X 13 Performing a dot product, we obtain the eleventh feature X. 11 .

[0126] This activation process can be performed using different activation functions such as Sigmoid, ReLU, or SiLU.

[0127] Step 6: Perform feature learning on the eleventh feature to obtain the twelfth feature.

[0128] In this embodiment of the disclosure, the eleventh feature X 11 Feature learning is performed to obtain the twelfth feature X. 12 As shown in Figure 6, this feature can be learned using a single convolutional layer. This convolutional operation can be set to Conv(C,C,3,1).

[0129] Step 7: Add the seventh feature and the twelfth feature together to obtain the spatial attention feature.

[0130] In this embodiment of the disclosure, the seventh feature X7 and the twelfth feature X7 are used. 12 By summing the features, we obtain the spatial attention feature X. A .

[0131] The optical flow-based video frame interpolation method provided in this disclosure incorporates an optical flow variation module and a spatial variation module within the forward optical flow estimation module. The optical flow variation module captures optical flow variation features between reference frames. The spatial variation module captures spatial variation features between reference frames. By combining optical flow variation features and spatial variation features, the variation features between reference frames can be captured more effectively, thereby enabling accurate prediction of future video frames. Specifically, a scaling factor is introduced into the optical flow variation module, and a residual feature learning module is used to capture the optical flow variation features. In the spatial variation module, dilated convolutions of different sizes are applied to extract features of different scales in the spatial domain of the video frames, enhancing the ability to capture different levels of detail and thus better capturing the spatial variation features of the video frames.

[0132] Based on the same inventive concept, this disclosure provides an optical flow-based video encoding apparatus, as described in the following embodiments. Since the principle by which this apparatus solves the problem is similar to that of the method embodiments described above, the implementation of this communication authentication apparatus can be referenced in real-time to the methods described above, and repeated details will not be repeated.

[0133] Figure 8 shows a schematic diagram of the structure of an optical flow-based video coding apparatus according to an embodiment of the present disclosure. As shown in Figure 8, the optical flow-based video coding apparatus 800 may include: a video frame stitching module 810, a forward optical flow estimation module 820, a fusion module 830, a change processing module 840, and a prediction frame module 850.

[0134] The video frame stitching module 810 is configured to stitch together the first video frame, the second video frame, the first changed frame and the second changed frame obtained in the (i-1)th round, and the mask to obtain a first feature; wherein the first changed frame and the second changed frame are obtained based on the changes of the first video frame and the second video frame, respectively.

[0135] The forward optical flow estimation module 820 is configured to estimate the optical flow of the (i-1)th round and the first feature based on the scaling factor of the i-th round, so as to obtain the optical flow of the i-th round and the residual information of the mask.

[0136] The fusion module 830 is configured to fuse the optical flow and mask obtained in the (i-1)th round and the residual information of the optical flow and mask in the i-th round to obtain the optical flow and mask in the i-th round.

[0137] The change processing module 840 is configured to perform change processing on the first video frame and the second video frame based on the optical flow of the i-th round to obtain the first changed frame and the second changed frame of the i-th round.

[0138] The prediction frame module 850 is configured to increment the number of rounds i by 1 in response to the number of rounds i not being equal to the preset number of rounds N, and repeat the aforementioned steps; and to generate a target prediction video frame based on the first change frame, the second change frame, and the mask obtained in the i-th round in response to the number of rounds i being equal to the preset number of rounds N.

[0139] In an exemplary embodiment, the first video frame, the second video frame, and the target predicted video frame are time-continuous video frames.

[0140] In an exemplary embodiment, the scaling factor is associated with the number of rounds i.

[0141] In an exemplary embodiment, the prediction frame module 850 is further configured to perform a weighted summation based on the first changed frame, the second changed frame, and the mask obtained in the i-th round to generate the target predicted video frame, with the corresponding formula as follows:

[0142] in, This represents the first changed frame obtained in the i-th round. I2 represents the second changed frame obtained in the i-th round, and M represents the target predicted video frame. i This represents the mask obtained in the i-th round.

[0143] In an exemplary embodiment, the forward optical flow estimation module 820 is further configured to, based on the scaling factor of the i-th round, perform optical flow variation estimation on the optical flow and the first feature of the (i-1)-th round using the optical flow variation module to obtain optical flow variation features; perform spatial variation estimation on the optical flow and the first feature of the (i-1)-th round using the spatial variation module to obtain spatial variation features; concatenate the optical flow variation features and the spatial variation features to obtain optical flow estimation features; and perform deconvolution operation on the optical flow estimation features to obtain the optical flow and mask residual information of the i-th round.

[0144] In an exemplary embodiment, the forward optical flow estimation module 820 is further configured to concatenate the optical flow of the (i-1)th round and the first feature to obtain a second feature; perform channel alignment operation on the second feature to obtain a third feature; and perform optical flow feature learning on the third feature through the residual feature learning module to obtain the optical flow change feature.

[0145] In an exemplary embodiment, the forward optical flow estimation module 820 is further configured to perform a downsampling operation on the optical flow and the first feature in the (i-1)th round to obtain the downsampled optical flow and the first feature; concatenate the downsampled optical flow and the first feature to obtain the second feature; the step of learning optical flow features on the third feature through the residual feature learning module to obtain the optical flow variation feature includes: learning optical flow features on the third feature through the residual feature learning module to obtain the fourth feature; performing a channel alignment operation on the fourth feature to obtain the fifth feature; and performing an upsampling operation on the fifth feature to obtain the optical flow variation feature; wherein the sampling ratio of the downsampling and upsampling is associated with the scaling factor.

[0146] In an exemplary embodiment, the forward optical flow estimation module 820 is further configured to perform at least one convolution operation on the second feature to obtain the third feature aligned with a preset number of channels.

[0147] In an exemplary embodiment, the forward optical flow estimation module 820 is further configured to concatenate the optical flow of the (i-1)th round and the first feature to obtain a sixth feature; perform channel alignment on the sixth feature to obtain a seventh feature; perform spatial attention learning on the seventh feature through a group-based spatial attention module (GSA) to obtain the spatial attention feature; and perform feature learning on the spatial attention feature to obtain the spatial variation feature.

[0148] In an exemplary embodiment, the forward optical flow estimation module 820 is further configured to perform a first dilated convolution operation on the seventh feature to obtain an eighth feature; divide the eighth feature into at least one channel feature according to the channel; perform dilated convolution operations on the at least one channel feature to obtain at least one channel spatial feature; concatenate the at least one channel spatial feature to obtain a ninth feature; perform feature learning on the seventh feature and the ninth feature respectively to obtain a tenth feature and a thirteenth feature; perform activation processing on the tenth feature and the thirteenth feature, and perform dot product on the activated tenth feature and the thirteenth feature to obtain an eleventh feature; perform feature learning on the eleventh feature to obtain a twelfth feature; and add the seventh feature and the twelfth feature to obtain the spatial attention feature.

[0149] In an exemplary embodiment, the forward optical flow estimation module 820 is further configured to divide the eighth feature into at least one channel feature according to the number of channels, based on a pre-set channel number ratio.

[0150] In an exemplary embodiment, the forward optical flow estimation module 820 is further configured to have different kernel sizes for dilated convolution operations corresponding to different channel features.

[0151] In an exemplary embodiment, the prediction frame module 850 is further configured to determine whether the number of generated target predicted video frames is greater than the number of remaining frames in the video to be encoded; and in response to the number of target predicted video frames being greater than the number of remaining frames, to discard the target predicted video frames based on the number of remaining frames.

[0152] Figure 9 shows a schematic diagram of the structure of an electronic device suitable for implementing exemplary embodiments of the present disclosure. An electronic device 900 according to this embodiment of the present invention will now be described with reference to Figure 9. The electronic device 900 shown in Figure 9 is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present invention.

[0153] As shown in Figure 9, the electronic device 900 is presented in the form of a general-purpose computing device. The components of the electronic device 900 may include, but are not limited to: at least one processing unit 910, at least one storage unit 920, a bus 930 connecting different system components (including storage unit 920 and processing unit 910), and a display unit 940.

[0154] Storage unit 920 may include readable media in the form of volatile storage units, such as random access memory (RAM) 9201 and / or cache memory 9202, and may further include read-only memory (ROM) 9203.

[0155] Storage unit 920 may also include a program / utility 9204 having a set (at least one) program module 9205, such program module 9205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0156] Bus 930 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0157] Electronic device 900 can also communicate with one or more external devices 970 (e.g., keyboard, pointing device, Bluetooth device, etc.), and with one or more devices that enable a user to interact with electronic device 900, and / or with any device that enables electronic device 900 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 950. Furthermore, electronic device 900 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 960. As shown, network adapter 960 communicates with other modules of electronic device 900 via bus 930. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 900, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0158] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the methods described above is stored.

[0159] In some possible implementations, various aspects of the present invention may also be implemented as a program product comprising program code that, when the program product is run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of the present invention.

[0160] According to embodiments of the present invention, a program product for implementing the above-described method may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with a signaling execution system, apparatus, or device.

[0161] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0162] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in conjunction with a signaling execution system, apparatus, or device.

[0163] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0164] Program code for performing the operations of this invention can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0165] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0166] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0167] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several signaling instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the method according to the embodiments of this disclosure. Industrial applicability

[0168] This disclosure applies to the field of video encoding and decoding technology, aiming to solve the problem in related technologies that rely on future frames as references, thus failing to meet high real-time requirements. It employs a multi-round cyclic optical flow estimation method to progressively approximate the true value of the video frame to be predicted, improving the accuracy of the prediction estimate. Simultaneously, by using only forward video frames as reference frames, it avoids the impact of introducing backward reference frames on coding efficiency, thus meeting the requirements of low-latency temporal coding modes.

[0169] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

[0170] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A video frame interpolation method based on optical flow, the method comprising: The first video frame, the second video frame, the first changed frame and the second changed frame obtained in the (i-1)th round, and the mask are stitched together to obtain the first feature; wherein the first changed frame and the second changed frame are obtained based on the changes of the first video frame and the second video frame, respectively. Based on the scaling factor of round i, the optical flow of round i-1 and the first feature are estimated by the forward optical flow estimation module to obtain the optical flow of round i and the residual information of the mask. The optical flow and mask obtained in the (i-1)th round are fused with the residual information of the optical flow and mask in the i-th round to obtain the optical flow and mask in the i-th round. Based on the optical flow of the i-th round, the first video frame and the second video frame are transformed to obtain the first transformed frame and the second transformed frame of the i-th round. In response to the fact that the number of rounds i is not equal to the preset number of rounds N, the number of rounds i is incremented by 1, and the aforementioned steps are repeated; In response to the round number i being equal to the preset round number N, a target prediction video frame is generated based on the first changed frame, the second changed frame, and the mask obtained in the i-th round.

2. The method according to claim 1, wherein, The first video frame, the second video frame, and the target prediction video frame are time-continuous video frames.

3. The method according to claim 1, wherein, The scaling factor is associated with the round number i.

4. The method according to claim 1, characterized in that, The generation of target prediction video frames based on the first changed frame, the second changed frame, and the mask obtained in the i-th round includes: The target prediction video frame is generated by weighted summation of the first changed frame, the second changed frame, and the mask obtained in the i-th round, as shown in the following formula: in, This represents the first changed frame obtained in the i-th round. I2 represents the second changed frame obtained in the i-th round, and M represents the target predicted video frame. i This represents the mask obtained in the i-th round.

5. The method according to claim 1, wherein, The scaling factor based on the i-th round is used to estimate the optical flow of the (i-1)-th round and the first feature through the forward optical flow estimation module, obtaining the optical flow of the i-th round and the residual information of the mask, including: Based on the scaling factor of the i-th round, the optical flow of the (i-1)-th round and the first feature are estimated by the optical flow variation module to obtain the optical flow variation feature; The spatial variation feature is obtained by estimating the spatial variation of the optical flow and the first feature in the (i-1)th round through the spatial variation module. The optical flow variation features and spatial variation features are concatenated to obtain the optical flow estimation features; The optical flow estimation features are deconvolved to obtain the optical flow and mask residual information in the i-th round.

6. The method according to claim 5, wherein, Based on the scaling factor of the i-th round, the optical flow variation estimation is performed on the optical flow and the first feature of the (i-1)-th round using the optical flow variation module to obtain the optical flow variation features, including: The optical flow of the (i-1)th round and the first feature are concatenated to obtain the second feature; The second feature is then channel aligned to obtain the third feature; The optical flow variation feature is obtained by learning the optical flow feature of the third feature through the residual feature learning module.

7. The method according to claim 6, wherein, The step of concatenating the optical flow of the (i-1)th round and the first feature to obtain the second feature includes: Perform a downsampling operation on the optical flow and the first feature in the (i-1)th round to obtain the downsampled optical flow and the first feature; The downsampled optical flow and the first feature are concatenated to obtain the second feature; The optical flow variation features are obtained by learning optical flow features from the third feature through the residual feature learning module, including: The fourth feature is obtained by performing optical flow feature learning on the third feature through the residual feature learning module; Perform channel alignment on the fourth feature to obtain the fifth feature; The optical flow variation feature is obtained by upsampling the fifth feature. The sampling ratios of downsampling and upsampling are associated with the scaling factor.

8. The method according to claim 6, wherein, The step of performing channel alignment on the second feature to obtain the third feature includes: Perform at least one convolution operation on the second feature to obtain the third feature aligned with the preset number of channels.

9. The method according to claim 5, wherein, The spatial variation estimation of the optical flow and the first feature in the (i-1)th round is performed by the spatial variation module to obtain the spatial variation features, including: The optical flow of the (i-1)th round and the first feature are concatenated to obtain the sixth feature; The sixth feature is then subjected to channel alignment to obtain the seventh feature; Spatial attention features are obtained by performing spatial attention learning on the seventh feature through a group-based spatial attention module (GSA). The spatial attention features are subjected to feature learning to obtain the spatial change features.

10. The method according to claim 9, wherein, The step of performing spatial attention learning on the seventh feature through a grouped spatial attention module to obtain the spatial attention feature includes: Perform a first dilated convolution operation on the seventh feature to obtain the eighth feature; The eighth feature is divided into at least one channel feature according to the channel; dilated convolution operation is performed on the at least one channel feature to obtain at least one channel spatial feature; By splicing together the spatial features of the at least one channel, a ninth feature is obtained; Feature learning is performed on the seventh and ninth features respectively to obtain the tenth and thirteenth features; The tenth and thirteenth features are activated, and the activated tenth and thirteenth features are multiplied by a dot to obtain the eleventh feature; Feature learning is performed on the eleventh feature to obtain the twelfth feature; The spatial attention feature is obtained by adding the seventh and twelfth features together.

11. The method according to claim 10, wherein, The step of dividing the eighth feature into at least one channel feature according to the number of channels includes: dividing the eighth feature into at least one channel feature according to the number of channels according to a preset channel number ratio.

12. The method according to claim 10, wherein, The step of performing dilated convolution operations on the at least one channel feature to obtain at least one channel spatial feature further includes: the kernel size of the dilated convolution operation corresponding to different channel features is different.

13. The method according to claim 1, wherein, Also includes: Determine whether the number of generated target prediction video frames is greater than the number of remaining frames in the video to be encoded; In response to the number of target predicted video frames being greater than the number of remaining frames, the target predicted video frames are discarded based on the number of remaining frames.

14. A video frame interpolation device based on optical flow, comprising: The video frame stitching module is configured to stitch together the first video frame, the second video frame, the first changed frame and the second changed frame obtained in the (i-1)th round, and the mask to obtain the first feature; wherein the first changed frame and the second changed frame are obtained based on the changes of the first video frame and the second video frame, respectively. The forward optical flow estimation module is configured to estimate the optical flow of the (i-1)th round and the first feature based on the scaling factor of the i-th round, so as to obtain the optical flow of the i-th round and the residual information of the mask. The fusion module is configured to fuse the optical flow and mask obtained in the (i-1)th round and the residual information of the optical flow and mask in the i-th round to obtain the optical flow and mask in the i-th round. The change processing module is configured to perform change processing on the first video frame and the second video frame based on the optical flow of the i-th round to obtain the first changed frame and the second changed frame of the i-th round. The prediction frame module is configured to increment the number of rounds i by 1 and repeat the aforementioned steps in response to the round number i not being equal to the preset round number N; and to generate a target prediction video frame based on the first change frame, the second change frame, and the mask obtained in the i-th round in response to the round number i being equal to the preset round number N.

15. An electronic device comprising: One or more processors; A storage device configured to store one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the method as described in any one of claims 1 to 13.

16. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 13.

17. A computer program product comprising a computer program / signaling that, when executed by a processor, implements the method as described in any one of claims 1 to 13.

Images