Data processing method, apparatus, device, and storage medium

Abstract

The present disclosure provides a data processing method, an apparatus, a device, and a storage medium, and relates to the technical field of computers. The method comprises: acquiring a global parameter and, on the basis of a frame interpolation mode in the global parameter, performing frame interpolation. The efficiency of video encoding and decoding is improved.

Description

Data processing methods, apparatus, equipment and storage media

[0001] This application claims priority to Chinese Patent Application No. 202411538011.5, filed on October 30, 2024, entitled “Data Processing Method, Apparatus, Device, and Storage Medium”, and to Chinese Patent Application No. 202411595420.9, filed on November 8, 2024, both of which are incorporated herein by reference in their entirety. Technical Field

[0002] This disclosure relates to the field of computer technology, and in particular to a data processing method, apparatus, device and storage medium. Background Technology

[0003] With the development of technology, the speed of information transmission has increased significantly. Currently, video information constitutes a large proportion of the transmitted information. In the process of information transmission, frame extraction coding technology is a commonly used technique in the field of signal coding technology. Summary of the Invention

[0004] This disclosure provides a data processing method, apparatus, device, and storage medium.

[0005] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0006] According to one aspect of this disclosure, a data processing method is provided, applied at a decoding end, comprising: acquiring global parameters, the global parameters including a frame interpolation method; and performing frame interpolation based on the frame interpolation method.

[0007] In one embodiment of this disclosure, the global parameters further include a sampling rate and / or a sampling interval frame; performing frame interpolation based on an interpolation method includes performing frame interpolation based on the interpolation method and at least one of the sampling rate and the number of sampling interval frames.

[0008] In one embodiment of this disclosure, the number of sampling interval frames includes the number of sampling frames and the number of prediction frames.

[0009] In one embodiment of this disclosure, the sampling rate includes the sampling rate in the case of forward and backward frame interpolation.

[0010] In one embodiment of this disclosure, frame interpolation based on the interpolation method includes: in response to the interpolation method being backward interpolation, obtaining an initial reference frame sequence based on frame data in the bitstream, the initial reference frame sequence including sampled frames obtained based on the frame data in the bitstream; determining whether the number of reference frames in the initial reference frame sequence reaches a preset threshold, wherein the preset threshold is determined by the number of sampled frames; and if the number of reference frames in the initial reference frame sequence reaches the preset threshold, determining the current prediction frame based on the initial reference frame sequence using a prediction model.

[0011] In one embodiment of this disclosure, the frame interpolation based on the interpolation method further includes: updating the current prediction count and inputting the current predicted frame into the current reference frame sequence to obtain an updated current reference frame sequence; determining whether the current prediction count has reached a preset count threshold, the preset count threshold being obtained based on the number of predicted frames and the number of predicted frames obtained by the prediction model each time; if the current prediction count has not reached the preset count threshold, determining the current predicted frame based on the current reference frame sequence using the prediction model, and then returning to the step of updating the current prediction count.

[0012] In one embodiment of this disclosure, inputting the current predicted frame into the current reference frame sequence includes: appending the current predicted frame after the last reference frame in the current reference frame sequence; determining the current predicted frame based on the current reference frame sequence using the prediction model includes: inputting a preset number of reference frames at the end of the current reference frame sequence into the prediction model, and outputting the current predicted frame.

[0013] In one embodiment of this disclosure, the method further includes: obtaining frame-level parameters corresponding to the current frame, the frame-level parameters including a temporal parameter change flag; and modifying the global parameters according to the frame-level parameters when the temporal parameter change flag is in the position.

[0014] In one embodiment of this disclosure, the frame-level parameters further include a frame interpolation method; modifying the global parameters according to the frame-level parameters includes: changing the frame interpolation method in the global parameters from backward frame interpolation to forward and backward frame interpolation; or, changing the frame interpolation method in the global parameters from forward and backward frame interpolation to backward frame interpolation.

[0015] In one embodiment of this disclosure, when the frame interpolation method in the frame-level parameters is forward and backward interpolation, the frame-level parameters further include a sampling rate; modifying the global parameters according to the frame-level parameters further includes: modifying the sampling rate in the global parameters according to the sampling rate in the frame-level parameters to obtain modified global parameters; performing frame interpolation based on the interpolation method includes: performing frame interpolation on the current frame based on the interpolation method and sampling rate in the modified global parameters.

[0016] In one embodiment of this disclosure, when modifying the global parameters according to the frame-level parameters includes changing the interpolation method in the global parameters from forward interpolation to backward interpolation, performing interpolation based on the interpolation method includes: in response to interpolating the portion before the current frame, performing interpolation based on the current frame using the forward interpolation method; and in response to interpolating the portion after the current frame, performing interpolation based on the current frame using the backward interpolation method.

[0017] In one embodiment of this disclosure, frame interpolation based on the interpolation method includes: interpolating the current frame based on the interpolation method to obtain a plurality of predicted frames of the current frame; in response to a recovery data end marker in the bitstream data, obtaining the number of remaining frames corresponding to the current frame from the information corresponding to the recovery data end marker; in response to interpolating the current frame using a backward interpolation method, if the number of remaining frames is a first number greater than zero, retaining a first number of predicted frames from the plurality of predicted frames.

[0018] In one embodiment of this disclosure, in response to performing frame interpolation on the current frame using a backward interpolation method, the plurality of predicted frames are not retained when the number of remaining frames is zero.

[0019] In one embodiment of this disclosure, in response to performing frame interpolation on the current frame using a forward and backward interpolation method, if the number of remaining frames is a second number greater than zero, the current frame is copied to obtain a copied frame, and the second number of copied frames are retained.

[0020] In one embodiment of this disclosure, frame interpolation based on the interpolation method further includes: outputting the corresponding image output time according to the image sequence.

[0021] In one embodiment of this disclosure, the frame interpolation based on the interpolation method further includes: in response to the current frame being interpolated using the backward interpolation method, interpolating the current frame based on the backward interpolation method to obtain a plurality of predicted frames of the current frame, the current frame including the last sampled frame; and outputting a third number of predicted frames from the plurality of predicted frames according to the image sequence counting.

[0022] In one embodiment of this disclosure, the image sequence counting includes a target image sequence counting; frame interpolation based on the interpolation method further includes: in response to interpolating the current frame using the forward and backward interpolation method, interpolating the current frame based on the forward and backward interpolation method, the current frame including the last sampled frame; outputting the frames in the buffer according to the image sequence counting, wherein, if the bitstream does not contain the frame data corresponding to the target image sequence counting, the target image sequence counting corresponds to outputting the buffered frame corresponding to the previous image sequence counting in the buffer.

[0023] According to another aspect of this disclosure, a data processing method is provided, applied at an encoding end, comprising: encoding a video to obtain bitstream data, so that a decoding end can implement any of the data processing methods described in the above embodiments based on the bitstream data.

[0024] According to another aspect of this disclosure, a data processing apparatus is provided for use at a decoding end, comprising: a first acquisition module for acquiring global parameters, the global parameters including a frame interpolation method; and a first frame interpolation module for performing frame interpolation based on the frame interpolation method.

[0025] In one embodiment of this disclosure, the global parameters further include a sampling rate and / or a sampling interval frame; the first frame interpolation module is further configured to perform frame interpolation based on the frame interpolation method and at least one of the sampling rate and the number of sampling interval frames.

[0026] In one embodiment of this disclosure, the first frame interpolation module includes: an acquisition unit, configured to, in response to the frame interpolation method being backward frame interpolation, obtain an initial reference frame sequence based on frame data in the bitstream, the initial reference frame sequence including sampled frames obtained based on the frame data in the bitstream; a first determination unit, configured to determine whether the number of reference frames in the initial reference frame sequence reaches a preset threshold, wherein the preset threshold is determined by the number of sampled frames; and a second determination unit, configured to, if the number of reference frames in the initial reference frame sequence reaches the preset threshold, determine a current prediction frame based on the initial reference frame sequence using a prediction model.

[0027] In one embodiment of this disclosure, the apparatus further includes: a first input module, configured to update the current prediction count and input the current prediction frame into the current reference frame sequence to obtain an updated current reference frame sequence; the first frame interpolation module further includes: a third determining unit, configured to determine whether the current prediction count has reached a preset count threshold, the preset count threshold being obtained based on the number of prediction frames and the number of prediction frames obtained by the prediction model each time; and a fourth determining unit, configured to determine the current prediction frame based on the current reference frame sequence through the prediction model if the current prediction count has not reached the preset count threshold, and then return to the step of updating the current prediction count.

[0028] In one embodiment of this disclosure, the first input module is further configured to append the current prediction frame to the last reference frame in the current reference frame sequence; the fourth determining unit is further configured to input a preset number of reference frames at the end of the current reference frame sequence into the prediction model and output the current prediction frame.

[0029] In one embodiment of this disclosure, the apparatus further includes: a second acquisition module, configured to acquire frame-level parameters corresponding to the current frame, the frame-level parameters including a temporal parameter change flag; and a modification module, configured to modify the global parameters according to the frame-level parameters when the temporal parameter change flag is in position.

[0030] In one embodiment of this disclosure, the frame-level parameters further include a frame interpolation method; the modification module is further configured to modify the frame interpolation method in the global parameters from backward frame interpolation to forward and backward frame interpolation; or, to modify the frame interpolation method in the global parameters from forward and backward frame interpolation to backward frame interpolation.

[0031] In one embodiment of this disclosure, when the frame interpolation method in the frame-level parameters is forward and backward interpolation, the frame-level parameters further include a sampling rate; the modification module is further configured to modify the sampling rate in the global parameters according to the sampling rate in the frame-level parameters to obtain the modified global parameters; the first interpolation module is further configured to interpolate the current frame based on the interpolation method and sampling rate in the modified global parameters.

[0032] In one embodiment of this disclosure, the first frame interpolation module is further configured to perform frame interpolation based on the current frame using the forward and backward frame interpolation method in response to performing frame interpolation on the portion before the current frame; and to perform frame interpolation based on the current frame using the backward frame interpolation method in response to performing frame interpolation on the portion after the current frame.

[0033] In one embodiment of this disclosure, the first frame interpolation module is further configured to interpolate the current frame based on the frame interpolation method to obtain a plurality of predicted frames of the current frame; in response to the recovery data end marker in the bitstream data, to obtain the number of remaining frames corresponding to the current frame from the information corresponding to the recovery data end marker; and in response to interpolating the current frame using the backward frame interpolation method, if the number of remaining frames is a first number greater than zero, to retain a first number of predicted frames among the plurality of predicted frames.

[0034] In one embodiment of this disclosure, the first frame interpolation module is further configured to, in response to performing frame interpolation on the current frame using a backward frame interpolation method, not retain the plurality of predicted frames when the number of remaining frames is zero.

[0035] In one embodiment of this disclosure, the first frame interpolation module is further configured to, in response to performing frame interpolation on the current frame using a forward and backward frame interpolation method, copy the current frame to obtain a copy frame when the number of remaining frames is a second number greater than zero, and retain the second number of copy frames.

[0036] According to another aspect of this disclosure, a data processing apparatus is provided for use at an encoding end, comprising: an encoding module for encoding video to obtain bitstream data, so that a decoding end can implement any of the data processing methods described in the above embodiments based on the bitstream data.

[0037] According to another aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the above-described data processing method by executing the executable instructions.

[0038] According to another aspect of this disclosure, a computer-readable storage medium is provided that stores a computer program thereon, which, when executed by a processor, implements the above-described data processing method.

[0039] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0040] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0041] Figure 1 shows a flowchart of a data processing method according to an embodiment of the present disclosure.

[0042] Figure 2 shows a flowchart of another data processing method in an embodiment of this disclosure.

[0043] Figure 3 shows a flowchart of another data processing method according to an embodiment of the present disclosure.

[0044] Figure 4 illustrates an exemplary implementation of step S120 in Figure 1 in an exemplary embodiment.

[0045] Figure 5 illustrates an exemplary implementation of step S120 in Figure 1 in another exemplary embodiment.

[0046] Figure 6 illustrates an exemplary implementation of step S120 in Figure 1 in yet another exemplary embodiment.

[0047] Figure 7 shows a flowchart of another data processing method according to an embodiment of this disclosure.

[0048] Figure 8 illustrates an exemplary data processing method based on Figures 1 and 7.

[0049] Figure 9 illustrates another exemplary data processing method based on Figures 1 and 7.

[0050] Figure 10 illustrates a frame interpolation schematic diagram according to Figure 9, based on an embodiment of the present disclosure.

[0051] Figure 11 illustrates a frame interpolation method according to an exemplary embodiment.

[0052] Figure 12 illustrates another frame interpolation method according to an exemplary embodiment.

[0053] Figure 13 illustrates another frame interpolation method according to an exemplary embodiment.

[0054] Figure 14 shows a structural diagram of a data processing apparatus according to an embodiment of the present disclosure.

[0055] Figure 15 shows a structural diagram of a data processing apparatus according to an embodiment of the present disclosure.

[0056] Figure 16 shows a structural block diagram of an electronic device according to an embodiment of the present disclosure. Detailed Implementation

[0057] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that this disclosure will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0058] Furthermore, the accompanying drawings are merely illustrative of this disclosure and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0059] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0060] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0061] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0062] It should be noted that no parameters are limited in this disclosure.

[0063] This disclosure applies to image input and feature tensor input.

[0064] Current frame-drop encoding techniques typically extract frames from a video at a fixed ratio before encoding the extracted frames. However, with technological advancements, existing frame-dropping and interpolation methods are no longer sufficient to meet user needs. How to efficiently encode and decode video is a pressing issue that needs to be addressed in this field.

[0065] To address the aforementioned problems, embodiments of this disclosure provide a data processing method, apparatus, device, and storage medium that at least to some extent improve the efficiency of video encoding and decoding.

[0066] Figure 1 shows a flowchart of a data processing method according to an embodiment of the present disclosure. The method shown in Figure 1 can be applied to a decoding end. As shown in Figure 1, the method may include the following steps S110 to S120.

[0067] S110, obtain global parameters, including frame interpolation method.

[0068] In some embodiments, the method provided in this disclosure can be applied to process video data, such as encoding the video at the encoding end to obtain a video bitstream and transmitting it to the decoding end for decoding processing. The frame data in the bitstream received by the decoding end is the frame to be supplemented. Global parameters can be located in the bitstream.

[0069] In some embodiments, global parameters can be sequence set parameters.

[0070] In some embodiments, the global parameter can be srd_temporal_resampling_ratio_idx.

[0071] In some embodiments, the frame interpolation method can be TemporalRestorationMode. In some embodiments, TemporalRestorationMode[i] = prd_temporal_restoration_mode, that is, the frame interpolation method adopts the currently updated value. TemporalRestorationMode[i] = TemporalRestorationMode[i-1] indicates that the frame interpolation method does not change.

[0072] In some embodiments, the frame interpolation method can be backward frame interpolation or forward and backward frame interpolation. For example, the frame interpolation method can be prd_temporal_extrapolation_flag.

[0073] For example, the frame interpolation method can be srd_temporal_restoration_mode.

[0074] 0 indicates forward and backward prediction

[0075] 1 indicates backward prediction.

[0076] In some embodiments, global parameters may also include sampling rate and / or sampling interval frames.

[0077] In some embodiments, the number of sampling interval frames includes the number of sampled frames and the number of predicted frames.

[0078] In some embodiments, the sampling rate includes the sampling rate in the case of forward and backward prediction.

[0079] For example, the frame interpolation method can be the current temporal restoration mode; the sampling ratio can be the current temporal interpolation ratio; and the sampling interval frame number can be the current temporal extrapolation predict frames number or the current temporal extrapolation resampling frames number.

[0080] For example, the sampling ratio can be expressed as ResamplingRatio.

[0081] In some embodiments, the sampling ratio TemporalInterpolationRatio can be determined by the following formula:

[0082] TemporalInterpolationRatio=2^(srd_temporal_interpolation_ratio_idx+1)

[0083] The value of srd_temporal_interpolation_ratio_idx ranges from 0, 1, to 2.

[0084] In some embodiments, the number of sampling frames can be TemporalExtrapolationResampleNum.

[0085] For example, the sampling interval frame number can be represented as prd_temporal_extra_resampling_length_idx.

[0086] In some embodiments, the sampling interval frame number is used to represent the number and position of sampled frames when sampling is performed between multiple frames.

[0087] In some embodiments, the sampling interval frame number is used to represent the number and location of non-sampled frames when sampling is performed between multiple frames.

[0088] In some embodiments, the number of sampling frames, TemporalExtralationResampleNum, can be determined by the following formula:

[0089] Number of sampling frames = srd_temporal_extraplation_resample_num_idx + 2.

[0090] In some embodiments, when srd_temporal_restoration_mode = 0 in the global parameters, it indicates that the frame interpolation method is forward and backward frame interpolation, and the bitstream does not contain the frame interpolation parameter srd_temporal_extraplation_resample_num_idx for backward frame interpolation.

[0091] In some embodiments, the number of predicted frames can be TemporalExtrapolationPredictNum.

[0092] In some embodiments, the number of predicted frames, TemporalExtrapolationPredictNum, can be determined by the following formula:

[0093] TemporalExtrapolationPredictNum=srd_temporal_extrapolation_predict_num_idx+1

[0094] For example, the sampling interval frame number can be ResamplingLen, and the prediction frame number and sampling frame number can be RestorationLen.

[0095] In some embodiments, when srd_temporal_restoration_mode = 0, i.e., the interpolation mode is forward and backward interpolation, the bitstream does not contain srd_temporal_extrapolation_predict_num_idx. The value range of srd_temporal_extrapolation_predict_num_idx is 0, 1, and 2.

[0096] In some embodiments, when prd_temporal_resampling_ratio_changed = 1:

[0097] TemporalExtrapolationPredictNum[i]=prd_temporal_extrapolation_predict_num_idx

[0098] otherwise:

[0099] TemporalExtrapolationPredictNum[i]=TemporalExtrapolationPredictNum[i-1]

[0100] That is, the predicted frame number can be the historical predicted frame number, and the historical predicted frame number can be the previous predicted frame number.

[0101] In some embodiments, prd_temporal_extrapolation_predict_num_idx will not be in the bitstream when prd_temporal_resampling_ratio_changed = 0 or prd_temporal_restoration_mode = 0. The value range of prd_temporal_extrapolation_predict_num_idx is 0, 1, 2, and 3. prd_temporal_extrapolation_predict_num_idx = 0 indicates that temporal back-padding is currently disabled.

[0102] S120 performs frame interpolation based on the interpolation method.

[0103] In some embodiments, frame interpolation is performed based on a frame interpolation method, including:

[0104] Frame interpolation is performed based on at least one of the following: interpolation method, sampling rate, and sampling interval (number of frames).

[0105] Frame interpolation based on at least one of the interpolation method and sampling rate, and sampling interval frame number, can include:

[0106] Frame interpolation can be performed based on at least one of the interpolation method and the sampling rate.

[0107] Frame interpolation can be performed based on the interpolation method and the number of frames between sampling intervals.

[0108] In some embodiments, if the interpolation method is forward and backward interpolation, the predicted frame can be inserted between the current frame and the frames preceding the current frame. For example, the specific insertion position of the predicted frame can be determined based on global parameters, and this disclosure does not make any specific limitations.

[0109] In some embodiments, if the frame interpolation method is forward and backward frame interpolation, the predicted frame can be inserted between the current frame and the frames preceding the current frame. For example, the specific insertion position of the predicted frame can be determined based on the number of sampling interval frames, and this disclosure does not make any specific limitations.

[0110] In some embodiments, the images are output according to the image output time corresponding to the Picture Order Count (POC).

[0111] The data processing method provided in the embodiments of this disclosure obtains global parameters and performs frame interpolation based on the frame interpolation method in the global parameters, thereby improving the efficiency of video encoding and decoding.

[0112] Figure 2 shows a flowchart of another data processing method according to an embodiment of this disclosure. The connection between Figure 2 and Figure 1 is that step S210 in Figure 2 is an exemplary implementation of step S110 in Figure 1, and step S220 is an exemplary implementation of step S120 in Figure 1. As shown in Figure 2, the method may include the following steps S210 to S220.

[0113] S210, obtain the frame interpolation method and sampling rate from the global parameters. The frame interpolation method parameter value is forward and backward frame interpolation.

[0114] S220 uses a forward and backward frame interpolation method to perform frame interpolation based on the sampling rate.

[0115] Figure 3 shows a flowchart of another data processing method according to an embodiment of the present disclosure. Figure 3 is related to Figures 1 and 2 in that step S310 in Figure 3 is an exemplary implementation of step S110 in Figure 1, different from step S210, and step S330 is an exemplary implementation of step S120 in Figure 1, different from step S220. As shown in Figure 3, the method may include the following steps S310 to S320.

[0116] S310, obtain the frame interpolation method, number of sampled frames and number of predicted frames from the global parameters, and the frame interpolation method parameter value is backward frame interpolation.

[0117] The S330 uses a backward frame interpolation method to perform frame interpolation based on the number of sampled frames and the number of predicted frames.

[0118] Figure 4 illustrates an exemplary implementation of step S120 in Figure 1 in an exemplary embodiment. As shown in Figure 4, step S120 may include steps S410 to S430.

[0119] S410, in response to the frame interpolation method being backward frame interpolation, an initial reference frame sequence is obtained based on the frame data in the bitstream. The initial reference frame sequence includes the sampled frames obtained based on the frame data in the bitstream.

[0120] In some embodiments, the reference frame sequence may be a frameBuffer.

[0121] S420, determine whether the number of reference frames in the initial reference frame sequence has reached a preset threshold, wherein the preset threshold is determined by the number of sampled frames.

[0122] In some embodiments, after determining the number of sampling interval frames, the number of reference frames required to obtain the prediction frame can be determined, and the number of required reference frames is the preset threshold.

[0123] For example, the preset threshold can be M1.

[0124] S430, if the number of reference frames in the initial reference frame sequence reaches a preset threshold, the current prediction frame is determined by the prediction model based on the initial reference frame sequence.

[0125] In this embodiment, all reference frames in the reference frame sequence are sampling frames.

[0126] For example, a predicted frame can be obtained by making a prediction based on a video prediction model and a reference frame.

[0127] For example, a prediction frame can be obtained by using the extrapolation function and a reference frame.

[0128] In some embodiments, the current frame, reference frames in the reference sequence, and the number of sampling interval frames can be input into the prediction module to obtain the prediction frame.

[0129] In some embodiments, the reference sequence can be cleared when the interpolation ends or the interpolation method changes.

[0130] In some embodiments, the sampling interval frame number can be the prediction frame number TemporalExtrapolationPredictNum[i].

[0131] For example, when acquiring a new frame of data, if the acquisition is successful, the acquired frame data is pushed into the frameBuffer; otherwise, the remaining frame is processed, the temporal information of the current frame is acquired, and when prdTemChangedFlag is 1, the sequence-level parameter RestorationRatio is updated to the RestorationRatio corresponding to the current frame = curRestorationRatio = prd_temporal_resampling_ratio_idx. It is then checked whether the frameBuffer is full; if so, the frameBuffer and RestorationRatio are input to the extrapolation function to obtain RestorationRatio prediction frames. The sampled frame (frameBuffer) and prediction frames are saved sequentially. If the frameBuffer is empty, a new frame of data is acquired again.

[0132] The data processing method provided in the embodiments of this disclosure performs backward frame interpolation according to the frame interpolation method parameter value in the global parameters, thereby improving the efficiency of video encoding and decoding.

[0133] Figure 5 exemplarily illustrates an implementation of step S120 in Figure 1 in another exemplary embodiment. As shown in Figure 5, step S120 may also include the following steps S510 to S530. Figure 5 may be performed, for example, after Figure 4.

[0134] S510, update the current prediction count, and input the current prediction frame into the current reference frame sequence to obtain the updated current reference frame sequence.

[0135] In some embodiments, after determining the predicted frame, the current frame can be input into the reference frame sequence, and the number of frames in the reference frame sequence can be incremented by 1.

[0136] In some embodiments, if the number of frames in the reference frame sequence exceeds a second preset threshold after the current frame is input into the reference frame sequence, then the reference frame with the longest duration in the reference frame sequence is deleted. The second preset threshold may be the same as a preset threshold.

[0137] In some embodiments, after the predicted frame is determined, the insertion position of the predicted frame in the interpolated frame segment can be determined according to the interpolation method and global parameters.

[0138] In some embodiments, if the interpolation method is backward interpolation, the predicted frame can be inserted after the current frame.

[0139] In some embodiments, updating the reference frame sequence includes updating the reference sequence based solely on the generated prediction frames.

[0140] S520, determine whether the current prediction count has reached the preset count threshold. The preset count threshold is obtained based on the number of prediction frames and the number of prediction frames obtained by the prediction model each time.

[0141] S530: If the current prediction count has not reached the preset count threshold, the current prediction frame is determined by the prediction model based on the current reference frame sequence, and then the process returns to the step of updating the current prediction count.

[0142] In some embodiments, the reference frames in the current reference frame sequence may include sampled frames and predicted frames, wherein the predicted frames are frames that have been predicted.

[0143] In some embodiments, after updating the reference frame sequence, a predicted frame is obtained based on the updated reference frame sequence, and the step of obtaining a predicted frame based on the updated reference frame sequence is repeated until a preset number of times threshold is reached.

[0144] In some embodiments, the reference frame may include a sampled frame and / or a predicted frame.

[0145] Figure 6 illustrates an exemplary implementation of step S120 in Figure 1 in yet another exemplary embodiment. As shown in Figure 6, step S120 may further include steps S610 to S630. Figure 6 may be performed, for example, after Figure 4. The connection between Figure 6 and Figure 5 is that step S610 in Figure 6 is an exemplary implementation of step S510 in Figure 5, and step S630 is an exemplary implementation of step S530 in Figure 5.

[0146] S610, update the current prediction count, and append the current prediction frame to the last reference frame in the current reference frame sequence to obtain the updated current reference frame sequence.

[0147] S620, determine whether the current prediction count has reached the preset count threshold. The preset count threshold is obtained based on the number of prediction frames and the number of prediction frames obtained by the prediction model each time.

[0148] S630: If the current prediction count has not reached the preset count threshold, input the preset number of reference frames at the end of the current reference frame sequence into the prediction model, output the current prediction frame, and then return to the step of updating the current prediction count.

[0149] In some embodiments, after the number of repetitions reaches a preset number, all sampled frames and unsampled frames in the supplementary frame segment can be obtained.

[0150] For example, based on the sampling interval frame number RestorationRatio and the number of frames n predicted by the video prediction model, the video prediction module extrapolation(refList, RestorationRatio) is used, where refList is the image of the last m frames in the sampling and prediction frames. When RestorationRatio is 0, the module directly returns an empty list.

[0151] The reference frame must contain the sample frame.

[0152] Initially i = 0, execute the video prediction model len([0, floor(m / / n))) times:

[0153] The predicted frames returned by extrapolation are added to the prediction sequence queue preList;

[0154] Determine the length of preList and the size of RestorationRatio. If the length of preList is greater than or equal to RestorationRatio, return the last RestorationRatio frame of the predicted sequence.

[0155] When i == floor(m / / n), the extrapolation function returns (N - preList.size()) * preList[-1].

[0156] In some embodiments, the reference frame sequence is updated, and the predicted frame is obtained based on the updated reference frame sequence. The step of obtaining the predicted frame based on the updated reference frame sequence is repeated until the number of sampled frames and the number of predicted frames reach the total number of frames in the interpolation segment.

[0157] In some embodiments, updating the reference frame sequence includes updating the reference frame sequence based on sampled frames outside the current reference frame sequence and / or generated predicted frames.

[0158] In some embodiments, the preset quantity is related to the predictive model's capabilities.

[0159] For example, if the preset threshold is 2 and the number of sampled frames is 2, then 1 prediction frame will be obtained based on 2 sampled frames. Then, based on the 1 prediction frame and 1 sampled frame, the next prediction frame will be obtained. Then, based on 2 prediction frames, the next prediction frame will be obtained. Then, the prediction will continue to roll to obtain prediction frames of length N. The total number of frames in the supplementary frame segment is N+2.

[0160] The data processing method provided in the embodiments of this disclosure obtains global parameters and performs rolling backward frame interpolation according to the frame interpolation method parameter value in the global parameters, thereby improving the efficiency of video encoding and decoding.

[0161] Figure 7 shows a flowchart of another data processing method according to an embodiment of this disclosure. The connection between Figure 7 and Figure 1 is that Figure 7 can be used to modify global parameters based on the frame-level parameters of the current frame when processing the current frame during frame interpolation. The method shown in Figure 7 may include steps S710 to S730.

[0162] S710, obtain the frame-level parameters corresponding to the current frame, including the temporal parameter change flag.

[0163] In some embodiments, the frame-level parameter can be frame temporal information.

[0164] For example, the frame time domain information may include prdTemChangedFlag, and when prdTemChangedFlag is 1, the global parameters are updated.

[0165] In some embodiments, frame-level parameters corresponding to the current frame can be obtained. The frame-level parameters may include a temporal sampling ratio change flag, wherein the temporal sampling ratio change flag may be prd_temporal_resampling_ratio_changed.

[0166] prd_temporal_resampling_ratio_changed = 1 indicates the position of the flag indicating the change in the temporal sampling ratio.

[0167] prd_temporal_resampling_ratio_changed = 0 indicates that the time-domain sampling ratio change flag is not set.

[0168] In some embodiments, when prd_temporal_resampling_ratio_changed = 1, TemporalInterpolationRatio[i] is updated by the following formula:

[0169] TemporalInterpolationRatio[i]=2^(prd_temporal_interpolation_ratio_idx).

[0170] In some embodiments, when prd_temporal_resampling_ratio_changed = 0, TemporalInterpolationRatio[i] is updated by the following formula:

[0171] TemporalInterpolationRatio[i]=TemporalInterpolationRatio[i-1].

[0172] In some embodiments, when prd_temporal_resampling_ratio_changed = 0 or prd_temporal_restoration_mode = 1, prd_temporal_interpolation_ratio_idx will not be in the bitstream. The value range of prd_temporal_interpolation_ratio_idx is 0, 1, 2, and 3. When prd_temporal_interpolation_ratio_idx = 0, it indicates that temporal forward and backward frame interpolation is disabled.

[0173] In some embodiments, prd_temporal_interpolation_ratio_idx can represent the sampling ratio.

[0174] TemporalInterpolationRatio[i]=2^(prd_temporal_interpolation_ratio_idx).

[0175] TemporalInterpolationRatio[i]=TemporalInterpolationRatio[i-1].

[0176] In some embodiments, when prd_temporal_resampling_ratio_changed = 1:

[0177] TemporalExtrapolationResampleNum[i]=prd_temporal_extrapolation_resample_num_idx+2.

[0178] Among them, prd_temporal_extrapolation_resample_num_idx is a preset value, which takes the value 0 or 1.

[0179] otherwise:

[0180] TemporalExtrapolationResampleNum[i]=TemporalExtrapolationResampleNum[i-1]

[0181] In some embodiments, when prd_temporal_resampling_ratio_changed = 0 or prd_temporal_restoration_mode = 0, prd_temporal_extrapolation_resample_num_idx is not present in the bitstream. The value range of prd_temporal_extrapolation_resample_num_idx is 0 and 1.

[0182] S720 modifies global parameters based on frame-level parameters when the time-domain parameter change flag is set.

[0183] In some embodiments, global parameters can be updated based on frame-level parameters.

[0184] For example, the global parameters can be modified based on RestorationRatio = curRestorationRatio = prd_temporal_resampling_ratio_idx corresponding to the current frame.

[0185] In some embodiments, the time-domain sampling rate change flag is set when at least one of the following occurs: a change in sampling rate, a change in frame interpolation method, and a change in the number of frames in the sampling interval.

[0186] In some embodiments, when the temporal sampling rate change flag is set, the frame interpolation mode, TemporalRestorationMode[i], is updated. If the updated frame interpolation mode, TemporalRestorationMode[i] = 0, the sampling rate is also updated. That is, the sampling rate is updated for both forward and backward sampling. If the updated frame interpolation mode, TemporalRestorationMode[i] = 1, the number of sampling interval frames, namely the number of sampled frames, TemporalExtrapolationResampleNum[i], and the number of predicted frames, TemporalExtralationPredictNum[i], are also updated.

[0187] In some embodiments, global parameters can also be updated based on information contained in the bitstream data.

[0188] In some embodiments, modifying global parameters includes modifying the sampling rate to obtain a modified sampling rate.

[0189] According to the method provided in the embodiments of this disclosure, by setting the temporal parameter modification flag in the frame-level parameters and modifying the global parameters according to the frame-level parameters, different frame interpolation methods and / or different frame interpolation parameters can be applied to different frames, thereby improving the accuracy of video decoding.

[0190] Figure 8 illustrates an exemplary data processing method based on Figures 1 and 7. The method shown in Figure 8 may include the following steps S810 to S830.

[0191] S810 changes the frame interpolation method in the global parameters from backward frame interpolation to forward and backward frame interpolation.

[0192] S820 modifies the sampling rate in the global parameters according to the sampling rate in the frame-level parameters to obtain the modified global parameters.

[0193] S830 performs frame interpolation on the current frame based on the interpolation method and sampling rate in the modified global parameters.

[0194] Figure 9 illustrates another exemplary data processing method based on Figures 1 and 7. The method shown in Figure 9 may include the following steps S910 to S930.

[0195] S910 changes the frame interpolation method in the global parameters from forward and backward frame interpolation to backward frame interpolation.

[0196] S920 performs frame interpolation based on the portion preceding the current frame, using a forward and backward interpolation method based on the current frame.

[0197] S930 performs frame interpolation in response to the portion following the current frame, using a backward interpolation method based on the current frame.

[0198] In some embodiments, if the previous frame interpolation mode TemporalRestorationMode[i-1] is 0 and the current frame interpolation mode TemporalRestorationMode[i] is 1, then forward and backward frame interpolation is performed based on the current frame, followed by backward frame interpolation. To provide a detailed description of the data processing method in this embodiment, Figure 10 illustrates a frame interpolation schematic diagram according to Figure 9. As shown in Figure 10, the frame interpolation method used before the current frame is forward and backward frame interpolation. After the current frame serves as a reference frame for predicting the predicted frames before it, it then serves as a reference frame for predicting the predicted frames after it.

[0199] Figures 11 to 13 exemplarily illustrate several exemplary implementation methods after the last frame of the object to be framed is framed in step S120 of Figure 1.

[0200] Figure 11 illustrates a frame interpolation method according to an exemplary embodiment. The method shown in Figure 11 may include the following steps S1110 to S1130.

[0201] S1110, perform frame interpolation on the current frame based on the interpolation method to obtain multiple predicted frames of the current frame.

[0202] In this embodiment of the disclosure, the current frame can be the last sampled frame in the interpolation segment.

[0203] In some embodiments, the last sampled frame can be determined based on the number of sampled frames in the interpolation frame segment.

[0204] S1120, in response to the end-of-recovery data marker in the bitstream data, obtain the number of remaining frames corresponding to the current frame from the information corresponding to the end-of-recovery data marker.

[0205] S1130, in response to using a backward frame interpolation method to interpolate the current frame, if the number of remaining frames is a first number greater than zero, retain the first number of predicted frames among multiple predicted frames.

[0206] In some embodiments, prediction can be performed based on the current frame and the frames preceding the current frame as reference frames to obtain the predicted frame.

[0207] In some embodiments, erd_num_temporal_remain can be the remaining frames, i.e., the number of images after the last sampled frame.

[0208] In some embodiments, the number of remaining frames can be determined based on the bitstream data. This disclosure does not specifically limit the method for determining the number of remaining frames.

[0209] For example, if there are 3 predicted frames and 2 remaining frames, then the 2 predicted frames can be retained as remaining frames.

[0210] Figure 12 illustrates another frame interpolation method according to an exemplary embodiment. The method shown in Figure 12 may include the following steps S1210 to S1230.

[0211] S1210, perform frame interpolation on the current frame based on the interpolation method to obtain multiple predicted frames of the current frame.

[0212] S1220, in response to the end-of-recovery data marker in the bitstream data, obtain the number of remaining frames corresponding to the current frame from the information corresponding to the end-of-recovery data marker.

[0213] S1230, in response to using the backward interpolation method to interpolate the current frame, if the number of remaining frames is zero, multiple predicted frames are not retained.

[0214] Figure 13 illustrates another frame interpolation method according to an exemplary embodiment. The method shown in Figure 13 may include the following steps S1310 to S1330.

[0215] S1310, perform frame interpolation on the current frame based on the interpolation method to obtain multiple predicted frames of the current frame.

[0216] S1320, in response to the end-of-recovery data marker in the bitstream data, obtain the number of remaining frames corresponding to the current frame from the information corresponding to the end-of-recovery data marker.

[0217] S1330, in response to using forward and backward frame interpolation to interpolate the current frame, if the number of remaining frames is a second number greater than zero, the current frame is copied to obtain a copied frame, and the second number of copied frames are retained.

[0218] In some embodiments, the current frame can be a sampled frame. Since the current frame is the last sampled frame, forward and backward prediction cannot be completed. Therefore, the current frame can be copied according to the number of remaining frames to obtain the final remaining frames.

[0219] For example, if the encoding end finishes sampling during a sampling interval, the actual number of frames within the last sampling interval needs to be transmitted in the bitstream. `erd_temporal_restoration_data()` is the header data following the last payload data in the V3C framework.

[0220] erd_num_temporal_remain:

[0221] This indicates the number of images after the last temporal sampling frame.

[0222] When srd_temporal_restoration_mode is 0, erd_num_temporal_remain ranges from [0, 2srd_temporal_resampling_ratio_idx+1).

[0223] When srd_temporal_restoration_mode is 1, erd_num_temporal_remain ranges from [0, RestorationRatio);

[0224] The video prediction model is capable of predicting 2 consecutive frames out of 4 consecutive frames, where the TemporalRatio is 6 / 4. The encoder finishes sampling precisely during the sampling interval, which is 1 frame. `erd_temporal_restoration_data()` data needs to be sent after the final payload data, with `erd_num_temporal_remain` set to 1. Upon receiving the last decoded sampled subsequence, the decoder calls the `extrapolation` function to generate 2 predicted frames. When `erd_temporal_restoration_data()` data is received and `erd_num_temporal_remain` is found to be 1, only the first predicted frame is retained.

[0225] Specifically, when `srd_temporal_restoration_mode` is 1, the encoding end finishes sampling precisely when frame extraction ends. Therefore, `erd_temporal_restoration_data()` data needs to be sent after the final payload data, with `erd_num_temporal_remain` set to 0. When the decoding end receives the last decoded sampled subsequence, it calls the `extrapolation` function to generate two prediction frames. If `erd_temporal_restoration_data()` data is received and parsed to show `erd_num_temporal_remain` as 0, no prediction frames are displayed.

[0226] For example, `prd_temporal_resampling_ratio_changed_flag`: a flag indicating a change in the temporal sampling ratio.

[0227] 1: Changes have occurred

[0228] 0: No change

[0229] Any change in prd_temporal_resampling_ratio_idx, rd_temporal_extrapolation_flag, or prd_temporal_extra_resampling_length_idx is considered a change in the temporal sampling ratio.

[0230] prd_temporal_resampling_ratio_idx indicates a change in the sampling ratio.

[0231] prd_temporal_extrapolation_flag indicates that the sampling method has changed.

[0232] prd_temporal_extra_resampling_length_idx will cause changes in the number of frames after the frame skipping.

[0233] prd_temporal_extra_resampling_length_idx: Index of the frame-level temporal extrapolation sampling frame length

[0234] If prd_temporal_extra_resampling_length is represented by M, then

[0235] prd_temporal_extra_resampling_length_idx=M-2

[0236] When the mode is extrapolation, prd_temporal_resampling_ratio_idx is represented by i.

[0237] prd_temporal_resampling_ratio=(M+i) / M

[0238] When the mode is interpolation, prd_temporal_resampling_ratio_idx is represented by i.

[0239] prd_temporal_resampling_ratio=2i.

[0240] The data processing method provided in the embodiments of this disclosure obtains global parameters and performs frame interpolation based on the frame interpolation method in the global parameters, thereby improving the efficiency of video encoding and decoding.

[0241] In some embodiments, when predicting the current frame including the last sampled frame, the corresponding frame interpolation method for the current frame can be determined. If the frame interpolation method is backward frame interpolation, the current frame is interpolated using the backward frame interpolation method to obtain a predicted frame, and the predicted frames are output according to the image sequence count. For predicted frames whose image sequence count is after the last sampled frame, the third number of predicted frames among multiple predicted frames can be placed into the buffer for output according to the image sequence count. If the third number is less than the actual number of predicted frames obtained by frame interpolation (i.e., the total number of multiple predicted frames obtained by predicting with the reference frame including the last sampled frame), the remaining buffered frames can be removed. That is, for predicted frames in the buffer after receiving the bitstream end signal, these frames can be cleared from the output buffer.

[0242] In other embodiments, when predicting the current frame, including the last sampled frame, the corresponding frame interpolation method can be determined. If the frame interpolation method is forward and backward interpolation, the current frame is interpolated using the forward and backward interpolation method to obtain the predicted frame, and the predicted frame is output according to the image sequence count. For the last sampled frame, if forward and backward interpolation is used, no predicted frame with an image sequence count following it will be obtained. Therefore, if the last sampled frame is not the last frame to be output, when outputting according to the image sequence count, the frame buffer corresponding to the target image sequence count after the image sequence count of the last sampled frame in the buffer is empty. In this case, the buffered frame corresponding to the image sequence count before the target image sequence count in the buffer can be output, which is the last sampled frame.

[0243] Figure 14 shows a structural diagram of a data processing device according to an embodiment of this disclosure. It should be noted that this data processing device is applied at the decoding end.

[0244] As shown in Figure 14, the device may include:

[0245] The first acquisition module 1410 is used to acquire global parameters, including the frame interpolation method;

[0246] The first frame interpolation module 1420 is used for frame interpolation based on the frame interpolation method.

[0247] In some embodiments, the global parameters also include the sampling rate and / or the sampling interval frame;

[0248] The first frame interpolation module is also used to perform frame interpolation based on at least one of the frame interpolation method and the sampling multiplier and the number of sampling interval frames.

[0249] In some embodiments, the first frame interpolation module includes:

[0250] The acquisition unit is used to acquire the current frame and the reference frame sequence;

[0251] The first determining unit is used to determine whether the number of frames in the reference frame sequence reaches a preset threshold, wherein the preset threshold is determined by the number of frames in the sampling interval;

[0252] The second determining unit is used to determine the prediction frame based on the current frame, the reference frames in the reference frame sequence, the number of sampling interval frames, and the prediction module when the number of frames in the reference frame sequence reaches a preset threshold.

[0253] In some embodiments, the apparatus further includes:

[0254] The first input module is used to input the current frame into the reference frame sequence after the prediction frame is determined.

[0255] In some embodiments, the first frame interpolation module includes:

[0256] The input unit is used to input the current frame into the reference frame sequence;

[0257] The third determining unit is used to determine whether the number of frames in the reference frame sequence reaches a preset threshold, wherein the preset threshold is determined by the number of frames in the sampling interval;

[0258] The fourth determining unit is used to determine the prediction frame based on the reference frames in the reference frame sequence, the number of sampling interval frames, and the prediction module when the number of frames in the reference frame sequence reaches a preset threshold.

[0259] In some embodiments, the current frame includes a sampled frame and a predicted frame, and the apparatus further includes:

[0260] The second determining module is used to determine the number of predictions based on at least one of the number of sampled frames in the frame interpolation segment, the total number of frames in the frame interpolation segment, the number of sampling interval frames, and the prediction capability of the prediction model. The prediction capability of the prediction model includes the number of prediction frames obtained by the prediction model each time it makes a prediction.

[0261] The second input module is used to input the sampled frame into the reference frame sequence;

[0262] The third determining module is used to determine whether the number of sampled frames in the reference frame sequence has reached a preset threshold.

[0263] The fourth determining module is used to obtain the predicted frame when the number of sampled frames in the reference frame sequence reaches a preset threshold.

[0264] The first repeating module is used to repeat the above steps until the preset number of times is reached.

[0265] In some embodiments, the current frame includes a sampled frame and a predicted frame, and the apparatus further includes:

[0266] The third input module is used to input the sampled frame and the predicted frame into the reference frame sequence;

[0267] The fifth determining module is used to determine whether the number of sampled frames in the reference frame sequence has reached a preset threshold.

[0268] The sixth determining module is used to obtain the predicted frame when the number of sampled frames and predicted frames in the reference frame sequence reaches a preset threshold.

[0269] The second repeating module is used to repeat the above steps until the number of sampled frames and predicted frames reaches the total number of frames in the interpolation segment.

[0270] In some embodiments, the apparatus further includes:

[0271] The second acquisition module is used to acquire the frame-level parameters corresponding to the current frame, including the temporal sampling rate change flag.

[0272] The modification module is used to modify global parameters when the frame-level parameter flag is set.

[0273] In some embodiments, modifying global parameters includes modifying the sampling rate to obtain a modified sampling rate;

[0274] The device also includes:

[0275] The second frame interpolation module is used to perform frame interpolation based on the modified sampling rate and frame interpolation method.

[0276] In some embodiments, the modification module includes:

[0277] The modification unit is used to change the frame interpolation method from backward frame interpolation to forward and backward frame interpolation, and to change the frame interpolation method from forward and backward frame interpolation to backward frame interpolation.

[0278] In some embodiments, the apparatus further includes:

[0279] The third frame interpolation module is used to perform frame interpolation based on the modified global parameters after the global parameters have been modified.

[0280] In some embodiments, the modification module includes:

[0281] The interpolation unit is used to perform both forward and backward interpolation based on the current frame when the interpolation method is changed from forward interpolation to backward interpolation.

[0282] In some embodiments, the apparatus further includes:

[0283] The seventh determination module is used to determine the frame interpolation method corresponding to the current frame and the number of remaining frames. The current frame includes the last sampled frame.

[0284] The first retention module is used to obtain the predicted frame by backward interpolation when the interpolation method is backward interpolation, and to retain the predicted frame of the remaining frame number.

[0285] The second retention module, when the frame interpolation method is forward and backward frame interpolation, copies the current frame to obtain a copied frame and retains the number of copied frames remaining.

[0286] Figure 15 shows a structural diagram of a data processing device according to an embodiment of this disclosure. It should be noted that this data processing device is applied to the encoding end.

[0287] As shown in Figure 15, the device 1500 may include:

[0288] The encoding module 1510 is used to encode the video to obtain bitstream data, so that the decoding end can implement any of the data processing methods in the above embodiments based on the bitstream data.

[0289] The data processing apparatus provided in this disclosure can be used to execute the positioning methods provided in the above-described method embodiments. The implementation principle and technical effect are similar, and for the sake of simplicity, they will not be described in detail here.

[0290] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0291] The electronic device 1600 according to this embodiment of the present disclosure will now be described with reference to FIG16. The electronic device 1600 shown in FIG16 is merely an example and should not be construed as limiting the functionality and scope of the embodiments of the present disclosure.

[0292] As shown in Figure 16, the electronic device 1600 is presented in the form of a general-purpose computing device. The components of the electronic device 1600 may include, but are not limited to: at least one processing unit 1616, at least one storage unit 1620, and a bus 1630 connecting different system components (including storage unit 1620 and processing unit 1612).

[0293] The storage unit stores program code, which can be executed by the processing unit 1612 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 1612 can perform the following steps of the above method embodiments:

[0294] Get global parameters, including the frame interpolation method;

[0295] Frame interpolation is performed based on the interpolation method.

[0296] Storage unit 1620 may include readable media in the form of volatile storage units, such as random access memory (RAM) 16201 and / or cache memory 16202, and may further include read-only memory (ROM) 16203.

[0297] Storage unit 1620 may also include a program / utility 16204 having a set (at least one) program module 16205, such program module 16205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0298] Bus 1630 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0299] Electronic device 1600 can also communicate with one or more external devices 1640 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 1600, and / or any device that enables electronic device 1600 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1650. Furthermore, electronic device 1600 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1660. As shown, network adapter 1660 communicates with other modules of electronic device 1600 via bus 1630. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0300] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0301] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, which may be a readable signal medium or a readable storage medium. A program product capable of implementing the methods described above is stored thereon. In some possible implementations, various aspects of this disclosure may also be implemented as a program product including program code, which, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0302] More specific examples of computer-readable storage media in this disclosure may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0303] In this disclosure, a computer-readable storage medium may include a data signal propagated in baseband or as part of a carrier wave, wherein readable program code is carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of transmitting, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device.

[0304] Optionally, the program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof.

[0305] In practical implementation, program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0306] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0307] Furthermore, although the steps of the method in this disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in that specific order, or that all the steps shown must be performed to achieve the desired result. Additional or alternative steps may be omitted, multiple steps may be combined into one step, and / or a step may be broken down into multiple steps.

[0308] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0309] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the appended claims.

Claims

1. A data processing method, wherein, Applied to the decoding end, including: Obtain global parameters, including the frame interpolation method; Frame interpolation is performed based on the aforementioned interpolation method.

2. The method according to claim 1, wherein, The global parameters also include the sampling rate and / or the number of frames per sampling interval; The frame interpolation based on the interpolation method includes: Frame interpolation is performed based on the aforementioned interpolation method and at least one of the sampling rate and the number of frames per sampling interval.

3. The method according to claim 2, wherein, The sampling interval frame number includes the number of sampling frames and the number of prediction frames.

4. The method according to claim 2, wherein, The sampling rate includes the sampling rate in the case of forward and backward frame interpolation.

5. The method according to claim 3, wherein, The frame interpolation based on the interpolation method includes: In response to the frame interpolation method being backward frame interpolation, an initial reference frame sequence is obtained based on the frame data in the bitstream, the initial reference frame sequence including sampled frames obtained based on the frame data in the bitstream; Determine whether the number of reference frames in the initial reference frame sequence reaches a preset threshold, wherein the preset threshold is determined by the number of sampled frames; If the number of reference frames in the initial reference frame sequence reaches the preset threshold, the current prediction frame is determined by the prediction model based on the initial reference frame sequence.

6. The method according to claim 5, wherein, The frame interpolation based on the aforementioned interpolation method further includes: Update the current prediction count and input the current prediction frame into the current reference frame sequence to obtain the updated current reference frame sequence; Determine whether the current prediction count has reached a preset count threshold, which is obtained based on the number of prediction frames and the number of prediction frames obtained by the prediction model each time it makes a prediction. If the current prediction count does not reach the preset threshold, the current prediction frame is determined by the prediction model based on the current reference frame sequence, and then the process returns to the step of updating the current prediction count.

7. The method according to claim 6, wherein, Input the current predicted frame into the current reference frame sequence, including: The current predicted frame is appended after the last reference frame in the current reference frame sequence; The prediction model determines the current prediction frame based on the current reference frame sequence, including: Input a preset number of reference frames at the end of the current reference frame sequence into the prediction model, and output the current prediction frame.

8. The method according to any one of claims 1 to 7, wherein, The method further includes: Obtain the frame-level parameters corresponding to the current frame, including the temporal parameter change flag bit; When the time-domain parameter change flag is set, the global parameter is modified according to the frame-level parameter.

9. The method according to claim 8, wherein, The frame-level parameters also include the frame interpolation method; Modifying the global parameters based on the frame-level parameters includes: Change the frame interpolation method in the global parameters from backward frame interpolation to forward and backward frame interpolation; or... Change the frame interpolation method in the global parameters from forward and backward frame interpolation to backward frame interpolation.

10. The method according to claim 9, wherein, When the frame interpolation method in the frame-level parameters is forward and backward interpolation, the frame-level parameters also include the sampling rate; Modifying the global parameters based on the frame-level parameters further includes: The sampling rate in the global parameters is modified according to the sampling rate in the frame-level parameters to obtain the modified global parameters; Frame interpolation based on the aforementioned interpolation method includes: The current frame is interpolated based on the interpolation method and sampling rate in the modified global parameters.

11. The method according to claim 9, wherein, When modifying the global parameters according to the frame-level parameters, including changing the frame interpolation method in the global parameters from forward-backward frame interpolation to backward frame interpolation, performing frame interpolation based on the frame interpolation method includes: In response to performing frame interpolation on the portion preceding the current frame, frame interpolation is performed based on the current frame using the forward and backward interpolation method described above; In response to performing frame interpolation on the portion following the current frame, frame interpolation is performed based on the current frame using the backward interpolation method.

12. The method according to any one of claims 1 to 11, wherein, Frame interpolation based on the aforementioned interpolation method includes: The current frame is interpolated using the interpolation method to obtain multiple predicted frames of the current frame; In response to the end-of-recovery data marker in the bitstream data, the number of remaining frames corresponding to the current frame is obtained from the information corresponding to the end-of-recovery data marker; In response to performing frame interpolation on the current frame using a backward interpolation method, if the number of remaining frames is a first number greater than zero, a first number of predicted frames are retained among the plurality of predicted frames.

13. The method according to claim 12, wherein, Frame interpolation based on the aforementioned interpolation method further includes: In response to performing frame interpolation on the current frame using a backward interpolation method, if the number of remaining frames is zero, the multiple predicted frames are not retained.

14. The method according to claim 12 or the method thereof, wherein, Frame interpolation based on the aforementioned interpolation method further includes: In response to performing frame interpolation on the current frame using a forward and backward interpolation method, if the number of remaining frames is a second number greater than zero, the current frame is copied to obtain a copied frame, and the second number of copied frames are retained.

15. The method according to any one of claims 1 to 11, wherein, Frame interpolation based on the aforementioned interpolation method further includes: The images are output according to their sequential order and corresponding output times.

16. The method according to claim 15, wherein, Frame interpolation based on the aforementioned interpolation method further includes: In response to using a backward interpolation method to interpolate the current frame, the current frame is interpolated based on the backward interpolation method to obtain multiple predicted frames of the current frame, wherein the current frame includes the last sampled frame; The third number of predicted frames is output according to the order of the images.

17. The method according to claim 15, wherein, The image sequence counting includes the target image sequence counting; Frame interpolation based on the aforementioned interpolation method further includes: In response to performing frame interpolation on the current frame using a forward and backward interpolation method, the current frame is interpolated based on the forward and backward interpolation method, and the current frame includes the last sampled frame; The frames in the buffer are output according to the image sequence count. If the bitstream does not contain the frame data corresponding to the target image sequence count, the target image sequence count corresponds to the buffer frame corresponding to the previous image sequence count in the buffer.

18. A data processing method, wherein, Applied to the encoding end, including: The video is encoded to obtain bitstream data, so that the decoding end can implement any one of the data processing methods as described in claims 1-17 based on the bitstream data.

19. A data processing apparatus, wherein, Applied to the decoding end, including: The first acquisition module is used to acquire global parameters, including frame interpolation methods; The first frame interpolation module is used to perform frame interpolation based on the frame interpolation method.

20. A data processing apparatus, wherein, Applied to the encoding end, including: The encoding module is used to encode the video to obtain bitstream data, so that the decoding end can implement any one of the data processing methods as described in claims 1-17 based on the bitstream data.

21. An electronic device, wherein, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to execute the data processing method according to any one of claims 1 to 18 by executing the executable instructions.

22. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by the processor, it implements the data processing method according to any one of claims 1 to 18.

Images