Video processing method and apparatus, device, medium and program product

Abstract

Provided in the embodiments of the present disclosure are a video processing method and apparatus, a device, a medium, and a program product. The method comprises: determining whether a current frame of a video is an intra-predicted frame or an inter-predicted frame; and in response to determining that the current frame is an intra-predicted frame, and on the basis of the number of bits allocated to the current frame, using a first model to determine quantization parameters for the current frame, parameters of the first model being determined on the basis of hidden features of the current frame; or in response to determining that the current frame is an inter-predicted frame, and on the basis of the number of bits allocated to the current frame, using a second model to determine the quantization parameters for the current frame, the second model being different from the first model.

Description

Methods, apparatus, devices, media, and programs for video processing Technical Field

[0001] Examples of this disclosure generally relate to the field of video computing, and more specifically to methods, apparatus, devices, media, and program products for video processing. Background Technology

[0002] In recent years, bitrate control in neural video coding has become a hot research area. However, existing bitrate control methods in neural network video coding are relatively simple, ignoring the actual bitrate distortion characteristics of neural network video coding. This makes them unsuitable for the characteristics of different video content, easily leading to problems such as large bitrate control errors and unstable image quality. Therefore, how to achieve accurate and efficient bitrate control and improve the practicality and stability of neural network video coding has become an urgent problem to be solved. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for video processing is provided. The method includes: determining whether a current frame of a video is an intra-predicted frame or an inter-predicted frame; and, in response to determining that the current frame is an intra-predicted frame, determining quantization parameters for the current frame using a first model based on the number of bits allocated to the current frame, the parameters of the first model being determined based on latent features of the current frame; or, in response to determining that the current frame is an inter-predicted frame, determining quantization parameters for the current frame using a second model based on the number of bits allocated to the current frame, the second model being different from the first model.

[0004] In a second aspect of this disclosure, an apparatus for video processing is provided. The apparatus includes a determination module and a determination module. The determination module is configured to: determine whether the current frame of a video is an intra-predicted frame or an inter-predicted frame. The determination module is configured to: in response to determining that the current frame is an intra-predicted frame, determine quantization parameters for the current frame based on the number of bits allocated to the current frame using a first model, the parameters of the first model being determined based on latent features of the current frame; or in response to determining that the current frame is an inter-predicted frame, determine quantization parameters for the current frame based on the number of bits allocated to the current frame using a second model, the second model being different from the first model.

[0005] In a third aspect of this disclosure, an electronic device is provided. The electronic device includes: at least one processor; and at least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to a first aspect of this disclosure when executed by the at least one processor.

[0006] In a fourth aspect of this disclosure, a computer-readable storage medium is provided having instructions stored thereon that, when executed by a processor, cause the processor to implement the method according to a first aspect of this disclosure.

[0007] In a fifth aspect of this disclosure, a computer program product is provided. The computer program product is tangibly stored in a computer storage medium and includes computer-executable instructions that, when executed by a device, cause the device to perform the method according to a first aspect of this disclosure.

[0008] The present invention is provided to present, in a simplified form, the selection of concepts further described below in the detailed description. The present invention is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0009] The above and other objects, features, and advantages of this disclosure will become more apparent from the following detailed description with reference to the accompanying drawings. In the examples of this disclosure, the same reference numerals generally refer to the same components.

[0010] Figure 1 shows a block diagram illustrating an example video codec system according to some examples of this disclosure;

[0011] Figure 2 shows a flowchart of a method according to some examples of this disclosure;

[0012] Figure 3 shows the network structure of intra-frame prediction image RQ and RD model predictors according to some examples of this disclosure;

[0013] Figure 4 shows a schematic diagram of an intra-frame predictive image bitrate control method according to some examples of the present disclosure;

[0014] Figure 5 shows a schematic diagram of an inter-frame predictive image bitrate control method according to some examples of the present disclosure;

[0015] Figure 6 shows a schematic diagram of an RD-based bit allocation method according to some examples of this disclosure;

[0016] Figure 7 shows a flowchart of a method for video processing according to some examples of this disclosure;

[0017] Figure 8 shows a block diagram of an example apparatus for video processing according to some examples of this disclosure; and

[0018] Figure 9 shows a block diagram of a device in which one or more examples of the present disclosure may be implemented.

[0019] Throughout all the accompanying figures, the same or similar reference numerals generally refer to the same or similar elements. Detailed Implementation

[0020] The principles of this disclosure will now be described with reference to some examples. It should be understood that these examples are described for illustrative purposes only and to help those skilled in the art to understand and implement this disclosure, and do not imply any limitation on the scope of this disclosure. In addition to the methods described below, the disclosure described herein can be implemented in various other ways.

[0021] In the following description and claims, unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0022] The terms "an example," "example," "example scenario," etc., used in this disclosure indicate that the example described may include a specific feature, structure, or characteristic, but not every example must include that specific feature, structure, or characteristic. Furthermore, these phrases do not necessarily refer to the same example. Moreover, when a specific feature, structure, or characteristic is described in conjunction with an example, it is claimed that, whether explicitly described or not, such a feature, structure, or characteristic affecting its relation to other examples is within the knowledge of a person skilled in the art.

[0023] It should be understood that although the terms “first” and “second”, etc., may be used herein to describe various elements, these elements should not be limited to these terms. These terms are used only to distinguish one element from another. For example, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element, without departing from the scope of the example. As used herein, the term “and / or” includes any and all combinations of one or more of the listed terms.

[0024] The terminology used herein is for the purpose of describing a particular example only and is not intended to limit the examples. As used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising,” “including,” “having,” “having,” “containing,” and / or “comprising” as used herein indicate the presence of the stated feature, element, and / or component, but do not preclude the presence or addition of one or more other features, elements, components, and / or combinations thereof. Example Context

[0025] Figure 1 is a block diagram illustrating an example video encoding / decoding system 100 that can utilize the techniques of this disclosure. As shown, the video encoding / decoding system 100 may include a source device 110 and a destination device 120. The source device 110 may also be referred to as a video encoding device, and the destination device 120 may also be referred to as a video decoding device. In operation, the source device 110 may be configured to generate encoded video data, and the destination device 120 may be configured to decode the encoded video data generated by the source device 110. The source device 110 may include a video source 112, a video encoder 114, and an input / output (I / O) interface 116.

[0026] Video source 112 may include sources such as video capture devices. Examples of video capture devices include, but are not limited to, interfaces for receiving video data from video content providers, computer graphics systems for generating video data, and / or combinations thereof.

[0027] Video data may include one or more images. Video encoder 114 encodes the video data from video source 112 to generate a bitstream. The bitstream may include a sequence of bits forming an encoded representation of the video data. The bitstream may include encoded images and associated data. An encoded image is an encoded representation of an image. Associated data may include sequence parameter sets, image parameter sets, and other syntax structures. I / O interface 116 may include a modulator / demodulator and / or a transmitter. Encoded video data can be directly transmitted to destination device 120 via network 130A through I / O interface 116. Encoded video data may also be stored on storage medium / server 130B for access by destination device 120.

[0028] The destination device 120 may include an I / O interface 126, a video decoder 124, and a display device 122. The I / O interface 126 may include a receiver and / or a modem. The I / O interface 126 may acquire encoded video data from the source device 110 or the storage medium / server 130B. The video decoder 124 may decode the encoded video data. The display device 122 may display the decoded video data to a user. The display device 122 may be integrated with the destination device 120, or it may be external to the destination device 120, which is configured to interface with an external display device.

[0029] The video encoder 114 and video decoder 124 can be implemented using neural networks. In some cases, the video encoder 114 and video decoder 124 can operate according to neural network-based video compression standards.

[0030] In recent years, with low-latency configurations, the coding performance of neural network video coding has surpassed that of traditional video coding. A limited number of studies have begun to investigate bitrate control methods for neural network video coding. Among these methods, some attempt to use empirical formulas to adjust the λ or quantization parameters in neural network video coding. Some methods use simple buffer-based strategies to dynamically adjust the quantization parameters. Some methods train the neural network to directly adjust the λ parameter in neural network video coding.

[0031] Previous research did not consider the actual rate-distortion characteristics of neural network video coding. Some methods used simple rules or empirical formulas, which could not handle various video content with different rate-distortion characteristics. Furthermore, some methods relied solely on training the neural network without considering and utilizing prior information about neural network video coding, making it difficult for the neural network to learn accurate rate control methods. These methods also lack interpretability and scalability. Moreover, previous research did not distinguish between intra-frame prediction images and inter-frame prediction images. Since inter-frame prediction images can utilize temporal information to aid compression, intra-frame and inter-frame prediction images have completely different rate-distortion characteristics. These methods did not take this into account, resulting in significant rate errors. Additionally, past research only focused on neural network video coding methods with a small range of quality variation. Because previous neural network video coding had a small rate variation range, its quality variation range was narrower compared to traditional video coding. This resulted in the lack of effectiveness of rate control methods based on these neural network video coding methods. Example video coding workflow.

[0032] Figure 2 illustrates a flowchart of a method according to some examples of this disclosure. The RD-based bit allocation method first allocates bits to the intra-predictive image and the inter-predictive image within an intra-frame period. Subsequently, the proposed rate control method for the intra-predictive image and the inter-predictive image calculates the quantization parameters corresponding to the image based on the allocated bits. Finally, after encoding a frame, the encoding information of the encoded frame is used to derive the RQ and RD models of the inter-predictive image for subsequent frames.

[0033] This paper discloses an intra-frame prediction image bitrate control method. Two models are established for intra-frame prediction images: RQ model and RD model. The RQ model represents the relationship between bitrate and quantization parameters, constructed by a logarithmic function, as shown in the following equation: QP=αln(βR) (1) QP represents the quantization parameter, R represents the bitrate, and α and β are model parameters. The RD model represents the relationship between bitrate and distortion, constructed by a power function, as shown in the following equation: D=γR ∈(2) D represents distortion, and γ, ε are model parameters.

[0034] For model parameters α, β, γ, ε, this disclosure establishes an intra-frame prediction image RQ and RD model predictor, specifically implemented by a designed neural network. The structure diagram of the neural network is shown in Figure 3, which illustrates the network structure diagram of the intra-frame prediction image RQ and RD model predictor according to some examples of this disclosure. The input to the neural network uses high-dimensional features obtained before video coding quantization. The model parameter prediction is expressed by the following equation.

[0035] Therefore, the intra-frame prediction image bitrate control method, as shown in Figure 4, derives the RQ model through high-dimensional features and an intra-frame prediction image RQ model predictor, and then calculates the corresponding quantization parameters based on the derived RQ model and the target number of bits. In this paper, the target number of bits can also be simply referred to as the target bit.

[0036] This paper discloses a method for inter-frame prediction image bitrate control. Two models are established for inter-frame prediction images: the RQ model and the RD model. The RQ model represents the relationship between bitrate and quantization parameters, constructed from a logarithmic function, as shown in the following equation: R int α represents the intersection of the low bitrate and high bitrate models. l ,β l and α h ,β h These represent the model parameters in the low bitrate and high bitrate segments, respectively. The RD model for inter-frame predicted images is consistent with that for intra-frame predicted images.

[0037] For the model parameters α, β, γ, ε, this disclosure proposes a gradient descent-based RQ and RD model update method to obtain the model parameters of the inter-frame predicted image. The RQ and RD models are updated using the actual encoded information after compression of the previous frame for encoding the next frame. The update method is as follows. α, β, γ, ε are the model parameters of the previous frame, BPP a MSE a QP a η represents the actual bitrate, actual distortion, and actual quantization parameters of the previous frame. R-Q ,η R-D This refers to the learning rate for the RQ and RD models. The model update method needs to select the appropriate update model (low bitrate or high bitrate) based on the actual target bitrate of a frame.

[0038] Therefore, the inter-frame predictive image rate control method, as shown in Figure 5, determines the corresponding RQ model based on the target bit size and then calculates the corresponding quantization parameters. After encoding a frame, its corresponding encoding information is used to update the RQ and RD models for rate control in the next frame.

[0039] This paper also discloses a bit allocation method based on RD (Reference-Oriented Model). Based on the derived RD models of intra-frame and inter-frame predicted images, a bit allocation optimization problem is established to rationally allocate the corresponding bits B_I and B_P of the intra-frame and inter-frame predicted images, as shown below. The following constraints must be met: B I +(N ip -1)·B P =B ip , and Where p0 and p1 are the MSE relationship parameters between the inter-frame predicted image and its reference image, and N ip B represents the number of frames in one intra-frame period. ip The number of bits allocated to the current intra-frame period is represented by the following formula: N r B represents the number of frames remaining to be encoded. t B represents the target number of bits. c This indicates the number of bits consumed.

[0040] Furthermore, during subsequent encoding, the target bits of the inter-frame predicted image need to be updated based on the actual remaining bits, as shown in the following formula. B ip,c N represents the number of bits remaining in the current frame period. ip,c This indicates the number of frames remaining to be encoded in the current intra-frame period.

[0041] Due to the parameter coupling and nonlinearity of this problem, an analytical solution cannot be obtained. Therefore, this disclosure uses the differential evolution algorithm to solve it.

[0042] Therefore, the bit allocation method based on RD is shown in Figure 6. First, the RD model of the intra-predicted image is derived using high-dimensional features and the intra-predicted image RD model predictor. Then, the bit allocation optimization problem is established using the RD models of the intra-predicted image and the inter-predicted image. Finally, the allocated bits of the intra-predicted image and the inter-predicted image are solved using the differential evolution algorithm.

[0043] Here is an example: Input: N frames of video to be encoded 1. Initialize the inter-frame prediction image RQ and RD model parameters. 2. Input the intra-frame prediction image into the neural network video encoder to obtain its high-dimensional features before quantization. 3. Input the high-dimensional features into the intra-frame prediction image RQ and RD model predictor to obtain its corresponding model. 4. Solve for the target bits of the corresponding intra-frame prediction image and inter-frame prediction image using the proposed bit allocation method based on the RD models of the intra-frame prediction image and inter-frame prediction image. 5. Calculate its quantization parameters based on the target bits of the intra-frame prediction image and formula (1), and then encode the current image. 6. Calculate its quantization parameters based on the target bits of the inter-frame prediction image and formula (5), and then encode the current image. 7. Update the inter-frame prediction image RQ and RD model parameters based on the encoded information and formulas (7), (8), (9), (10). 8. Repeat steps 6-7 to encode all frames in the current intra-frame period. 9. Repeat steps 2-8 to encode all frames.

[0044] It should be understood that the examples provided above are for illustrative purposes only and should not be construed as limiting the scope of this disclosure. Furthermore, while the steps are described in a specific order, this should not be construed as requiring that they be performed in the specific order shown or sequentially, or that all shown steps be performed to achieve the desired result. In some cases, multitasking and parallel processes may be advantageous. Similarly, while several specific implementation details are included in the foregoing discussion, they should not be considered as limiting the scope of this disclosure, but rather as descriptions of features that may be specific to certain examples. Unless explicitly stated otherwise, certain features described in the context of a single example may also be implemented in combination in a single example. Conversely, unless explicitly stated otherwise, various features described in the context of a single example may also be implemented individually or in any suitable sub-combination in multiple examples.

[0045] Extensive experiments demonstrate that the proposed method outperforms other methods in terms of bitrate error and coding performance. Compared to other methods, the proposed method reduces the average bitrate error by more than 3.97% and achieves a bitrate saving of more than 7.03%. Attributable to the practical RQ model derived by the proposed method, the bitrate control is highly accurate, with an average bitrate error of only 0.77%.

[0046] In this disclosure, there is a direct correspondence between the number of bits allocated to a video frame and the bitrate of the video frame. Specifically, the bitrate can be expressed as bits per pixel (BPP), which can be calculated by dividing the number of bits allocated to a frame by the total number of pixels in the frame (i.e., the product of the frame's width and height). Therefore, when referring to the number of bits allocated to a particular frame, that number can be converted to the corresponding bitrate by dividing it by the frame's resolution. In the context of this disclosure, bitrate and bitrate can be converted to each other based on the frame's resolution; they essentially describe different expressions of the allocation of the same coding resource. Furthermore, since all frames in the same video typically have the same dimensions, i.e., the same width and height, the conversion factor between bitrate and bitrate is the same for different frames in the same video, and they are equivalent. This means that when performing bit allocation and bitrate control, one can freely convert between bitrate and bitrate without introducing additional complexity.

[0047] The following description, with reference to FIG7, further illustrates video processing schemes according to some examples of the present disclosure. FIG7 shows a flowchart of a method for video processing according to some examples of the present disclosure. In some examples, method 700 may be performed at source device 110 as shown in FIG1. ​​It should be understood that method 700 may also include additional boxes not shown and / or some (or more) of the boxes shown may be omitted, and the scope of the present disclosure is not limited in this respect.

[0048] At 702, it is determined whether the current frame of the video is an intra-predicted frame or an inter-predicted frame. In this disclosure, an intra-predicted frame may also be referred to as an intra-frame, intra-predicted frame, or I-frame, and an inter-predicted frame may also be referred to as an inter-frame, inter-predicted frame. At 704, in response to determining that the current frame is an intra-predicted frame, quantization parameters for the current frame are determined using a first model based on the number of bits allocated to the current frame. The parameters of the first model are determined based on the latent features of the current frame. The term "latent features" as used herein refers to feature representations obtained by a neural network encoder after feature extraction of a video frame. Latent features may also be referred to as high-dimensional features, latent variables, latent feature representations, etc. In some cases, the latent features of the current frame may be generated by processing the current frame using a neural network-based encoder and are not quantized. At 706, in response to determining that the current frame is an inter-predicted frame, quantization parameters for the current frame are determined using a second model, which differs from the first model, based on the number of bits allocated to the current frame.

[0049] The first model can be used to describe the relationship between the bitrate and quantization parameters of intra-frame predicted frames. This model calculates the corresponding quantization parameters based on the target number of bits allocated to the intra-frame predicted frames. In this disclosure, the first model can also be referred to as the intra-frame RQ model. The second model can be used to describe the relationship between the bitrate and quantization parameters of inter-frame predicted frames. This model calculates the corresponding quantization parameters based on the target number of bits allocated to the inter-frame predicted frames. In this disclosure, the second model can also be referred to as the inter-frame RQ model. Examples of the first and second models will be described in further detail below. In some cases, the quantization parameters determined by the first or second model can be used to encode the current frame by a neural network-based encoder.

[0050] As shown in Figure 2, after the input video frame is processed by the neural video encoder, unquantized high-dimensional features (i.e., latent features) are obtained before the quantization operation. Image type selection determines whether the current frame of the video is an intra-predicted frame or an inter-predicted frame. Based on the determination result and the target number of bits allocated by the RD-based bit allocation module, the corresponding intra-frame rate control module or inter-frame rate control module determines the quantization parameters for different types of frames, thereby using the final quantization parameters to complete the video encoding. It should be understood that the method flow shown in Figure 2 is for illustrative purposes only; modules and steps not shown may exist, and the shown modules may also be omitted.

[0051] By using different models to determine quantization parameters for intra-frame and inter-frame frames, compared to traditional methods that do not distinguish between frame types, this approach can accurately adapt to the completely different coding and distortion characteristics of the two types of frames, thereby reducing bitrate control errors. Since the frame type is determined first and then the corresponding model is selected for video processing, no complex parameter conversion is required, which can well adapt to the processing requirements of low-latency video coding.

[0052] In some cases, in response to the current frame being the first frame in an intra-period, the number of bits allocated to the current frame can be determined based on the third model, the fourth model, and the total number of frames in the intra-period. The third model can be used to determine the distortion of the intra-predicted frame based on the number of bits allocated to it, and the fourth model can be used to determine the distortion of the inter-predicted frame based on the number of bits allocated to it. In this disclosure, the third model can also be referred to as the intra-RD model, and the fourth model can also be referred to as the inter-RD model. Furthermore, an intra-period can refer to a set of consecutive video frames starting with an intra-predicted frame (i.e., an I-frame) followed by several inter-predicted frames.

[0053] In some cases, the third model can be constructed based on power functions. As an example, and not a limitation, the third model can be constructed as follows: Where DI R represents the distortion of frames predicted intra-frame. I Indicates the number of bits allocated to the frame predicted within this frame, γ I and ∈ I The parameters represent the third model. By utilizing power functions, the third model can effectively fit the nonlinear relationship between bitrate and distortion, thereby improving the accuracy of the RD model and facilitating more accurate bit allocation. It should be understood that the third model can also be constructed based on logarithmic functions, polynomial functions, etc., to adapt to different types of neural network video encoders and various coding application scenarios; the scope of this disclosure is not limited in this respect.

[0054] In some cases, the parameters of the third model can be determined by processing latent features using a first neural network. The structure of the first neural network can be as shown in Figure 3. The first neural network can process high-dimensional features before quantization to generate model parameters. The first neural network may include a feature mapping subnetwork for transforming the latent features and a regression subnetwork for generating model parameters based on the transformed latent features. The feature mapping subnetwork may include convolutional layers (denoted as "Conv" in Figure 3), a leaky corrected linear unit layer (denoted as "LeakyReLU" in Figure 3), a channel splitting layer, and a corrected linear unit layer (denoted as "ReLU" in Figure 3). The regression subnetwork may include an average pooling layer (denoted as "AvgPool" in Figure 3), a fully connected layer (denoted as "FC" in Figure 3), a leaky corrected linear unit layer, and a sigmoid function layer. It should be understood that the neural network structure shown in Figure 3 is illustrated for illustrative purposes only, and the first neural network may have other network structures (e.g., more / fewer / different layers) depending on the application scenario. Any reasonable modification of the network structure should be considered within the scope of this disclosure.

[0055] In some cases, the training loss function of the first neural network can be the mean squared error (MSE). During training, the model parameters can be normalized to the range [0,1], and the sigmoid activation function can be used to improve training stability and promote convergence. By adaptively determining the parameters of the third model through the neural network, no manual parameter tuning is required, which can efficiently and accurately adapt to the encoding characteristics of different video content.

[0056] In some cases, the fourth model can be constructed based on power functions. Exemplarily, but not restrictively, the fourth model can be constructed as follows: Where D P R represents the distortion of frames predicted between frames. P Indicates the number of bits allocated to the frames in this inter-frame prediction, γ P and ∈ PThe parameters represent the fourth model. By utilizing power functions, the fourth model can effectively fit the nonlinear relationship between bit rate and distortion, thereby improving the accuracy of the RD model and facilitating more accurate bit allocation. It should be understood that the fourth model can also be constructed based on logarithmic functions, polynomial functions, etc., to adapt to different types of neural network video encoders and various coding application scenarios; the scope of this disclosure is not limited in this respect.

[0057] In some cases, the parameters of the fourth model can be initialized when processing a video. For example, they can be set to preset values. Alternatively, the last used parameter values ​​can be reused. Or, they can be determined via a neural network in a similar manner to the parameters of the third model type. During video processing, the parameters of the fourth model can be updated. For example, the parameters of the fourth model can be updated based on the parameters of the fourth model used for the previous inter-frame prediction, the actual bitrate of the previous inter-frame prediction frame, and the actual distortion.

[0058] As an example, the parameters of the fourth model can be updated in the following way: Where ∈ pre and γ pre η represents the parameters of the frame predicted from the previous inter-frame. R-D BPP represents the learning rate. a MSE represents the actual bitrate of the frame predicted by the previous inter-frame prediction. a This indicates the actual distortion of the frame predicted from the previous inter-frame.

[0059] By updating the parameters, the method proposed in this disclosure can update the model in real time, better adapting to changes in video content and improving encoding quality and bitrate control. It should be understood that the parameters of the fourth model can be updated in any way, such as averaging the parameters of all previous frames, or using other optimization methods such as the Adam optimizer or momentum method, and this disclosure does not impose any restrictions on this.

[0060] Furthermore, in some cases, the optimization problem for bit allocation can be constructed based on the third and fourth models. This optimization problem aims to minimize the total distortion of all frames in the intra-frame period, and constraints may include: the total number of bits allocated to intra-predicted frames and all inter-predicted frames does not exceed the total number of bits in the intra-frame period; the coding quality of intra-predicted frames is higher than that of inter-predicted frames; and the coding quality of inter-predicted frames is not lower than a predetermined proportion of that of intra-predicted frames.

[0061] As an example, and not a limitation, the optimization problem can be formulated as follows: so that Minimize as the objective to determine B I and B P And satisfy the following constraints: B I +(Nip -1)·B P =B ip , and Where, γ I and ∈ I γ represents the parameters of the third model. P and ∈ P The parameters of the fourth model are represented by W and H, which represent the width and height of the frame, respectively, and N. ip B represents the total number of frames in the intra-frame period. I B represents the number of bits allocated to the frame for intra-frame prediction. P B represents the number of bits allocated to the frames for inter-frame prediction. ip This indicates the number of bits allocated to the intra-frame period, with p0 and p1 representing preset values. In this example, the preset ratio is set to 10. 0.5 However, it can be set to any value according to actual needs. It should be understood that the above optimization problem is only an example. In addition to minimizing the total distortion, the goal can also be to maximize coding quality, maximize coding efficiency, etc. The optimization problem can have more or fewer constraints, and the constraints can also include, for example: the coding quality of the current frame is not lower than the preset ratio of the previous frame, and the coding quality fluctuation of the predicted frames between frames within the intra-frame period does not exceed a preset threshold, etc., which are not limited in this disclosure.

[0062] Solving this optimization problem determines the number of bits allocated to each frame in an intra-frame period. In some cases, this optimization problem can be solved using a differential evolution algorithm. Alternatively, it can be solved using optimization algorithms such as genetic algorithms and particle swarm optimization. This disclosure does not impose any restrictions on the solution method.

[0063] By constructing and solving a bit allocation optimization problem, the optimal allocation of bit resources between intra-frame and inter-frame predicted frames can be achieved, thereby minimizing the overall distortion of intra-frame periods and improving coding quality. Simultaneously, by constraining the quality relationship between intra-frame and inter-frame predicted frames, excessive quality differences between them can be avoided, thus improving the subjective smoothness of video playback.

[0064] In some cases, the number of bits allocated to an intra-period can be determined based on the target number of bits for the video, the number of bits consumed, the number of frames remaining to be encoded, and the total number of frames in the intra-period. As an example, the number of bits allocated to an intra-period can be determined as follows: Among them B ip B represents the number of bits allocated to an intra-frame period. t B represents the target number of bits.c N represents the number of bits consumed. r N represents the number of frames remaining to be encoded. ip This represents the total number of frames in the intra-period. By allocating the bits of the intra-period proportionally based on the remaining bits and the remaining number of frames, the uniformity and stability of bitrate control can be ensured. It should be understood that the method described above for determining the number of bits allocated to the intra-period is merely exemplary and not limiting, and the number of bits allocated to the intra-period can also be determined in any other suitable manner, such as a weighted allocation based on the complexity of the video content. The scope of this disclosure is not limited in this respect.

[0065] In some cases, in response to the current frame not being the first frame in an intra-period, the number of bits allocated to the current frame can be determined based on the number of bits allocated to the intra-period, the number of bits consumed in the intra-period, and the number of remaining frames to be encoded in the intra-period. Exemplarily, but not limitingly, the number of bits allocated to the current frame can be determined as follows: Among them, B P B represents the number of bits allocated to the frames for inter-frame prediction. ip B represents the number of bits allocated to an intra-frame period. ip,c N represents the number of bits consumed within an intra-frame period. ip,r This indicates the number of remaining frames to be encoded within the intra-frame period.

[0066] By dynamically calculating the allocated bits in real time, the allocation strategy can be dynamically adjusted based on the actual bits consumed, thereby achieving better bitrate control. It should be understood that the method described above for determining the number of bits allocated to the current frame is merely exemplary and not limiting. The number of bits allocated to the current frame can also be determined in any other suitable manner, such as differentiated allocation based on the content complexity of inter-frame prediction frames. The scope of this disclosure is not limited in this respect.

[0067] The bit allocation method based on RD is shown in Figure 6. A first neural network (the intra-frame RD model predictor in Figure 6) is used to process latent features, thereby obtaining a third model (the RD model of the intra-frame in Figure 6). A bit allocation optimization problem is constructed based on the third model and the fourth model (the RD model of the inter-frame in Figure 6). This problem is solved using a differential evolution-based solver, thereby obtaining the target bits allocated to intra-frames and inter-frames. It should be understood that the above method flow is shown for illustrative purposes only and should not be construed as a limitation of this disclosure.

[0068] In some cases, the first model can be constructed based on a logarithmic function. As an example, the first model can be constructed as a QP function. I =α I ln(βI R I ), where QP I R represents the quantization parameters for intra-frame prediction of a frame. I α represents the number of bits allocated to the frame predicted within this frame. I and β I The parameters of the first model are represented. By using a logarithmic function, the relationship between the bit rate and the quantization parameters can be effectively characterized, thereby improving the accuracy of the RQ model and facilitating more accurate determination of the quantization parameters. It should be understood that the construction method of the first model described above is merely exemplary and not limiting; the first model can also be constructed in any other suitable manner, such as using power functions, polynomials, etc. The scope of this disclosure is not limited in this respect.

[0069] The parameters of the first model can be determined by processing latent features using a second neural network. This second neural network can have the same structure as the first neural network. The structure of the neural network has been described in detail above and will not be repeated here. A unified network structure can reduce system complexity and save computational resources and storage overhead. Adaptively determining the parameters of the first model through the second neural network allows for efficient adaptation to the coding characteristics of different intra-frame prediction images. It should be understood that the second neural network can have the same structure as the first neural network but different network parameters, and the second neural network can also have a different structure than the first neural network.

[0070] In some cases, the second model can be constructed based on a logarithmic function and segmented across multiple different bitrate ranges. As an example, the second model can be constructed as follows: Among them, QP P R represents the quantization parameters for frames predicted between frames. P R represents the number of bits allocated to the frames in this inter-frame prediction. int Represents the segmented threshold, α l β l α h and β h These represent the parameters of the second model for the two bitrate bands, respectively. For example, the segmentation threshold R... int It can be determined in the following ways: In this example, the second model is constructed in two segments. By constructing the second model in segments, it is possible to better capture the differences in bitrate distortion characteristics of inter-frame prediction frames at different bitrate ranges. Since the encoding behavior of inter-frame prediction frames differs significantly between low and high bitrate ranges, a single model is insufficient to accurately describe the relationship between bitrate and quantization parameters across the entire bitrate range. Segmented modeling allows for the use of targeted model parameters for each bitrate range, thereby improving the accuracy of quantization parameter prediction and reducing bitrate control errors. Furthermore, segmented modeling offers good scalability, allowing for flexible adjustment of the number of segments and segment thresholds according to actual application needs to adapt to different neural network video encoders and encoding scenarios. It should be understood that the second model can also be constructed in three or more segments to accommodate a wider range of bitrate variations and achieve more precise adaptation. Additionally or alternatively, the second model can also be constructed based on functions such as power functions or polynomial functions; this disclosure does not impose any limitations on this.

[0071] In some cases, the parameters of the second model can be initialized when processing a video. For example, they can be set to preset values. Alternatively, the last used parameter values ​​can be reused. Or, they can be determined via a neural network in a similar manner to the parameters of the first model. During video processing, the parameters of the second model can be updated. These updates can be based on the parameters of the second model used for the previous inter-frame prediction frame, the number of bits allocated to the previous inter-frame prediction frame, the actual bitrate of the previous inter-frame prediction frame, and the actual quantization parameters.

[0072] As an example, the parameters of the second model can be updated in the following way: Where α pre and β pre η represents the parameters of the frame predicted from the previous inter-frame. R-Q BPP represents the learning rate. a QP represents the actual bitrate of the frame predicted by the previous inter-frame. a This represents the actual quantization parameters of the frame predicted by the previous inter-frame prediction, where the number of bits allocated to the frame predicted by the previous inter-frame prediction is less than or equal to R. int Then the parameter α l and β l Based on α respectively P and β P Updated; if the number of bits allocated to the frame in the previous inter-frame prediction is greater than R. int Then the parameter α h and β h Based on α respectively P and β PUpdated. As shown in Figure 5, after encoding the previous inter-frame prediction frame is completed, parameters can be updated based on the actual encoding information of that frame. This is based on the number of bits allocated to the previous inter-frame prediction frame and R... int Based on the relationship, the corresponding model branch is selected for parameter update. The updated RQ model is then used to derive the quantization parameters. By updating parameters independently in segments, models for different bitrate segments can be accurately adapted to the coding characteristics of the corresponding bitrate segments. It should be understood that the process shown in Figure 5 should be considered as an explanation of the scheme of this disclosure and should not be considered as a limitation thereof.

[0073] By dynamically updating the parameters of the second model, the model can adapt to changes in video content in real time, improving the bitrate control accuracy of inter-frame prediction frames. It should be understood that the update method described above is exemplary and not limiting; the parameters of the second model can also be updated using other optimization methods, such as the Adam optimizer, momentum methods, etc. The scope of this disclosure is not limited in this respect.

[0074] The methods for video processing according to the examples of this disclosure can be applied to various video coding configurations. For example, the method can be applied to a low-delay configuration, in which each inter-frame prediction frame references only frames that precede it in time, suitable for latency-sensitive applications such as real-time communication and video conferencing. The method can also be applied to a random access configuration, in which inter-frame prediction frames can reference frames that precede or follow them in time, improving coding efficiency through bidirectional prediction, suitable for applications such as video-on-demand and streaming media. Furthermore, the method can be applied to an all-intra configuration, in which all frames are encoded using intra-frame prediction, suitable for applications requiring frame-by-frame access such as video editing and professional production. It should be understood that the methods of this disclosure can flexibly adjust the length of the intra-frame period, the bit allocation strategy, and the update method of the model parameters according to different coding configurations to adapt to the needs of different application scenarios. The scope of this disclosure is not limited in this respect. Example Methods

[0075] Figure 7 illustrates a flowchart of a method 700 for video processing according to some examples of the present disclosure. In some examples, method 700 may be performed at the source device 110 as shown in Figure 1. It should be understood that method 700 may also include additional boxes not shown and / or one (or some) of the boxes shown may be omitted, and the scope of the present disclosure is not limited in this respect.

[0076] At position 702, determine whether the current frame of the video is an intra-predicted frame or an inter-predicted frame.

[0077] At 704, in response to determining that the current frame is an intra-predicted frame, quantization parameters for the current frame are determined using a first model based on the number of bits allocated to the current frame. The parameters of the first model are determined based on the latent features of the current frame. At 706, in response to determining that the current frame is an inter-predicted frame, quantization parameters for the current frame are determined using a second model, which differs from the first model, based on the number of bits allocated to the current frame.

[0078] In some cases, method 700 further includes: in response to the current frame being the first frame in an intra-period, determining the number of bits allocated to the current frame based on a third model, a fourth model, and the total number of frames in the intra-period, wherein the third model is used to determine the distortion of the intra-predicted frame based on the number of bits allocated to the intra-predicted frame, and the fourth model is used to determine the distortion of the inter-predicted frame based on the number of bits allocated to the inter-predicted frame; or in response to the current frame not being the first frame in an intra-period, determining the number of bits allocated to the current frame based on the number of bits allocated to the intra-period, the number of bits consumed in the intra-period, and the number of remaining frames to be encoded in the intra-period.

[0079] In some cases, determining the number of bits allocated to the current frame based on the third model, the fourth model, and the total number of frames in the intra-period includes: constructing an optimization problem for bit allocation based on the third and fourth models; and determining the number of bits allocated to each frame in the intra-period by solving the optimization problem.

[0080] In some cases, the objective of the optimization problem is to minimize the total distortion of all frames in an intra-period. The constraints of the optimization problem include at least one of the following: the sum of the number of bits allocated to intra-predicted frames in the intra-period and the number of bits allocated to all inter-predicted frames in the intra-period does not exceed the number of bits allocated to the intra-period; the coding quality of intra-predicted frames is higher than that of inter-predicted frames; and the coding quality of inter-predicted frames is not lower than a preset proportion of the coding quality of intra-predicted frames.

[0081] In some cases, the optimization problem is formulated as follows: so that Minimize as the objective, while satisfying the following constraints: B I +(N ip -1)·B P =B ip , and Where, γ I and ∈ I γ represents the parameters of the third model. P and ∈ P The parameters of the fourth model are represented by W and H, which represent the width and height of the frame, respectively, and N. ip B represents the total number of frames in the intra-frame period.I B represents the number of bits allocated to the frame for intra-frame prediction. P B represents the number of bits allocated to the frames for inter-frame prediction. ip This indicates the number of bits allocated to the intra-frame period, with p0 and p1 representing preset values.

[0082] In some cases, the optimization problem is solved using the differential evolution algorithm.

[0083] In some cases, the third and fourth models are constructed based on power functions.

[0084] In some cases, the third model is constructed as follows: Among them, D I R represents the distortion of frames predicted intra-frame. I Indicates the number of bits allocated to the frame predicted within this frame, γ I and ∈ I This represents the parameters of the third model.

[0085] In some cases, the parameters of the third model are determined by processing latent features using the first neural network.

[0086] In some cases, the fourth model is constructed as follows: Among them, D P R represents the distortion of frames predicted between frames. P Indicates the number of bits allocated to the frames in this inter-frame prediction, γ P and ∈ P This represents the parameters of the fourth model.

[0087] In some cases, the parameters of the fourth model are updated based on the parameters of the fourth model used for the previous inter-frame prediction, the actual bitrate of the previous inter-frame prediction frame, and the actual distortion.

[0088] In some cases, the parameters of the fourth model are updated in the following ways: Where, ∈ pre and γ pre η represents the parameters of the frame predicted from the previous inter-frame. R-D BPP represents the learning rate. a MSE represents the actual bitrate of the frame predicted by the previous inter-frame prediction. a This indicates the actual distortion of the frame predicted from the previous inter-frame.

[0089] In some cases, method 700 further includes determining the number of bits allocated to the intra-period based on the target number of bits for the video, the number of bits consumed, the number of remaining frames to be encoded, and the total number of frames in the intra-period.

[0090] In some cases, the number of bits allocated to an intra-frame period is determined in the following way: Among them, B ip B represents the number of bits allocated to an intra-frame period. t B represents the target number of bits. c N represents the number of bits consumed. r N represents the number of frames remaining to be encoded. ip This indicates the total number of frames in the intra-frame period.

[0091] In some cases, in response to the current frame not being the first frame in an intra-frame period, the number of bits allocated to the current frame is determined as follows: Among them, B P B represents the number of bits allocated to the frames for inter-frame prediction. ip B represents the number of bits allocated to an intra-frame period. ip,c N represents the number of bits consumed within an intra-frame period. ip,r This indicates the number of remaining frames to be encoded within the intra-frame period.

[0092] In some cases, the first model is constructed based on a logarithmic function.

[0093] In some cases, the first model is constructed as follows: QP I =α I ln(β I R I ), where QP I R represents the quantization parameters for intra-frame prediction of a frame. I α represents the number of bits allocated to the frame predicted within this frame. I and β I This represents the parameters of the first model.

[0094] In some cases, the parameters of the first model are determined by using a second neural network to process latent features.

[0095] In some cases, the first neural network has the same structure as the second neural network, which includes a feature mapping subnetwork for transforming latent features and a regression subnetwork for generating model parameters based on the transformed latent features.

[0096] In some cases, the second model is constructed in segments for multiple different bitrate ranges.

[0097] In some cases, multiple different bitrate bands include two bitrate bands, and the second model is constructed as follows: Among them, QP P R represents the quantization parameters for frames predicted between frames. P R represents the number of bits allocated to the frames in this inter-frame prediction.int Represents the segmented threshold, α l β l α h and β h These represent the parameters of the second model for the two bit rate bands, respectively.

[0098] In some cases, the segmentation threshold R int It is determined in the following way:

[0099] In some cases, the parameters of the second model are updated based on the parameters of the second model used for the previous inter-frame prediction, the number of bits allocated to the previous inter-frame prediction, the actual bitrate of the previous inter-frame prediction, and the actual quantization parameters.

[0100] In some cases, the parameters of the second model are updated in the following ways: Where, α pre and β pre η represents the parameters of the frame predicted from the previous inter-frame. R-Q BPP represents the learning rate. a QP represents the actual bitrate of the frame predicted by the previous inter-frame. a This represents the actual quantization parameters of the frame predicted by the previous inter-frame prediction, where the number of bits allocated to the frame predicted by the previous inter-frame prediction is less than or equal to R. int Then the parameter α l and β l Based on α respectively P and β P Updated; if the number of bits allocated to the frame in the previous inter-frame prediction is greater than R. int Then the parameter α h and β h Based on α respectively P and β P Updated.

[0101] In some cases, the latent features of the current frame are generated by processing the current frame using a neural network-based encoder and are not quantized, while quantization parameters are used by the encoder to encode the current frame. Example apparatus and devices.

[0102] Examples of this disclosure also provide corresponding apparatus and devices for implementing the methods or processes described above. Figure 8 shows a block diagram of an example apparatus 800 for video processing according to some examples of this disclosure. This apparatus 800 can, for example, be used to implement methods according to some examples of this disclosure. In some examples, apparatus 800 can be implemented at the source device 110 shown in Figure 1.

[0103] As shown in Figure 8, the device 800 may include a judgment module 802 and a determination module 804. The judgment module 802 is configured to determine whether the current frame of the video is an intra-predicted frame or an inter-predicted frame. The determination module 804 is configured to: in response to determining that the current frame is an intra-predicted frame, determine quantization parameters for the current frame based on the number of bits allocated to the current frame using a first model, the parameters of the first model being determined based on the latent features of the current frame; or in response to determining that the current frame is an inter-predicted frame, determine quantization parameters for the current frame based on the number of bits allocated to the current frame using a second model, the second model being different from the first model.

[0104] In some cases, the apparatus 800 may further include a first bit number determination module, configured to: determine the number of bits allocated to the current frame based on a third model, a fourth model, and the total number of frames in the intra-frame period, in response to the current frame being the first frame in an intra-frame period, the third model being used to determine the distortion of the intra-frame predicted frame based on the number of bits allocated to the intra-frame predicted frame, and the fourth model being used to determine the distortion of the inter-frame predicted frame based on the number of bits allocated to the inter-frame predicted frame; or determine the number of bits allocated to the current frame based on the number of bits allocated to the intra-frame period, the number of bits consumed in the intra-frame period, and the number of remaining frames to be encoded in the intra-frame period, in response to the current frame not being the first frame in an intra-frame period.

[0105] In some cases, the first bit number determination module includes an optimization problem construction module and a bit number allocation module. The optimization problem construction module is configured to construct an optimization problem for bit allocation based on the third and fourth models. The bit number allocation module is configured to determine the number of bits allocated to each frame in the intra-frame period by solving the optimization problem.

[0106] In some cases, the objective of the optimization problem is to minimize the total distortion of all frames in an intra-period. The constraints of the optimization problem include at least one of the following: the sum of the number of bits allocated to intra-predicted frames in the intra-period and the number of bits allocated to all inter-predicted frames in the intra-period does not exceed the number of bits allocated to the intra-period; the coding quality of intra-predicted frames is higher than that of inter-predicted frames; and the coding quality of inter-predicted frames is not lower than a preset proportion of the coding quality of intra-predicted frames.

[0107] In some cases, the optimization problem is formulated as follows: so that Minimize as the objective, while satisfying the following constraints: B I +(N ip -1)·B P =B ip , and Where, γ I and ∈I γ represents the parameters of the third model. P and ∈ P The parameters of the fourth model are represented by W and H, which represent the width and height of the frame, respectively, and N. ip B represents the total number of frames in the intra-frame period. I B represents the number of bits allocated to the frame for intra-frame prediction. P B represents the number of bits allocated to the frames for inter-frame prediction. ip This indicates the number of bits allocated to the intra-frame period, with p0 and p1 representing preset values.

[0108] In some cases, the optimization problem is solved using the differential evolution algorithm.

[0109] In some cases, the third and fourth models are constructed based on power functions.

[0110] In some cases, the third model is constructed as follows: Among them, D I R represents the distortion of frames predicted intra-frame. I Indicates the number of bits allocated to the frame predicted within this frame, γ I and ∈ I This represents the parameters of the third model.

[0111] In some cases, the parameters of the third model are determined by processing latent features using the first neural network.

[0112] In some cases, the fourth model is constructed as follows: Among them, D P R represents the distortion of frames predicted between frames. P Indicates the number of bits allocated to the frames in this inter-frame prediction, γ P and ∈ P This represents the parameters of the fourth model.

[0113] In some cases, the parameters of the fourth model are updated based on the parameters of the fourth model used for the previous inter-frame prediction, the actual bitrate of the previous inter-frame prediction frame, and the actual distortion.

[0114] In some cases, the parameters of the fourth model are updated in the following ways: Where, ∈ pre and γ pre η represents the parameters of the frame predicted from the previous inter-frame. R-D BPP represents the learning rate. a MSE represents the actual bitrate of the frame predicted by the previous inter-frame prediction. a This indicates the actual distortion of the frame predicted from the previous inter-frame.

[0115] In some cases, the apparatus 800 also includes a second bit number determination module configured to determine the number of bits allocated to the intra-frame period based on the target bit number for the video, the bit number consumed, the number of remaining frames to be encoded, and the total number of frames in the intra-frame period.

[0116] In some cases, the number of bits allocated to an intra-frame period is determined in the following way: Among them, B ip B represents the number of bits allocated to an intra-frame period. t B represents the target number of bits. c N represents the number of bits consumed. r N represents the number of frames remaining to be encoded. ip This indicates the total number of frames in the intra-frame period.

[0117] In some cases, in response to the current frame not being the first frame in an intra-frame period, the number of bits allocated to the current frame is determined as follows: Among them, B P B represents the number of bits allocated to the frames for inter-frame prediction. ip B represents the number of bits allocated to an intra-frame period. ip,c N represents the number of bits consumed within an intra-frame period. ip,r This indicates the number of remaining frames to be encoded within the intra-frame period.

[0118] In some cases, the first model is constructed based on a logarithmic function.

[0119] In some cases, the first model is constructed as follows: QP I =α I ln(β I R I ), where QP I R represents the quantization parameters for intra-frame prediction of a frame. I α represents the number of bits allocated to the frame predicted within this frame. I and β I This represents the parameters of the first model.

[0120] In some cases, the parameters of the first model are determined by using a second neural network to process latent features.

[0121] In some cases, the first neural network has the same structure as the second neural network, which includes a feature mapping subnetwork for transforming latent features and a regression subnetwork for generating model parameters based on the transformed latent features.

[0122] In some cases, the second model is constructed in segments for multiple different bitrate ranges.

[0123] In some cases, multiple different bitrate bands include two bitrate bands, and the second model is constructed as follows: Among them, QP P R represents the quantization parameters for frames predicted between frames. P R represents the number of bits allocated to the frames in this inter-frame prediction. int Represents the segmented threshold, α l β l α h and β h These represent the parameters of the second model for the two bit rate bands, respectively.

[0124] In some cases, the segmentation threshold R int It is determined in the following way:

[0125] In some cases, the parameters of the second model are updated based on the parameters of the second model used for the previous inter-frame prediction, the number of bits allocated to the previous inter-frame prediction, the actual bitrate of the previous inter-frame prediction, and the actual quantization parameters.

[0126] In some cases, the parameters of the second model are updated in the following ways: Where, α pre and β pre η represents the parameters of the frame predicted from the previous inter-frame. R-Q BPP represents the learning rate. a QP represents the actual bitrate of the frame predicted by the previous inter-frame. a This represents the actual quantization parameters of the frame predicted by the previous inter-frame prediction, where the number of bits allocated to the frame predicted by the previous inter-frame prediction is less than or equal to R. int Then the parameter α l and β l Based on α respectively P and β P Updated; if the number of bits allocated to the frame in the previous inter-frame prediction is greater than R. int Then the parameter α h and β h Based on α respectively P and β P Updated.

[0127] In some cases, the latent features of the current frame are generated by processing the current frame using a neural network-based encoder and are not quantized, while the quantization parameters are used by the encoder to encode the current frame. Example device.

[0128] Figure 9 shows a block diagram of a device 900 in which one or more examples of the present disclosure may be implemented. It should be understood that the electronic device 900 shown in Figure 9 is merely exemplary and should not constitute any limitation on the functionality and scope of the examples described herein. The electronic device 900 shown in Figure 9 can be used to implement the source device 110, terminal device 120, and / or the methods described above shown in Figure 1.

[0129] As shown in Figure 9, the electronic device 900 is in the form of a general-purpose electronic device. The components of the electronic device 900 may include, but are not limited to, one or more processing units or processors 910, memory 920, storage devices 930, one or more communication units 940, one or more input devices 950, and one or more output devices 960. The processor 910 may be a physical or virtual processor and is capable of performing various processes according to the programs stored in the memory 920. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of the electronic device 900.

[0130] Electronic device 900 typically includes multiple computer storage media. Such media can be any available media accessible to electronic device 900, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 920 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 930 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data (e.g., training data for training) and can be accessed within electronic device 900.

[0131] Electronic device 900 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not shown in FIG. 9, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks may be provided. In these cases, each drive may be connected to a bus (not shown) via one or more data media interfaces. Memory 920 may include computer program product 925 having one or more program modules configured to perform various methods or actions of various implementations of this disclosure.

[0132] The communication unit 940 enables communication with other electronic devices via a communication medium. Additionally, the functionality of the components of the electronic device 900 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, the electronic device 900 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0133] Input device 950 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 960 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 900 can also communicate with one or more external devices (not shown) via communication unit 940 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 900, or with any device that enables electronic device 900 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0134] According to exemplary implementations of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to exemplary implementations of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above. According to exemplary implementations of this disclosure, a computer program product is provided that stores a computer program thereon, which, when executed by a processor, implements the methods described above.

[0135] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0136] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0137] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0138] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0139] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for video processing, comprising: Determine whether the current frame of the video is an intra-predicted frame or an inter-predicted frame; as well as In response to determining that the current frame is an intra-predicted frame, quantization parameters for the current frame are determined using a first model based on the number of bits allocated to the current frame, the parameters of the first model being determined based on the latent features of the current frame; or In response to determining that the current frame is an inter-frame predicted frame, a second model is used to determine quantization parameters for the current frame based on the number of bits allocated to the current frame, the second model being different from the first model.

2. The method according to claim 1, further comprising: In response that the current frame is the first frame in an intra-frame period, the number of bits allocated to the current frame is determined based on the third model, the fourth model, and the total number of frames in the intra-frame period. The third model is used to determine the distortion of the intra-predicted frame based on the number of bits allocated to the intra-predicted frame, and the fourth model is used to determine the distortion of the inter-predicted frame based on the number of bits allocated to the inter-predicted frame; or In response to the fact that the current frame is not the first frame in an intra-frame period, the number of bits allocated to the current frame is determined based on the number of bits allocated to the intra-frame period, the number of bits consumed in the intra-frame period, and the number of remaining frames to be encoded in the intra-frame period.

3. The method of claim 2, wherein determining the number of bits allocated to the current frame based on the third model, the fourth model, and the total number of frames in the intra-frame period comprises: An optimization problem for bit allocation is constructed based on the third and fourth models; as well as The number of bits allocated to each frame in the intra-frame period is determined by solving the optimization problem.

4. The method of claim 3, wherein the objective of the optimization problem is to minimize the total distortion of all frames in the intra-frame period, and the constraints of the optimization problem include at least one of the following: The sum of the number of bits allocated to intra-predicted frames in the intra-period and the number of bits allocated to all inter-predicted frames in the intra-period does not exceed the number of bits allocated to the intra-period. The coding quality of intra-frame predicted frames is higher than that of inter-frame predicted frames. The coding quality of inter-frame predicted frames is no less than a preset ratio of the coding quality of intra-frame predicted frames.

5. The method according to claim 4, wherein the optimization problem is formulated as follows: To make Minimize as the objective, while satisfying the following constraints: B I +(N ip -1)·B P =B ip , and in, γ I and ∈ I The parameter γ represents the third model. P and ∈ P The parameters of the fourth model are represented by W and H, which represent the width and height of the frame, respectively, and N. ip B represents the total number of frames in the intra-frame period. I B represents the number of bits allocated to the frame for intra-frame prediction. P B represents the number of bits allocated to the frames for inter-frame prediction. ip This indicates the number of bits allocated to the intra-frame period, with p0 and p1 representing preset values.

6. The method according to any one of claims 3 to 5, wherein the optimization problem is solved using a differential evolution algorithm.

7. The method according to any one of claims 2 to 6, wherein the third model and the fourth model are constructed based on power functions.

8. The method according to claim 7, wherein the third model is constructed as follows: in, D I R represents the distortion of frames predicted intra-frame. I Indicates the number of bits allocated to the frame predicted within this frame, γ I and ∈ I The parameters represent the third model.

9. The method according to claims 2 to 8, wherein the parameters of the third model are determined by processing the latent features using a first neural network.

10. The method of claim 7, wherein the fourth model is constructed as follows: in, D P R represents the distortion of frames predicted between frames. P Indicates the number of bits allocated to the frames in this inter-frame prediction, γ P and ∈ P The parameters represent the fourth model.

11. The method of claim 10, wherein the parameters of the fourth model are updated based on the parameters of the fourth model for the frame of the previous inter-frame prediction, the actual bitrate of the frame of the previous inter-frame prediction, and the actual distortion.

12. The method of claim 11, wherein the parameters of the fourth model are updated in the following manner: in, ∈ pre and γ pre η represents the parameter of the frame predicted in the previous inter-frame prediction. R-D BPP represents the learning rate. a MSE represents the actual bitrate of the frame predicted by the previous inter-frame prediction. a This indicates the actual distortion of the frame predicted in the previous inter-frame prediction.

13. The method according to any one of claims 2 to 12, further comprising: The number of bits allocated to the intra-frame period is determined based on the target number of bits for the video, the number of bits consumed, the number of frames remaining to be encoded, and the total number of frames in the intra-frame period.

14. The method of claim 13, wherein the number of bits allocated to the intra-frame period is determined in the following manner: in, B ip B represents the number of bits allocated to the intra-frame period. t B represents the target number of bits. c N represents the number of bits consumed. r N represents the number of remaining frames to be encoded. ip The total number of frames represents the intra-frame period.

15. The method according to any one of claims 2 to 14, wherein in response to the current frame not being the first frame in an intra-frame period, the number of bits allocated to the current frame is determined in the following manner: in, B P B represents the number of bits allocated to the frames in the inter-frame prediction. ip B represents the number of bits allocated to the intra-frame period. ip,c N represents the number of bits consumed in the intra-frame period. ip,r This indicates the number of remaining frames to be encoded within the intra-frame period.

16. The method according to any one of claims 1 to 15, wherein the first model is constructed based on a logarithmic function.

17. The method of claim 16, wherein the first model is constructed as follows: QP I =α I ln(β I R I ), in, QP I R represents the quantization parameters for intra-frame prediction of a frame. I α represents the number of bits allocated to the frame predicted within this frame. I and β I The parameters represent the parameters of the first model.

18. The method according to any one of claims 1 to 17, wherein the parameters of the first model are determined by processing the latent features using a second neural network.

19. The method of claim 18, wherein the structure of the first neural network is the same as the structure of the second neural network, the structure comprising a feature mapping subnetwork for transforming the latent features and a regression subnetwork for generating model parameters based on the transformed latent features.

20. The method according to any one of claims 1 to 19, wherein the second model is constructed segmentally for multiple different code rate segments.

21. The method of claim 20, wherein the plurality of different code rate segments comprises two code rate segments, and the second model is constructed as follows: in, QP P R represents the quantization parameters for frames predicted between frames. P R represents the number of bits allocated to the frames in this inter-frame prediction. int Represents the segmented threshold, α l β l α h and β h These represent the parameters of the second model for the two bitrate bands, respectively.

22. The method of claim 21, wherein the segmentation threshold R int It is determined in the following way:

23. The method according to any one of claims 21 to 22, wherein the parameters of the second model are updated based on the parameters of the second model for the frame of the previous inter-frame prediction, the number of bits allocated to the frame of the previous inter-frame prediction, the actual bit rate of the frame of the previous inter-frame prediction, and the actual quantization parameters.

24. The method of claim 23, wherein the parameters of the second model are updated in the following manner: in, α pre and β pre η represents the parameter of the frame predicted in the previous inter-frame prediction. R-Q BPP represents the learning rate. a The QP represents the actual bitrate of the frame predicted by the previous inter-frame. a This represents the actual quantization parameters of the frame predicted by the previous inter-frame. If the number of bits allocated to the previous inter-frame prediction frame is less than or equal to R int Then the parameter α l and β l Based on α respectively P and β P Updated; If it is determined that the number of bits allocated to the frame of the previous inter-frame prediction is greater than R int Then the parameter α h and β h Based on α respectively P and β P Updated.

25. The method according to claims 1 to 24, wherein the latent features of the current frame are generated by processing the current frame using a neural network-based encoder and are not quantized, and the quantization parameters are used to encode the current frame by the encoder.

26. An apparatus for video processing, comprising: The judgment module is configured to determine whether the current frame of the video is an intra-frame predicted frame or an inter-frame predicted frame. as well as The determination module is configured to: in response to determining that the current frame is an intra-predicted frame, determine quantization parameters for the current frame using a first model based on the number of bits allocated to the current frame, wherein the parameters of the first model are determined based on the latent features of the current frame; Alternatively, in response to determining that the current frame is an inter-frame predicted frame, a second model is used to determine quantization parameters for the current frame based on the number of bits allocated to the current frame, the second model being different from the first model.

27. An electronic device comprising: At least one processor; as well as At least one memory coupled to the at least one processor and storing instructions for execution by the at least one processor, the instructions causing the electronic device to perform the method according to any one of claims 1 to 25 when executed by the at least one processor.

28. A computer-readable storage medium having instructions stored thereon, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 25.

29. A computer program product tangibly stored in a computer storage medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1 to 25.

Images