Video decoding method, video encoding method, and apparatus

Abstract

The present application relates to the technical field of video encoding and decoding, and provides a video decoding method, a video encoding method, and an apparatus, for use in improving video encoding efficiency. In some embodiments of the present application, the video decoding method comprises: decoding encoded data of the current frame, and determining residual information of the current frame, and the hyperprior residual mean and hyperprior residual variance corresponding to each pixel point in the current frame; determining a residual skip threshold of the current frame on the basis of a first quality adjustment parameter; for each pixel point in the current frame, if the hyperprior residual variance of the pixel point is less than or equal to the residual skip threshold of the current frame, determining the hyperprior residual mean of the pixel point as a reconstructed residual feature of the pixel point, and if the hyperprior residual variance of the pixel point is greater than the residual skip threshold of the current frame, determining a reconstructed residual feature of the pixel point on the basis of the residual information; and reconstructing the current frame on the basis of the reconstructed residual features of the pixel points to obtain a reconstructed frame of the current frame.

Description

Video decoding method, video encoding method and apparatus

[0001] Cross-reference to related applications

[0002] This application claims priority to Chinese application No. 202510487845.6, filed on April 17, 2025, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of video encoding and decoding technology. Specifically, it relates to a video decoding method, a video encoding method, and an apparatus. Background Technology

[0004] Current video encoding and decoding schemes rely on manually designed feature extractors, which struggle to effectively identify and extract high-dimensional redundant information from videos. This leads to excessive storage space consumption and wasted transmission resources. With the rapid development of deep learning technology, numerous learning-based encoding and decoding methods have emerged.

[0005] Among them, neural network encoding / decoding based on Variational Autoencoder (VAE) can perform complex nonlinear mapping on pixel-level features of images, thereby transforming them into a latent feature space, and then extracting redundant information in the video to determine key latent features. To alleviate the problem that the spatial coupling relationship of a single neural network is difficult to completely eliminate when processing the latent features of image pixels, a super-prior prediction network is added to the related technology. The super-prior prediction network can extract super-prior features from the determined latent features and decode them into the mean and variance of the latent features. For the mean and variance predicted by the super-prior network, the skip model module determines the closeness of the mean of the super-prior prediction network to the true value based on the relationship between the variance and the skip threshold. If the variance is less than the skip threshold, the mean is used to replace the true value, thereby significantly reducing the bit rate required for video transmission and improving transmission efficiency while ensuring that the loss of video quality is controllable.

[0006] However, different video sequences will be encoded using different coding quality levels based on video quality requirements, transmission stability requirements, network bandwidth, etc. The selected coding quality level has a significant impact on the overall video quality, and videos at different bitrates should use different skip thresholds. It is evident that the relevant technologies do not determine the skip threshold based on the coding quality level, so there is still room for improvement in coding efficiency. Summary of the Invention

[0007] In a first aspect, a video decoding method is provided, comprising: decoding the encoded data of the current frame, determining the residual information of the current frame, and the mean and variance of the prior residuals corresponding to each pixel in the current frame; determining a residual skip threshold for the current frame according to a first quality adjustment parameter, wherein the first quality adjustment parameter is a parameter used to adjust the encoded quality level of the video sequence to which the current frame belongs; for each pixel in the current frame, if the variance of the prior residuals of the pixel is less than or equal to the residual skip threshold of the current frame, then determining the mean of the prior residuals of the pixel as the reconstructed residual feature of the pixel; if the variance of the prior residuals of the pixel is greater than the residual skip threshold of the current frame, then determining the reconstructed residual feature of the pixel according to the residual information; and reconstructing the current frame according to the reconstructed residual features of each pixel to obtain a reconstructed frame of the current frame.

[0008] Secondly, a video coding method is provided, comprising: determining the latent residual features of each pixel in the current frame; quantizing the latent residual features of each pixel in the current frame to obtain quantized residual features of each pixel in the current frame; determining the mean and variance of the prior residuals of each pixel based on the latent residual features of each pixel; determining a residual skip threshold for the current frame based on a first quality adjustment parameter, wherein the first quality adjustment parameter is a parameter used to adjust the coding quality level of the video sequence to which the current frame belongs; and determining the coded data of the current frame based on the residual information of the current frame; wherein the residual information includes the quantized residual features of each pixel whose variance of the prior residuals is greater than the residual skip threshold.

[0009] Thirdly, a decoding chip is provided, including a processor and a memory; the processor is used to call a program or instructions from the memory to cause the method described in the first aspect to be executed.

[0010] Fourthly, a decoder is provided, comprising: a memory for storing a computer program; and a processor for executing the method described in the first aspect when running the computer program.

[0011] Fifthly, an encoder is provided, comprising: a memory for storing a computer program; and a processor for executing the method described in the second aspect when running the computer program.

[0012] In a sixth aspect, an encoding chip is provided, comprising: a processor and a memory; the processor is configured to retrieve a program or instructions from the memory to cause the method described in the second aspect to be executed.

[0013] In a seventh aspect, a computer-readable storage medium is provided, on which a computer program / instructions and a bit stream are stored, wherein the computer program / instructions, when executed by a processor, are capable of generating the bit stream in accordance with the method described in the second aspect.

[0014] Eighthly, a bitstream is provided, including a bitstream generated according to the method described in the second aspect.

[0015] A ninth aspect provides a method for storing a bit stream, comprising: generating a bit stream by performing the method described in the second aspect, and storing the bit stream.

[0016] A tenth aspect provides a method for transmitting a bit stream, comprising: generating a bit stream by performing the method described in the second aspect, and transmitting the bit stream. Attached Figure Description

[0017] To more clearly illustrate the implementation methods in some embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0018] Figure 1 is a schematic diagram of the structure of a video decoder in some embodiments of this application;

[0019] Figure 2 is a schematic diagram of the structure of a video encoder in some embodiments of this application;

[0020] Figure 3 is a schematic diagram of the structure of a video decoding system in some embodiments of this application;

[0021] Figure 4 is a schematic diagram of the skip module in some embodiments of this application;

[0022] Figure 5 is a flowchart of the video decoding method in some embodiments of this application;

[0023] Figure 6 is a flowchart of the video encoding method in some embodiments of this application;

[0024] Figure 7 is a schematic diagram of the structure of a video decoding device in some embodiments of this application;

[0025] Figure 8 is another structural schematic diagram of the video decoding device in some embodiments of this application;

[0026] Figure 9 is a schematic diagram of the structure of a video encoding device in some embodiments of this application;

[0027] Figure 10 is another structural schematic diagram of a video encoding device in some embodiments of this application;

[0028] Figure 11 is a schematic diagram of the hardware structure of an electronic device in some embodiments of this application. Detailed Implementation

[0029] Figure 1 is a schematic diagram of the structure of a video decoder in some embodiments of this application. Referring to Figure 1, the video decoder includes a decompression module 101. The decompression module 101 is used to decompress the encoded data B of the current frame. x Decode the data to determine the mean value μ of the prior residuals for each pixel in the current frame. Y The variance of the super-prior residuals of each pixel in the previous frame The residual information Y of the current frame, and the hyper-prior motion mean μ of each pixel in the previous frame. M The hyperprior motion variance of each pixel in the previous frame and at least one of the motion information M of the current frame.

[0030] Wherein, the mean of the prior residuals of each pixel in the current frame is μ Y and prior residual variance The residual prior features are determined by using a priori coding network to encode the potential residual features of each pixel in the current frame, and then using a priori decoding network to decode these residual prior features. The mean of the priori residual is μ. Y This represents the most likely value of the quantization residual feature of the predicted pixel, and the variance of the prior residual. This represents the quantization residual feature of a pixel in the mean of the prior residual μ. Y The degree of dispersion or scattering in the vicinity. The residual information Y of the current frame includes the variance of the prior residuals. The quantized residual features of each pixel that exceeds the residual skip threshold. The latent residual features of a pixel refer to the residual of the pixel determined in the latent feature domain.

[0031] The hyperprior motion mean μ of each pixel in the current frame M and prior motion variance The motion prior features are determined by encoding the motion vectors of each pixel in the current frame using a priori coding network, and then by decoding these features using a priori decoding network. The mean of the priori residuals is μ. M This represents the most likely value of the quantized motion vector for each predicted pixel, and the variance of the prior residual. This represents the quantized motion vector of each pixel in the time-prior motion mean μ. M The degree of dispersion or scattering in the vicinity. The motion information M of the current frame includes the quantized motion vectors of each pixel whose prior motion variance is greater than the motion skip threshold.

[0032] In some embodiments, the video decoder shown in FIG1 further includes: a residual threshold determination module 102; wherein, the residual threshold determination module 102 is configured to determine a residual skip threshold t for the current frame based on a first quality adjustment parameter and / or a second quality level adjustment parameter.R .

[0033] The first quality adjustment parameter is used to adjust the coding quality level of the video sequence to which the current frame belongs, and the second quality adjustment parameter is used to adjust the coding quality of frames at different positions in the Group of Pictures (GOP) corresponding to the current frame.

[0034] In some embodiments, the video decoder shown in FIG1 further includes: a residual reconstruction module 103; wherein, the residual reconstruction module 103 is used for each pixel of the current frame, if the variance of the super-prior residual of the pixel is... The residual skip threshold t is less than or equal to that of the current frame. R Then the mean of the prior residuals of the pixels is μ. Y Reconstruction residual features identified as pixels If the variance of the prior residual of the pixel Residual greater than the current frame skip threshold t R Then, the reconstructed residual features of the pixels are determined based on the residual information Y.

[0035] In some embodiments, the video decoder shown in FIG1 further includes: a motion threshold determination module 104; wherein, the motion threshold determination module 104 is configured to determine a motion skip threshold t for the current frame based on a first quality adjustment parameter and / or a second quality level adjustment parameter. M .

[0036] It should be noted that the motion threshold determination module 104 determines the motion skip threshold t. M The implementation can be combined with the residual threshold determination module 102 to determine the residual skip threshold t. R The implementation method is the same. Correspondingly, the motion skipping threshold t determined by the motion threshold determination module 104 is... M The residual skip threshold t can be determined by the residual threshold determination module 102. R The motion threshold determination module 104 determines the motion skip threshold in a different way than the residual threshold determination module 102 determines the residual skip threshold. Correspondingly, the motion skip threshold t determined by the motion threshold determination module 104... M It can also be used in conjunction with the residual skip threshold t determined by the residual threshold determination module 102. R They are not equal.

[0037] In some embodiments, the video decoder shown in FIG1 further includes: a motion reconstruction module 105; wherein, the motion reconstruction module 105 is used for each pixel of the current frame, if the hyperprior motion variance of the pixel is... Motion skip threshold t less than or equal to the current frameM Then the hyperprior motion mean μ of the pixel will be... M Determined as the reconstructed motion vector of a pixel If the variance of the prior residual of the pixel Motion skipping threshold t greater than the current frame M Then, the reconstructed motion vector of the pixel is determined based on the motion information M.

[0038] In some embodiments, the video decoder shown in FIG1 further includes: a frame reconstruction module 106; wherein, the frame reconstruction module 106 is used to reconstruct the residual features of each pixel in the current frame based on the residual features output by the residual reconstruction module 103. and the reconstructed motion vectors of each pixel in the current frame output by the motion reconstruction module 105. Reconstruct the current frame to obtain the reconstructed frame of the current frame.

[0039] Figure 2 is a schematic diagram of the structure of a video encoder in some embodiments of this application. The video encoder includes an information extraction module 201. The information extraction module 201 is used to extract information from the current frame x in the latent feature domain. t Residual feature extraction is performed to obtain the latent residual features Y of each pixel in the current frame. t and for the current frame x t Perform motion estimation (ME) to obtain the current video frame x. t The motion vector M of each pixel t Motion estimation is a technique that finds the most similar pixels to a given pixel in a reference frame and calculates the offset between them as a motion vector to describe the motion of objects in a video.

[0040] In some embodiments, the video encoder shown in FIG2 further includes: a residual quantization module 202; wherein, the residual quantization module 202 is used to process the potential residual features Y of each pixel in the current frame. t Perform quantization to determine the current frame x t Quantization residual features y of each pixel t .

[0041] Quantization is a process that uses a specific algorithm to transform continuous sampled values ​​in a video signal into a finite number of discrete values. Its main purpose is to compress the amount of data to the greatest extent possible while ensuring the basic visual quality of the video. By reasonably setting the quantization parameter (QP), the originally wide range of values ​​can be mapped to a few specific values, thereby representing video data with fewer bits and achieving efficient encoding, storage and transmission.

[0042] In some embodiments, the video encoder shown in FIG2 further includes: a residual advanced prior prediction network 203; wherein, the residual advanced prior prediction network 203 is used to predict the potential residual features Y of each pixel in the current frame. t The mean and variance of the Gaussian distribution are used to determine the mean of the super-prior residuals μ for each pixel in the current frame. Y and prior residual variance

[0043] In some embodiments, the video encoder shown in FIG2 further includes: a residual threshold determination module 204; wherein, the residual threshold determination module 204 is configured to determine a residual skip threshold t for the current frame based on a first quality adjustment parameter and / or a second quality level adjustment parameter. Y The first quality adjustment parameter is used to adjust the coding quality level of the video sequence to which the current frame belongs, and the second quality adjustment parameter is used to adjust the coding quality of frames at different positions in the image group corresponding to the current frame.

[0044] In some embodiments, the video encoder shown in FIG2 further includes: a residual skipping module 205; wherein, the residual skipping module 205 is used to calculate the variance of the super-prior residuals of each pixel in the current frame. Mean μ of the prior residuals of each pixel in the current frame Y The residual skip threshold t of the current frame Y and the quantization residual features y of each pixel in the current frame t Determine the residual information Y of the current frame. The residual information Y of the current frame includes: the variance of the prior residual. greater than the residual skip threshold t Y Quantization residual features y of each pixel t .

[0045] It should be noted that, unlike the residual information obtained from a video decoder which only includes the prior residual variance, this information is different. greater than the residual skip threshold t Y The residual information Y obtained by the video encoder includes the residual information of each pixel in the current frame, and the variance of the residuals beyond the prior knowledge is also considered. greater than the residual skip threshold t Y The residual information of each pixel is the quantized residual feature y of the pixel. t The variance of the prior residuals Less than or equal to the residual skip threshold t Y The residual information of each pixel is the mean of the prior residuals of the pixel, μ. Y The residual information obtained from the video decoder only includes the prior residual variance. greater than the residual skip threshold t YThe residual information of each pixel, excluding the variance of the prior residual. Less than or equal to the residual skip threshold t Y The residual information of the pixels is because when performing arithmetic encoding on the residual information, it will not be the mean μ of the prior residual in the residual information. Y Allocate a specific bitrate.

[0046] In some embodiments, the video encoder shown in FIG2 further includes: a motion quantization module 206; wherein, the motion quantization module 206 is used to quantize the motion vector M of each pixel in the current frame. t Quantization is performed to obtain x of the current video frame. t The quantized motion vector m of each pixel t .

[0047] In some embodiments, the video encoder shown in FIG2 further includes: a motion prior prediction network 207; wherein, the motion prior prediction network 207 is used to predict the motion vector M of each pixel in the current frame. t The mean and variance of the Gaussian distribution are used to obtain the hyperprior motion mean μ of each pixel in the current frame. M and the variance of the preceding motion

[0048] In some embodiments, the video encoder shown in FIG2 further includes: a motion threshold determination module 208; wherein, the motion threshold determination module 208 is configured to determine a motion skip threshold t for the current frame based on a first quality adjustment parameter and / or a second quality level adjustment parameter. M The first quality adjustment parameter is used to adjust the coding quality level of the video sequence to which the current frame belongs, and the second quality adjustment parameter is used to adjust the coding quality of frames at different positions in the image group corresponding to the current frame.

[0049] In some embodiments, the video encoder shown in FIG2 further includes a motion skipping module 209; wherein, the motion skipping module 209 is configured to, based on the prior motion variance of each pixel in the current frame... The hyper-prior motion mean μ of each pixel in the current frame M The motion skip threshold t for the current frame M and the quantized motion vector m of each pixel in the current frame t Determine the motion information M of the current frame; the motion information M of the current frame includes: the prior motion variance. Greater than the motion skip threshold t M The quantized motion vector m of each pixel t .

[0050] The motion information M obtained by the video encoder includes the motion information of every pixel in the current frame, and the prior motion variance is also included. Greater than the motion skip threshold t M The motion information of a pixel is the quantized motion vector m of that pixel. t The variance of the prior motion Less than or equal to the motion skip threshold t M The motion information of a pixel is the hyperprior motion mean μ of the pixel. M .

[0051] In some embodiments, the video encoder shown in FIG2 further includes an encoding module 210; wherein the encoding module 210 is configured to generate encoded data B of the current frame based on the residual information Y of the current frame and the motion information M of the current frame. x .

[0052] In some embodiments, the encoding module 210 is specifically used to encode the residual information Y of the current frame to generate residual encoded data of the current frame, encode the motion information M of the current frame to generate motion encoded data of the current frame, and combine the residual encoded data, motion encoded data, residual prior features, and motion prior features of the current frame to determine the encoded data of the current frame. Specifically, the residual prior features and motion prior features of the current frame that determine the encoded data of the current frame are features obtained by performing prior encoding on the residual and motion information of the current frame, respectively. Performing prior decoding on the residual prior features of the current frame yields the mean prior residual value μ of each pixel in the current frame. Y and prior residual variance By performing hyperprior decoding on the hyperprior features of the current frame, the hyperprior motion mean μ of each pixel in the current frame can be obtained. M and prior motion variance

[0053] In some embodiments, standard deviation σ can also be used instead of variance σ. 2 When using standard deviation σ instead of variance σ 2 At that time, the encoded data of the video frame no longer includes the super-prior residual variance of each pixel. and prior motion variance Instead, it includes the standard deviation σ of the prior residuals of each pixel. Y and the prior motion standard deviation σ M Furthermore, when using standard deviation σ instead of variance σ... 2 At that time, the residuals obtained at the decoding and encoding ends skip the threshold t. Y and the motion jump threshold t M Each of the above embodiments should be modified accordingly to have a residual skip threshold t. Y arithmetic square root and the arithmetic square root of the motion jump threshold

[0054] Figure 3 is a schematic diagram of the structure of a decoding system that can be used to perform the video encoding and video decoding methods in some embodiments of this application.

[0055] Referring to Figure 3, the video decoding system may include: a feature extraction module 301; wherein, the feature extraction module 301 is used to extract features from the current frame x. t Perform latent feature extraction to obtain the latent features F of each pixel in the current frame. t .

[0056] The feature extraction module 301 can be a feature extraction network model composed of layer structures such as convolutional layers, pooling layers, activation layers, normalization layers, fully connected layers, and attention mechanism layers. The latent features F of a pixel... t It refers to high-dimensional features that cannot be directly observed from the content of video frames, and can include features such as texture features, edge features, and color features.

[0057] In some embodiments, the video decoding system shown in FIG3 further includes: a residual encoder 302; wherein, the residual encoder 302 is used to determine the optical flow information C of the current frame. t and the latent features F of each pixel in the current frame t Determine the latent residual features Y of each pixel in the current frame. t Optical flow information C of the current frame t This includes optical flow information at multiple scales.

[0058] In some embodiments, the video decoding system shown in FIG3 further includes: a residual quantization module 303; wherein, the residual quantization module 303 is used to process the potential residual features Y of each pixel in the current frame. t Quantization is performed to obtain the quantization residual features y of each pixel in the current frame. t .

[0059] In some embodiments, the video decoding system shown in FIG3 further includes: a residual prior coding network 304; wherein, the residual prior coding network 304 is used to determine the optical flow information C of the current frame. t The residual information Y' of the current frame and the reference frame are used to determine the potential residual features Y of each pixel in the current frame. t Perform super-prior encoding to obtain the residual super-prior features z of each pixel in the current frame.Y The reference frame for the current frame can be the previous video frame in the video sequence.

[0060] In some embodiments, the video decoding system shown in FIG3 further includes: a residual advanced prior decoding network 305; wherein, the residual advanced prior decoding network 305 is used to process the residual advanced prior features z of each pixel in the current frame. Y Decode the data to obtain the mean value μ of the prior residuals for each pixel in the current frame. Y and prior residual variance

[0061] In some embodiments, the video decoding system shown in FIG3 further includes: a residual skipping module 306; wherein, the residual skipping module 306 is used to determine the residual skipping threshold t of the current frame. Y And in the prior residual variance of the pixel The residual skip threshold t is less than or equal to that of the current frame. Y At that time, the mean of the prior residuals μ of the pixel points Y The residual information is determined as the pixel-level residual variance. Residual greater than the current frame skip threshold t Y At that time, the quantization residual feature y of the pixel is... t The residual information of the pixel is determined to obtain the residual information Y of the current frame.

[0062] In some embodiments, the video decoding system shown in FIG3 further includes a motion estimation module 307; wherein, the motion estimation module 307 is used to estimate based on a reference frame. Motion estimation is performed on the current frame to obtain the motion estimation parameters v of each pixel in the current frame. t .

[0063] In some embodiments, the video decoding system shown in FIG3 further includes a motion encoder 308; wherein the motion encoder 308 is used to estimate the motion parameters v of each pixel in the current video. t Reconstructed motion vectors of each pixel in the reference frame The latent features F of each pixel in the current frame t and the reconstructed latent features of each pixel in the reference frame Determine the motion vector M of each pixel in the current frame. t .

[0064] In some embodiments, the video decoding system shown in FIG3 further includes: a motion quantization module 309; wherein, the motion quantization module 309 is used to process the motion vector M of each pixel in the current frame. t Quantization is performed to obtain the quantized motion vector m of each pixel in the current frame.t .

[0065] In some embodiments, the video decoding system shown in FIG3 further includes: a motion prior coding network 310; wherein, the motion prior coding network 310 is used to determine the encoded motion vector mv of each pixel in the reference frame. t-1 The motion vectors M of each pixel in the current frame are compared with the motion information M' of the reference frame. t Perform hyperprior encoding to determine the motion hyperprior features z of each pixel in the current frame. M .

[0066] In some embodiments, the video decoding system shown in FIG3 further includes: a motion hyperpriority decoding network 311; wherein, the motion hyperpriority decoding network 311 is used to process the motion hyperpriority features z of each pixel in the current frame. M Decode the data to obtain the hyperprior motion mean μ of each pixel in the current frame. M and prior motion variance

[0067] In some embodiments, the video decoding system shown in FIG3 further includes a motion skipping module 312; wherein the motion skipping module 312 is used to determine the motion skipping threshold t of the current frame. M If the prior motion variance of a pixel is less than or equal to the motion skip threshold t of the current frame, M In this case, the hyperprior motion mean μ of the pixel is... M The motion information of the pixel is determined when the prior motion variance of the pixel is greater than the motion skip threshold t of the current frame. M In this case, the quantized motion vector m of the pixel is... t The motion information of the pixel is determined to obtain the motion information M of the current frame.

[0068] In some embodiments, the video decoding system shown in FIG3 further includes a compression module 313; wherein, the compression module 313 is used to encode the residual information Y of the current frame to determine the residual coded data B. Y Arithmetic coding is performed on the motion information M of the current frame to determine the motion-coded data B of the current frame. M And based on residual coding data B Y Motion coding data B M Residual prior features z Y and motion prior features z M Generate the encoded data for the current frame. It should be understood that the number of compression modules 313 in this embodiment can be set to one or more, and can be set according to actual business needs. This application does not limit this.

[0069] In some embodiments, the video decoding system shown in FIG3 further includes a motion decoder 314; wherein the motion decoder 314 is used to determine the motion mean μ of each pixel in the current frame based on the prior motion mean μ. M and prior motion variance and the motion-coded data B of the current frame M Determine the reconstructed motion vectors of each pixel in the current frame.

[0070] In some embodiments, the video decoding system shown in FIG3 further includes: a context model 315; wherein, the context model 315 is used to reconstruct the motion vectors of each pixel in the current frame. Reconstructed latent features of each pixel in the reference frame Determine the optical flow information c of the current frame t .

[0071] In some embodiments, the video decoding system shown in FIG3 further includes: a residual decoder 316; wherein, the residual decoder 316 is used to determine the mean of the super-prior residuals μ of each pixel in the current frame. Y and prior residual variance And the residual coded data B of the current frame Y Determine the reconstruction residual features of each pixel in the current frame.

[0072] In some embodiments, the video decoding system shown in FIG3 further includes a motion compensation module 317; wherein, the motion compensation module 317 is used to calculate the reconstructed residual features of each pixel in the current frame. And the optical flow information C of the current frame t Determine the reconstructed latent features of each pixel in the current frame.

[0073] In some embodiments, the video decoding system shown in FIG3 further includes a reconstruction module 318; wherein the reconstruction module 318 is used to reconstruct the latent features of each pixel in the current frame. Reconstruct the current frame to obtain the current frame x. t Reconstructed frames

[0074] Referring to FIG4, the residual skipping module or motion skipping module in the above embodiments may include: a first coefficient determination unit 41;

[0075] The first coefficient determination unit 41 is used to determine the first coefficient f(λ) based on the first quality adjustment parameter λ and the first mapping function.

[0076] It should be understood that the first mapping function in the residual skip module and the motion skip module can be different or the same. Accordingly, the first coefficient f(λ) determined by the residual skip module and the motion skip module based on the first quality adjustment parameter and the first mapping function can be the same or different.

[0077] In some embodiments, the residual skipping module or motion skipping module in the above embodiments may include: a second coefficient determination unit 42;

[0078] The second coefficient determination unit 42 is used to determine the second coefficient g(Δλ) based on the second mass adjustment parameter Δλ and the second mapping function.

[0079] Correspondingly, the second mapping function in the residual skip module and the motion skip module can be different or the same, and the second coefficient g(Δλ) determined by the residual skip module and the motion skip module based on the second quality adjustment parameter and the second mapping function can be the same or different.

[0080] In some embodiments, the residual skipping module or motion skipping module in the above embodiments may include: a skipping threshold determination unit 43;

[0081] The skip threshold determination unit 43 is used to determine the skip threshold (i.e., the aforementioned residual skip threshold or motion skip threshold) based on the first coefficient f(λ) and the second coefficient g(Δλ).

[0082] Referring to FIG4, the residual skipping module or motion skipping module in the above embodiments may further include: a discrimination unit 44;

[0083] The discrimination unit in the residual skip module is used to determine the mean of the prior residual of a pixel as the residual information of the pixel when the variance of the prior residual of the pixel is less than or equal to the residual feature skip threshold of the current frame; and to determine the quantized residual feature of the pixel as the residual information of the pixel when the variance of the prior residual of the pixel is greater than the residual feature skip threshold of the current frame.

[0084] The discrimination unit in the motion skip module is used to determine the mean of the prior motion of a pixel as the motion information of the pixel when the prior motion variance of the pixel is less than or equal to the motion vector skip threshold of the current frame; and to determine the quantized motion vector of the pixel as the motion information of the pixel when the prior motion variance of the pixel is greater than the motion vector skip threshold of the current frame.

[0085] Some embodiments of this application provide a video decoding method applied to the video decoder shown in FIG2 above; specifically as shown in FIG5, the video decoding method may include the following steps:

[0086] S51, decode the encoded data of the current frame, determine the residual information of the current frame, and the mean and variance of the prior residuals corresponding to each pixel in the current frame.

[0087] In some embodiments, the encoded data of the current frame can be determined by communicating with a media resource server.

[0088] In some embodiments, the encoded data of the current frame can be read from a specified storage space.

[0089] In some embodiments, the encoded data includes residual prior features obtained by performing prior coding on the residual information of the current frame and residual encoded data obtained by performing arithmetic coding on the residual information of the current frame. Decoding the encoded data of the current frame to determine the mean and variance of the prior residuals for each pixel in the current frame, as well as the residual information of the current frame, includes: performing prior decoding on the residual prior features to determine the mean and variance of the prior residuals for each pixel in the current frame, and performing arithmetic decoding on the residual encoded data to determine the residual information of the current frame.

[0090] In some embodiments of this application, the decoding end determines the mean and variance of the super-prior residuals of each pixel in the current frame by decoding the encoded data of the current frame, while the encoding end determines the mean and variance of the super-prior residuals of each pixel in the current frame by performing super-prior encoding on the potential residual features of each pixel in the current video frame, and then decoding the obtained residual super-prior features to obtain the mean and variance.

[0091] In some embodiments, the residual information of the current frame includes the quantized residual features of each pixel whose prior residual variance is greater than the residual skip threshold. That is, the residual information of the current frame only includes the quantized residual features of pixels whose prior residual variance is greater than the residual skip threshold, and does not include the residual information of pixels whose prior residual variance is less than or equal to the residual skip threshold.

[0092] S52, determine the residual skip threshold of the current frame according to the first quality adjustment parameter, wherein the first quality adjustment parameter is a parameter used to adjust the coding quality level of the video sequence to which the current frame belongs;

[0093] In some embodiments, the first mass adjustment parameter is a Lagrange multiplier.

[0094] Video coding needs to maintain good video quality while minimizing bitrate. However, bitrate and video quality are mutually restrictive factors. To balance bitrate and video quality, many optimization strategies have been introduced in the field of video coding, among which the Lagrange function plays a key role. The Lagrange function is a function constructed by combining bitrate and video quality distortion, providing support for finding a balance between video quality and bitrate in complex coding environments. The general form of the Lagrange function is shown in the following formula (1): J=λD+R (1)

[0095] Where J is the value of the Lagrange function, λ is the Lagrange multiplier, D represents the distortion between the original video frame and the reconstructed video frame, and R represents the bit rate of the video frame.

[0096] As shown in equation (1) above, the selection of the Lagrange multiplier will affect the overall coding quality of the video sequence. Therefore, in some embodiments of this application, the first quality adjustment parameter can be the Lagrange multiplier.

[0097] Since the selection of the first quality adjustment parameter affects the overall coding quality of the video sequence, and the residual skip threshold also affects the coding quality of the video frame, determining the residual skip threshold of the current frame based on the first quality adjustment parameter can make the residual skip threshold of the current frame compatible with the overall coding quality of the video sequence, thereby improving the coding efficiency of the video.

[0098] S53, for each pixel in the current frame, if the variance of the prior residual of the pixel is less than or equal to the residual skip threshold of the current frame, then the mean of the prior residual of the pixel is determined as the reconstruction residual feature of the pixel; if the variance of the prior residual of the pixel is greater than the residual skip threshold of the current frame, then the reconstruction residual feature of the pixel is determined according to the residual information.

[0099] That is, for any pixel, if the variance of the prior residual of the pixel is less than or equal to the residual skip threshold of the current frame, then it is not necessary to determine the reconstruction residual feature of the pixel based on the residual information. Instead, the mean of the prior residual of the pixel is determined as the reconstruction residual feature of the pixel. Conversely, if the variance of the prior residual of the pixel is greater than the residual skip threshold of the current frame, then the reconstruction residual feature of the pixel is determined based on the residual information.

[0100] In some embodiments, determining the reconstructed residual features of a pixel based on residual information includes: determining the quantization residual features of a pixel based on residual information, and performing inverse quantization on the quantization residual features to obtain the reconstructed residual features of the pixel.

[0101] The quantization residual feature of a pixel is represented as y. t The reconstructed residual features of pixels are represented as The mean of the prior residuals of a pixel is represented as μ. Y The variance of the super-prior residuals of a pixel is expressed as The residual skip threshold is represented as sk i Then the reconstructed residual features of the pixels can be determined by the following formula (2):

[0102] Here, IQ() represents the dequantization operation.

[0103] In practice, the reconstruction residual features of each pixel in the current frame can be determined through the following steps S531 and S532:

[0104] S531 generates a mask feature tensor based on the residual skip threshold and the variance of the super-prior residuals of each pixel in the current frame.

[0105] The mask feature tensor includes the mask feature values ​​of each pixel in the current frame. For each pixel in the current frame, if the variance of the prior residual is less than or equal to the residual skip threshold of the current frame, the mask feature value of the pixel is 1. If the variance of the prior residual is greater than the residual skip threshold of the current frame, the mask feature value of the pixel is 0.

[0106] That is, the variance of the prior residuals of the pixels is expressed as The residual skip threshold is represented as sk i If the pixel mask feature value is represented as mask, then the pixel mask feature value can be determined by the following formula (3):

[0107] S532, based on the mask feature values ​​of each pixel in the mask feature tensor, the residual information of the current frame, the mean of the super-prior residuals of each pixel in the current frame, and the following formula (4), determine the reconstructed residual features of each pixel in the current frame:

[0108] Among them, y t These are the quantized residual features of pixels read from the residual information of the current frame. μ represents the residual features of pixel reconstruction. Y is the mean of the prior residuals of the pixel, and mask is the mask feature value of the pixel.

[0109] The above formula can also be used to determine the reconstructed residual feature of a pixel as the mean of the prior residual when the variance of the prior residual of a pixel is less than or equal to the residual skip threshold of the current frame, and to determine the reconstructed residual feature of a pixel based on the residual information (quantized residual feature) when the variance of the prior residual of a pixel is greater than the residual skip threshold of the current frame.

[0110] S54, Reconstruct the current frame based on the reconstruction residual features of each pixel to obtain the reconstructed frame of the current frame.

[0111] In some embodiments, reconstructing the current frame based on the reconstruction residual features of each pixel in the current frame may include: reconstructing the current frame based on the reconstruction residual features of each pixel in the current frame and the reconstruction motion vector of each pixel in the current frame.

[0112] The method for determining the reconstructed motion vectors of each pixel in the current frame can be as follows: the video encoder determines the motion vectors of each pixel in the current frame and encodes them using a relevant technical encoding scheme to determine the motion-coded data of the current frame; the video decoder then decodes the motion-coded data using a corresponding decoding scheme to obtain the reconstructed motion vectors of each pixel in the current frame. That is, in some embodiments of this application, instead of using a skip mode to encode and reconstruct the motion vectors of each pixel in the current frame, relevant technical means can be used to encode and reconstruct the motion vectors of each pixel in the current frame.

[0113] In the video decoding method of some embodiments of this application, the encoded data of the current frame is first decoded to determine the mean of the prior residuals of each pixel in the current frame, the variance of the prior residuals of each pixel in the current frame, and the residual information of the current frame. Then, a residual skip threshold for the current frame is determined according to a first quality adjustment parameter used to adjust the encoded quality level of the video sequence to which the current frame belongs. For each pixel in the current frame, if the variance of the prior residuals of the pixel is less than or equal to the residual skip threshold of the current frame, the mean of the prior residuals of the pixel is determined as the reconstructed residual feature of the pixel. If the variance of the prior residuals of the pixel is greater than the residual skip threshold of the current frame, the reconstructed residual feature of the pixel is determined according to the residual information, thereby determining the reconstructed residual features of each pixel in the current frame. Finally, the current frame is reconstructed according to the reconstructed residual features of each pixel in the current frame to obtain the reconstructed frame of the current frame.

[0114] It should be understood that, since the video decoding method of some embodiments of this application can determine the residual skip threshold of the current frame based on the first quality adjustment parameter used to adjust the coding quality level of the video sequence to which the current frame belongs, some embodiments of this application can perform video coding based on the residual skip threshold adapted to the quality level, and specifically optimize the bitrate allocation under different coding quality levels. Therefore, some embodiments of this application can improve the coding efficiency of video.

[0115] Specifically, as mentioned earlier, the residual skipping threshold in related technologies is usually set to a fixed value, without considering the impact of the coding quality level used in the current video sequence on the importance of residual information. When a video sequence is encoded using different first quality adjustment parameters, their corresponding rate-distortion optimization objectives differ: a smaller first quality adjustment parameter corresponds to higher reconstruction quality requirements. In this case, even pixels with small prior residual variances may still contain information that contributes to reconstruction quality in their true residuals. If a fixed residual skipping threshold is used, these residuals may be incorrectly skipped, resulting in a decrease in reconstruction quality. Conversely, a larger first quality adjustment parameter corresponds to lower bitrate requirements. In this case, some pixels with slightly larger prior residual variances than the fixed threshold have limited residual information that improves the overall perceived quality, but they still need to be transmitted because they are not skipped, resulting in wasted bitrate.

[0116] Based on this, the residual skipping threshold for the current frame is determined using a first quality adjustment parameter in the above process, ensuring that the residual skipping decision matches the coding quality level of the current video sequence. When the first quality adjustment parameter is small, the determined residual skipping threshold value is relatively low, thereby increasing the sensitivity to residual information preservation. In this embodiment, skipping is only performed on pixels with extremely small prior residual variance to preserve more residual details. When the first quality adjustment parameter is large, the determined residual skipping threshold value is relatively large, thus relaxing the skipping conditions and allowing skipping on pixels with relatively large prior residual variance, reducing the amount of residual information to be transmitted. Therefore, the residual skipping strategy can be dynamically adjusted under different coding quality levels, avoiding quality loss or bitrate redundancy caused by using a fixed skipping threshold, thereby improving overall coding efficiency.

[0117] Some embodiments of this application also provide another video decoding method, which includes the following steps S61 to S66:

[0118] S61, decode the encoded data of the current frame, determine the residual information of the current frame, and the mean and variance of the prior residuals corresponding to each pixel in the current frame.

[0119] Step S61 is the same as step S51 mentioned above, and will not be described again here.

[0120] S62, determine the first quality adjustment parameter;

[0121] In step S62, the method for determining the first quality adjustment parameter can refer to the aforementioned step S52, and will not be repeated here.

[0122] S63, determine a second quality adjustment parameter based on the position of the current frame in its respective image group. The second quality adjustment parameter is a parameter corresponding to the current frame used to adjust the coding quality of frames at different positions in the image group.

[0123] When using a low-delay (LD) compression framework for video compression, only the first frame is transmitted using a high-quality image compression method, while the remaining frames are transmitted using motion estimation and motion compensation to achieve high-speed video transmission.

[0124] While the LD framework enables high-speed video transmission, it also leads to the continuous accumulation of errors. Current solutions involve introducing Group of Pictures (GOPs) into the video to ensure the quality stability of later frames during long encoding periods. GOP technology sets the earlier frames within a group as high-quality frames and the later frames as low-quality frames, thus preventing the cumulative propagation of errors. For example, the input video sequence can be grouped with a fixed step size, with the first frame of each group set to a higher video quality level and the remaining frames in the group set to lower quality levels, thereby achieving a balance between bitrate and quality. The encoding quality parameters corresponding to the current frame, used to adjust the frames at different positions within the GOP, are the second quality adjustment parameters in some embodiments of this application.

[0125] As mentioned above, the second quality adjustment parameter also affects the encoding quality of the current frame. Therefore, determining the second quality adjustment parameter and setting the residual skip threshold based on the second quality adjustment parameter can further improve the encoding efficiency of the video.

[0126] In some embodiments, determining a second quality adjustment parameter based on the position of the current frame in its respective image group includes: if the current frame is the first video frame in its respective image group, then determining the second quality adjustment parameter as a first value; if the current frame is not the first video frame in its respective image group, then determining the second quality adjustment parameter as a second value.

[0127] S64, determine the residual skip threshold for the current frame based on the first quality adjustment parameter and the second quality adjustment parameter;

[0128] In the coding compression model, two quality adjustment parameters are generally introduced. One quality adjustment parameter is λ, which controls the video sequence to be encoded at different coding quality levels (different bitrate points). The other quality adjustment parameter is Δλ, which controls the video frames at different positions within the GOP to be encoded at different coding quality levels after the overall coding quality level of the video sequence has been set. The influence of the quality adjustment parameter λ and the quality adjustment parameter Δλ on the video coding quality is shown in the following formula (5):

[0129] Where J is the value of the Lagrange function. The distortion is calculated based on the original video frames and the reconstructed video frames. The bitrate is calculated based on the likelihood of latent features and prior parameters. The bitrate is calculated based on the likelihood of the motion vector and the hyperprior parameters. This indicates the bitrate of the video frame.

[0130] Likelihood measures the probability of a given piece of encoded data (such as latent features or motion vectors) occurring given prior information. In video encoding and decoding, this probability is often represented by probability density; a higher probability density indicates that the data is more predictable and requires a lower encoding bitrate. In the embodiments of this application... In other words, given the motion prior parameters, the probability density value obtained by probabilistically modeling the motion vector of the current frame is obtained by the prior network performing contextual modeling of the motion vector based on the decoded motion prior features and outputting its conditional probability distribution.

[0131] The above embodiments determine the residual skip threshold for the current frame based on two quality adjustment parameters (a first quality adjustment parameter and a second quality adjustment parameter) that affect the video coding quality. Therefore, the above embodiments can comprehensively consider the relationship between video quality variation and the residual skip threshold, thereby further improving the video coding efficiency.

[0132] In some embodiments, determining the residual skip threshold of the current frame based on the first quality adjustment parameter and the second quality adjustment parameter may include the following steps S641 to S644:

[0133] S641, determine the first coefficient based on the first quality adjustment parameter and the first mapping function.

[0134] In some embodiments, the first mapping function may be as shown in equation (6):

[0135] Where f(λ) is the first coefficient, λ1, λ2, λ3, ... are different first mass adjustment parameters, and A, B, C, ... are the first coefficients corresponding to different first mass adjustment parameters. The first coefficient can be obtained by substituting the first mass adjustment parameter λ into formula (6).

[0136] S642, determine the second coefficient based on the second quality adjustment parameter and the second mapping function.

[0137] In practice, the second quality adjustment parameter is first determined based on the position of the current frame in its respective image group;

[0138] Specifically, if the current frame is the first video frame in its respective image group, the second quality adjustment parameter is determined to be the first value; if the current frame is not the first video frame in its respective image group, the second quality adjustment parameter is determined to be the second value.

[0139] It should be noted that the first and second values ​​are fixed values ​​preset based on the frame position in the group of pictures (GOP) and the preset requirements for coding quality; the first value is greater than the second value.

[0140] Specifically, if the current frame is the first video frame in its group of images, it serves as a reference frame for other frames within the group and requires a higher coding quality to suppress error accumulation. Therefore, a smaller first value is needed as the second quality adjustment parameter. Conversely, if the current frame is not the first video frame in its group of images, it relies on the reference frame for prediction and can tolerate a lower coding quality to save bitrate. Therefore, a larger second value is needed as the second quality adjustment parameter.

[0141] Since the residual skip threshold in this application is positively correlated with the quality adjustment parameter (i.e., the larger the quality adjustment parameter, the higher the skip threshold, the more aggressive the skipping, and the lower the reconstruction quality), the first value used for high-quality reference frames should be smaller than the second value used for non-reference frames.

[0142] In some embodiments, if the current frame is the first video frame in its respective image group, the first value is mapped to a third value using a second mapping function, and this third value is used as a second coefficient. Conversely, if the current frame is not the first video frame in its respective image group, the second value is mapped to a fourth value using a second mapping function, and this fourth value is used as a second coefficient. The third value is less than the fourth value.

[0143] Specifically, the second mapping function is a monotonically decreasing function. Since the first value is greater than the second value, the third value obtained by inputting the first value into the second mapping function will necessarily be less than the fourth value obtained by inputting the second value into the second mapping function. The second coefficient is determined by either the third or fourth value, hence the third value is less than the fourth value. This design results in a smaller second coefficient corresponding to the first video frame in the image group, thereby reducing its residual skip threshold, minimizing residual skipping behavior, and ensuring the reconstruction accuracy of the reference frame. For non-first video frames, a larger second coefficient increases their residual skip threshold, allowing more pixels to skip residual decoding, thus saving bitrate and improving compression efficiency.

[0144] In this embodiment, the second mapping function can be expressed as shown in equation (7):

[0145] Where g(Δλ) is the second coefficient, Δλ1 is the first value, Δλ2 is the second value, a is the third value, and b is the fourth value.

[0146] In other embodiments, if the current frame is the first video frame in its respective image group, the first value is mapped to a third value using a second mapping function, and this third value is used as the second coefficient. Conversely, if the current frame is not the first video frame in its respective image group, the second coefficient is determined based on the second mapping function, the second value, and the current frame's index in the image group. Specifically, when the current frame is not the first video frame in its respective image group, the current frame's index in the image group is positively correlated with the second coefficient, and the second coefficient determined based on the index is always greater than the third value.

[0147] Specifically, considering that the compression strategy for non-first video frames within a given image group can be further refined based on their specific position within the group to more fully utilize the redundancy characteristics in the video's temporal structure, this application can also use the current frame's index within the image group as an additional control factor, inputting it along with the second value into the second mapping function to obtain the second coefficient. Since the current frame's index within the image group reflects its temporal proximity to the reference frame, incorporating it into the calculation of the second coefficient allows the residual skip threshold to dynamically change with the frame position. This enables progressive residual skip control for non-first video frames at different positions within the image group while maintaining visual continuity. This method is particularly suitable for applications requiring high bitrate allocation accuracy, further improving compression efficiency while maintaining overall reconstruction quality.

[0148] In this embodiment, the second mapping function can be expressed as shown in equation (8):

[0149] Where Q2 is the second coefficient, Δλ1 is the first value, Δλ2 is the second value, a is the third value, n is the sequence number of the current frame in its respective image group, and a < g(Δλ2,n), when i is greater than j, g(Δλ2,i) > g(Δλ2,j).

[0150] S643, summing the first and second coefficients, yields the residual threshold multiplier.

[0151] Let the first coefficient be represented as Q1, the second coefficient as Q2, and the residual threshold multiplier as k. Then the residual threshold multiplier can be calculated as shown in the following equation (9): k=f(λ)+g(Δλ) (9)

[0152] S644, calculate the residual skip threshold for the current frame based on the residual threshold multiplier.

[0153] In some embodiments, calculating the residual skip threshold of the current frame based on the residual threshold multiplier includes: calculating the product of the residual threshold multiplier and the value of the residual skip threshold function to obtain the residual skip threshold of the current frame. The residual skip threshold function is a function with the frame index of the current frame as its independent variable, and the residual skip threshold function is monotonically increasing within the domain of the frame index of the video sequence to which the current frame belongs.

[0154] In some embodiments, the residual skip threshold function is given by the following equation (10):

[0155] Where i is the temporal sequence of the current frame, n is the total number of video frames in the video sequence to be encoded, and e is a constant.

[0156] Let sk represent the residual skip threshold of video frame i with time sequence i. i Then the formula for calculating the residual skip threshold can be shown in equation (11):

[0157] S65, for each pixel in the current frame, if the variance of the prior residual of the pixel is less than or equal to the residual skip threshold of the current frame, then the mean of the prior residual of the pixel is determined as the reconstruction residual feature of the pixel; if the variance of the prior residual of the pixel is greater than the residual skip threshold of the current frame, then the reconstruction residual feature of the pixel is determined according to the residual information.

[0158] S66, Reconstruct the current frame based on the reconstruction residual features of each pixel in the current frame to obtain the reconstructed frame of the current frame.

[0159] Similarly, reconstructing the current frame based on the reconstruction residual features of each pixel in the current frame can include: reconstructing the current frame based on the reconstruction residual features of each pixel in the current frame and the reconstruction motion vectors of each pixel in the current frame. The method for determining the reconstruction motion vectors of each pixel in the current frame can also be the same as in the above embodiments, and will not be repeated here to avoid redundancy.

[0160] In the above process, when the current frame is the first video frame in its image group, the second quality adjustment parameter is determined to be the first value. The first value is mapped to the third value through the second mapping function, and the third value is used as the second coefficient. This reduces the residual skip threshold of the current frame, thereby reducing the skipping of residual information in that frame and ensuring the reconstruction accuracy of the current frame. Correspondingly, when the current frame is not the first video frame in its image group, the second quality adjustment parameter is determined to be the second value. The second value is mapped to the fourth value through the second mapping function, and the fourth value is used as the second coefficient. This increases the residual skip threshold of the current frame, thereby reducing the amount of residual information to be transmitted by performing a skip operation on pixels with relatively large prior residual variance.

[0161] Furthermore, when the current frame is not the first video frame in its respective image group, a second coefficient is determined based on the second mapping function, the second value, and the current frame's index within the image group. This allows the residual skip threshold to be dynamically adjusted according to the frame's position within the image group, resulting in a more precise match to the temporal redundancy characteristics of the video. Thus, while adapting to the overall coding quality level of the video sequence, this approach further refines the control of the residual skip threshold by considering the current frame's position within its image group. This reduces error accumulation caused by insufficient reconstruction accuracy of the reference frame, thereby achieving superior rate-distortion performance and coding efficiency.

[0162] Some embodiments of this application also provide another video decoding method, which includes the following steps S71 to S75:

[0163] S71, decode the encoded data of the current frame, determine the residual information and motion information of the current frame, as well as the mean of the prior residuals, the variance of the prior residuals, the mean of the prior motion, and the variance of the prior motion for each pixel in the current frame.

[0164] S72, based on the mean of the prior residuals of each pixel in the current frame, the variance of the prior residuals, and the residual information of the current frame, determine the reconstruction residual features of each pixel.

[0165] The method for determining the reconstruction residual features of each pixel in the current frame based on the mean and variance of the prior residuals of each pixel and the residual information of the current frame can refer to steps S62 to S65 above. To avoid redundancy, it will not be repeated here.

[0166] S73, determine the motion skip threshold for the current frame based on the first quality adjustment parameter.

[0167] In some embodiments, determining the motion skip threshold of the current frame based on a first quality adjustment parameter includes: determining a second quality adjustment parameter based on the position of the current frame within its image group, and determining the motion skip threshold of the current frame based on the first and second quality adjustment parameters. The second quality adjustment parameter is a parameter corresponding to the current frame used to adjust the encoding quality of frames at different positions within the image group.

[0168] In some embodiments, determining the motion skip threshold of the current frame based on the first quality adjustment parameter and the second quality adjustment parameter may include the following steps S731 to S734:

[0169] S731, determine the third coefficient based on the first quality adjustment parameter and the third mapping function.

[0170] In some embodiments of this application, the third mapping function may be the same as or different from the first mapping function. When the third mapping function is the same as the first mapping function, the first coefficient and the third coefficient are equal; when the third mapping function is different from the first mapping function, the first coefficient and the third coefficient are not equal.

[0171] S732, determine the fourth coefficient based on the second quality adjustment parameter and the fourth mapping function.

[0172] In some embodiments, determining a second quality adjustment parameter based on the position of the current frame within its respective image group includes: if the current frame is the first video frame in its respective image group, determining the second quality adjustment parameter as a first value; if the current frame is not the first video frame in its respective image group, determining the second quality adjustment parameter as a second value; and determining a fourth coefficient based on the second quality adjustment parameter and a fourth mapping function includes: if the current frame is the first video frame in its respective image group, mapping a third value to a fifth value using the fourth mapping function to obtain a fourth coefficient; if the current frame is not the first video frame in its respective image group, mapping a fifth value to a sixth value using the fourth mapping function to obtain a second coefficient. The fifth value is less than the sixth value.

[0173] In some embodiments, the fourth mapping function is the same as the second mapping function. Accordingly, the fifth value is equal to the third value, and the sixth value is equal to the fourth value.

[0174] In some embodiments, determining a second quality adjustment parameter based on the position of the current frame in its respective image group includes: if the current frame is the first video frame in its respective image group, then determining the second quality adjustment parameter as a first value; if the current frame is not the first video frame in its respective image group, then determining the second quality adjustment parameter as a second value; determining a fourth coefficient based on the second quality adjustment parameter and a fourth mapping function includes: if the current frame is the first video frame in its respective image group, then mapping the first value to a fifth value through the fourth mapping function to obtain the fourth coefficient; if the current frame is not the first video frame in its respective image group, then determining the fourth coefficient based on the fourth mapping function, the second value, and the sequence number of the current frame in its respective image group; wherein, when the current frame is not the first video frame in its respective image group, the sequence number of the current frame in its respective image group is positively correlated with the second coefficient, and the determined fourth coefficient is always greater than the fifth value.

[0175] S733, summing the third and fourth coefficients, yields the motion threshold multiplier.

[0176] S734, calculates the motion skip threshold for the current frame based on the motion threshold multiplier.

[0177] In some embodiments, calculating the motion skip threshold of the current frame based on the motion threshold multiplier includes: calculating the product of the motion threshold multiplier and the value of the motion skip threshold function to obtain the motion skip threshold of the current frame;

[0178] The motion skip threshold function is a function with the frame index of the current frame as the independent variable, and the motion skip threshold function is monotonically increasing within the domain of the frame index of the video sequence to which the current frame belongs.

[0179] In some embodiments, the motion skipping threshold function is given by the following equation (12):

[0180] Where i is the temporal sequence of the current frame, n is the total number of video frames in the video sequence to be encoded, and e is a constant.

[0181] S74, for each pixel in the current frame, if the prior motion variance of the pixel is less than or equal to the motion skip threshold of the current frame, then the prior motion mean of the pixel is determined as the reconstructed motion vector of the pixel; if the prior residual variance of the pixel is greater than the motion skip threshold of the current frame, then the reconstructed motion vector of the pixel is determined according to the motion information.

[0182] S75, the current frame is reconstructed based on the reconstruction residual features and reconstruction motion vectors of each pixel in the current frame to obtain the reconstructed frame.

[0183] In some embodiments, reconstructing the current frame based on the reconstruction residual features and reconstruction motion vectors of each pixel in the current frame includes the following steps: determining the predicted features of each pixel in the current frame based on the reconstruction motion vectors of each pixel in the current frame; determining the potential reconstruction features of each pixel in the current frame based on the reconstruction residual features and the predicted features of each pixel in the current frame; and reconstructing the current frame based on the potential reconstruction features of each pixel in the current frame.

[0184] Specifically, as mentioned earlier, related technologies typically do not introduce a skipping mechanism for motion information, or even if they do, the motion skipping threshold is set to a fixed value, failing to consider the influence of the coding quality level used in the current video sequence and the position of the current frame in its respective image group on the importance of motion information. When a video sequence is encoded using different first quality adjustment parameters, its corresponding rate-distortion optimization objectives differ: a smaller first quality adjustment parameter corresponds to higher reconstruction quality requirements. In this case, even pixels with small prior motion variance may still contain information in their true motion vectors that contributes to motion compensation accuracy. If a fixed motion skipping threshold is used, this part of the motion information may be incorrectly skipped, resulting in inaccurate prediction and a decrease in reconstruction quality. Conversely, a larger first quality adjustment parameter corresponds to lower bitrate requirements. In this case, some pixels with slightly larger prior motion variance than the fixed threshold have limited contribution to the overall motion compensation effect, but because they are not skipped, they still need to be transmitted, resulting in wasted motion bitrate.

[0185] Furthermore, in the coding framework that adopts the group of pictures structure, the degree of dependence on motion information varies among frames at different positions within a group of pictures. For example, the first video frame in a group of pictures serves as the motion reference for subsequent frames and requires higher motion vector precision to suppress error propagation, while other frames can tolerate relatively coarse motion information to save bitrate. However, related technologies lack adaptation mechanisms for such temporal role differences.

[0186] Based on this, the motion skipping threshold for the current frame is jointly determined by the first quality adjustment parameter and the second quality adjustment parameter determined according to the position of the current frame in its respective image group. This allows the motion skipping decision to simultaneously adapt to the overall coding quality level of the video sequence and the role of the current frame in the image group. When the first quality adjustment parameter is small, the determined motion skipping threshold value is relatively low, which improves the sensitivity to motion information preservation. Based on this, skipping operations are performed only on pixels with minimal prior motion variance to preserve more accurate reconstructed motion vectors. Conversely, when the first quality adjustment parameter is large, the determined motion skipping threshold value is relatively large, thereby relaxing the skipping conditions. Skipping operations are performed on pixels with relatively large prior motion variance to reduce the amount of transmitted information.

[0187] Furthermore, when the current frame is the first video frame in its respective image group, the second quality adjustment parameter is determined to be the first value. After mapping by the fourth mapping function, a smaller fourth coefficient is obtained, which lowers the motion skip threshold, reduces the skipping of motion information in that frame, and ensures the motion prediction accuracy of it as a reference frame. When the current frame is not the first video frame in its respective image group, the second quality adjustment parameter is determined to be the second value. After mapping by the fourth mapping function, a larger fourth coefficient is obtained, which raises the motion skip threshold, allowing more pixels to skip motion information decoding and effectively saving bitrate. Thus, adaptive skip control that works in conjunction with residual information can be implemented for motion information under different coding quality levels and image group structures, reducing prediction inaccuracies or motion bitrate redundancy caused by using a fixed motion skip threshold, thereby improving video coding efficiency.

[0188] Some embodiments of this application provide a video encoding method. Referring to FIG6, the video encoding method may include the following steps:

[0189] S81, determine the potential residual features of each pixel in the current frame.

[0190] In video coding technology, pixel residuals are a key indicator for measuring the accuracy of coding predictions. They are defined as the difference between the actual value of a pixel and the predicted value obtained using a specific prediction method. Taking frame-based video coding as an example, the value of pixels within the current frame is predicted using a reference frame; the difference between the actual value and the predicted value is the residual for that pixel. Latent residual features further deepen the concept of residuals. They first transform the actual value of a pixel into a feature value in the latent domain. This feature value is an abstract expression containing the intrinsic characteristics and related information of the pixel. The difference between this feature value and the predicted feature is calculated; the information contained in this difference is the latent residual feature of the pixel.

[0191] For video compression tasks, the pixel domain, due to its low dimensionality, is often insufficient to represent enough local and global feature information. Encoding that performs motion estimation and compensation on video frames in the pixel domain before compression yields poor results. The main reason is that in the pixel domain, the data representation of video frames only reflects the original luminance or chrominance values, lacking explicit modeling of high-level features such as image structure, texture semantics, and cross-frame temporal correlations. This makes it difficult for motion estimation and compensation processes to accurately capture complex motions and subtle changes, resulting in high residual signal energy and a cluttered spatial distribution, which is detrimental to efficient compression.

[0192] In some embodiments of this application, video frames are mapped to a latent domain through an encoder network. The resulting latent features possess higher semantic abstraction capabilities and compactness. This allows the latent features to integrate local texture details with global contextual information, forming a more robust representation of the video content. Simultaneously, motion estimation and compensation can be performed in the latent domain, leveraging feature-level similarity to achieve more accurate inter-frame prediction, thereby significantly reducing residual energy. This enhances the compressibility of the residual signal, thus improving video coding performance.

[0193] For example, the latent residual features of video frames can be extracted using a neural compressor based on a variational autoencoder (VAE) architecture, which provides the latent residual features of each pixel in the video frame.

[0194] S82 quantizes the potential residual features of each pixel in the current frame to obtain the quantized residual features of each pixel.

[0195] Quantizing the latent residual features of each pixel in the current frame refers to the process of mapping continuous latent residual features to a finite number of discrete feature values. Quantizing the latent residual features of each pixel in the current frame can reduce the amount of data encoded for the latent residual features, but it will reduce the representation accuracy of the latent residual features.

[0196] S83, based on the potential residual characteristics of each pixel in the current frame, determine the mean of the prior residual and the variance of the prior residual for each pixel.

[0197] In some embodiments, a super-prior coding method can be employed, using a pre-trained residual super-prior coding network to predict the super-prior information of the potential residual features of each pixel in the current frame. The super-prior information of the potential residual features, after quantization, can approximate the residual information actually transmitted to the decoding end. Therefore, directly decoding the super-prior information of the potential residual features of each pixel in the current frame yields the mean and variance of the potential residual feature distribution of each pixel in the current frame. The mean and variance of the potential residual feature distribution of each pixel in the current frame are the mean and variance of the super-prior residuals of each pixel. The mean of the super-prior residuals of each pixel can represent the most likely value of the potential residual feature of each pixel, and the variance of the super-prior residuals of each pixel can represent the degree of dispersion or dispersion of the potential residual feature of each pixel around the mean.

[0198] S84, determine the residual skip threshold for the current frame based on the first quality adjustment parameter.

[0199] The method for determining the residual skip threshold of the current frame based on the first quality adjustment parameter is the same as the method for determining the residual skip threshold of the current frame based on the first quality adjustment parameter in the video decoding method described above. To avoid redundancy, it will not be repeated here.

[0200] S85 determines the encoded data of the current frame based on the residual information of the current frame.

[0201] The residual information includes the quantized residual features of each pixel whose prior residual variance is greater than the residual skip threshold.

[0202] To balance video quality and bitrate during transmission, and to achieve Rate-Distortion Optimization (RDO), some embodiments of this application replace the ground truth value of the quantization residual features of some pixels with the mean of the super-prior residuals. For each pixel, whether to trigger the replacement of the quantization residual feature with the mean of the super-prior residuals is determined by comparing the variance of the super-prior residuals with the residual skip threshold of the current frame. If the variance of the super-prior residuals is less than or equal to the residual skip threshold, it is considered that the ground truth value of the quantization residual feature will appear near the mean of the super-prior residuals. The mean of the super-prior residuals can approximate the ground truth value of the quantization residual feature and can be used to replace it. However, if the variance of the super-prior residuals is greater than the residual skip threshold, it is considered that the ground truth value of the quantization residual feature may appear at a position far from the mean of the super-prior residuals, and the mean of the super-prior residuals cannot reasonably replace the ground truth value of the quantization residual feature, and therefore is not used to replace it.

[0203] In some embodiments, determining the encoded data of the current frame based on the residual information of the current frame includes:

[0204] Encode the residual information of the current frame to determine the residual encoded data of the current frame;

[0205] The residual encoded data of the current frame is determined based on the residual encoded data and the residual prior features of the current frame. Specifically, the residual prior features and motion prior features of the current frame determine the encoded data of the current frame. These features are obtained by performing prior encoding on the residual and motion information of the current frame. Prior decoding of the residual prior features of the current frame yields the mean and variance of the prior residuals for each pixel in the current frame.

[0206] In practice, the residual information of the current frame can be encoded through the following steps S851 to S853 to obtain residual encoded data:

[0207] S851 generates a mask feature tensor based on the residual skip threshold and the variance of the super-prior residuals of each pixel in the current frame.

[0208] The mask feature tensor includes the mask feature values ​​of each pixel in the current frame. For each pixel in the current frame, if the variance of the prior residual is less than or equal to the residual skip threshold of the current frame, the mask feature value of the pixel is 1. If the variance of the prior residual is greater than the residual skip threshold of the current frame, the mask feature value of the pixel is 0.

[0209] That is, the variance of the prior residuals of the pixels is expressed as The residual skip threshold is represented as sk i If the pixel mask feature value is represented as mask, then the pixel mask feature value can be determined by the following formula (13):

[0210] S852, based on the mask feature values ​​of each pixel in the current frame, the quantization residual features of each pixel in the current frame, the mean of the super-prior residuals of each pixel in the current frame, and the following equations (14) and (15), determine the residual information and the likelihood of the residual information of each pixel in the current frame: y t =y t *(1-mask)+mask*μ Y (14) logy t =logy t *(1-mask)+mask (15)

[0211] Among them, y t μ represents the quantization residual feature of a pixel. Y Let be the mean of the prior residuals of the pixel, and mask be the mask feature value of the pixel. t The likelihood of residual information is used to represent the number of bits of data occupied by the residual information of pixels in the encoded data.

[0212] Based on equations (14) and (15) above, for pixels whose prior residual variance is less than or equal to the residual skip threshold of the current frame, since mask = 1, therefore y t =μ Y logy t =1, therefore the encoded data does not include residual information for pixels whose prior residual variance is less than or equal to the residual skip threshold of the current frame. For pixels whose prior residual variance is greater than the residual skip threshold of the current frame, since mask=0, y t =y t logy t =logy t Therefore, the encoded data includes data obtained by encoding the quantized residual features of pixels whose super-prior residual variance is greater than the residual skip threshold of the current frame.

[0213] S853: Based on the likelihood of the residual information of each pixel in the current frame, perform arithmetic encoding on the residual information of each pixel in the current frame to obtain the residual encoded data of the current frame.

[0214] Since the residual coding data only includes the data obtained by arithmetic coding of the quantized residual features of each pixel whose prior residual variance is greater than the residual skip threshold, determining the coding data of the current frame based on the reconstructed residual features of each pixel in the current frame can reduce the amount of coding data of the video frame while ensuring the quality of the reconstructed video frame.

[0215] In some embodiments, determining the encoded data of the current frame based on the residual information of the current frame may include: determining the encoded data of the current frame based on the residual information and the motion information of the current frame. The motion information of the current frame includes the motion vectors of each pixel in the current frame. That is, in some embodiments of this application, instead of using a skip mode to encode the motion vectors of each pixel in the current frame, conventional techniques may be used to encode the motion vectors of each pixel in the current frame.

[0216] In the above process, by mapping video frames to the latent domain and calculating latent residual features in this high-dimensional semantic space, the energy of the residual signal is more concentrated and the structure is more regular, making it easier to be accurately characterized by the prior probability model. Based on this, in the encoding stage, this embodiment determines the residual skip threshold of the current frame according to the first quality adjustment parameter, and generates a mask feature tensor by combining the prior residual variance of each pixel. Based on this, it decides whether to write the quantized residual features of the corresponding pixel into the bitstream: only when the prior residual variance is greater than the residual skip threshold is the quantized residual feature of the pixel included in the actual encoded data; when the prior residual variance is less than or equal to the residual skip threshold, it is considered that its true residual can be effectively approximated by the mean of the prior residual, so there is no need to transmit the quantized residual features, and the mean of the prior residual can be directly used as the reconstructed value. Since the mean of the prior residual can be obtained by decoding the residual prior features at the decoding end, this strategy can omit redundant residual information at the encoding source without additional signaling overhead.

[0217] Furthermore, the residual skipping threshold in the above process is dynamically determined by the first quality adjustment parameter. When the first quality adjustment parameter is small (i.e., high-quality mode), the residual skipping threshold is low, and only pixels with extremely low variance are skipped, thus preserving more residual details to ensure reconstruction accuracy. Conversely, when the first quality adjustment parameter is large (i.e., low bitrate mode), the residual skipping threshold is high, allowing more pixels to skip residual transmission, thereby reducing the bitstream size. Therefore, this application, referring to some embodiments shown in FIG6, achieves precise screening of residual information by collaboratively utilizing advanced prior modeling and a quality-level-based adaptive skipping mechanism in the latent domain—avoiding both quality degradation due to excessive skipping under a fixed threshold and bitrate waste caused by indiscriminate coding. This effectively reduces the total bitrate while ensuring reconstruction quality, significantly improving the rate-distortion performance and overall coding efficiency of video coding.

[0218] Some embodiments of this application provide another video encoding method, which includes the following steps S91 to S97:

[0219] S91, perform latent feature extraction on the current frame to obtain the latent features of each pixel in the current frame.

[0220] In some embodiments, latent feature extraction is performed on the current frame to obtain the latent features of each pixel in the current frame, including: inputting the current frame into a pre-trained latent feature extraction network and receiving the latent features of the current frame output by the latent feature extraction network. The latent features of the current frame include the latent features of each pixel in the current frame.

[0221] S92, determine the potential residual features of each pixel based on the potential features of each pixel in the current frame and the optical flow information of the current frame.

[0222] In some embodiments, the latent residual features of each pixel in the current frame are determined based on the latent features of each pixel in the current frame and the optical flow information of the current frame, including: inputting the latent features of each pixel in the current frame and the optical flow information of the current frame into a pre-trained residual encoder, and determining the latent residual features of each pixel in the current frame output by the residual encoder.

[0223] S93 quantizes the potential residual features of each pixel in the current frame to obtain the quantized residual features of each pixel in the current frame.

[0224] S94: Input the latent residual features of each pixel in the current frame, the optical flow information of the current frame, and the reconstructed residual features of each pixel in the reference frame into the residual super-prior coding network to obtain the residual super-prior features.

[0225] S95 decodes the residual prior features through a residual prior decoding network to obtain the mean and variance of the prior residuals for each pixel in the current frame.

[0226] S96, determine the residual skip threshold for the current frame based on the first quality adjustment parameter and the second quality adjustment parameter.

[0227] The method for determining the residual skip threshold of the current frame based on the first quality adjustment parameter and the second quality adjustment parameter is the same as the method for determining the residual skip threshold of the current frame based on the first quality adjustment parameter and the second quality adjustment parameter in the above video decoding method. To avoid redundancy, it will not be repeated here.

[0228] S97, determine the encoded data of the current frame based on the residual information and residual prior features of the current frame.

[0229] The residual information includes the quantized residual features of each pixel whose prior residual variance is greater than the residual skip threshold.

[0230] In practice, the encoded data of the current frame can be determined based on the residual information and motion information of the current frame. The method for determining the motion information of the current frame can also be the same as in the above embodiments, and will not be repeated here to avoid redundancy.

[0231] In the above process, video frames are first mapped to the latent domain, and latent residual features are generated by combining the optical flow information of the current frame. This approach allows the residual signal to not only reflect the content characteristics of the current frame but also incorporate temporal motion alignment information guided by optical flow, thus more accurately characterizing the detailed components that are difficult to cover by inter-frame prediction. Further, the latent residual features of the current frame, the optical flow information of the current frame, and the reconstructed residual features of each pixel in the reference frame are input into a residual super-prior coding network to generate residual super-prior features. After decoding by the residual super-prior decoding network, the mean and variance of the super-prior residuals for each pixel in the current frame can be obtained. Because this modeling process explicitly utilizes cross-frame residual correlation and temporal context, the obtained super-prior statistics more accurately characterize the true residual distribution of the current frame.

[0232] Based on this, during the encoding stage, the residual skip threshold for the current frame is determined jointly by the first and second quality adjustment parameters. A mask feature tensor is then generated by combining the variance of the prior residuals for each pixel. This tensor controls whether the quantized residual features are written into the bitstream: when the variance of the prior residuals for a pixel is greater than the residual skip threshold, its quantized residual features are included in the encoded data; when the variance of the prior residuals is less than or equal to the residual skip threshold, it is considered that the true residual can be effectively approximated by the mean of the prior residuals, and there is no need to transmit the quantized residual features. Instead, the mean of the prior residuals is directly used as the reconstructed value for that pixel. Since the mean of the prior residuals can be obtained by decoding the residual prior features at the decoding end, the above mechanism eliminates the need for additional signaling overhead, thus omitting redundant residual information at the encoding source and achieving efficient compression.

[0233] Some embodiments of this application provide another video encoding method, which includes the following steps S101 to S106:

[0234] S101, determine the residual information and residual prior features of the current frame, as well as the motion vector of each pixel in the current frame;

[0235] The method for determining the residual information of each pixel in the current frame can refer to any of the above embodiments. To avoid redundancy, it will not be repeated here.

[0236] In some embodiments, determining the motion vector of each pixel in the current frame may include the following steps S1011 and S1012:

[0237] S1011, perform motion estimation on the current frame based on the reference frame to determine the motion estimation parameters of the current video.

[0238] In video sequences, adjacent video frames often exhibit strong correlations, but objects in the frame frequently shift positions. Motion estimation, through specific algorithms and strategies, identifies pixel-level motion relationships between the current frame and reference frames within a series of video frames. This allows the use of existing information from the reference frame to predict the content of the current frame, effectively reducing redundancy in the video data.

[0239] S1012, determine the motion vector of each pixel in the current frame based on the motion estimation parameters of each pixel in the current frame, the reconstructed motion vector of the reference frame, the latent features of each pixel in the current frame, and the reconstructed latent features of the reference frame.

[0240] When the reference frame is the reconstructed frame of the previous frame of the current frame, the reconstructed motion vector of the reference frame encodes the coded motion vector of the previous frame of the current frame to obtain the coded data of the coded motion vector of the previous frame of the current frame. Then, the coded data of the coded motion vector of the previous frame of the current frame is decoded to obtain the motion vector. The reconstructed latent feature of the reference frame refers to the coded data of the reconstructed residual feature of the previous frame of the current frame obtained by encoding the reconstructed residual feature of the previous frame of the current frame, the residual feature obtained by decoding the coded data of the reconstructed residual feature of the previous frame of the current frame, and the latent feature obtained by compensating the residual feature with the optical flow information of the previous frame of the current frame.

[0241] In some embodiments, determining the motion vector of each pixel in the current frame based on the motion estimation parameters of each pixel in the current frame, the reconstructed motion vector of the reference frame, the latent features of each pixel in the current frame, and the reconstructed latent features of the reference frame includes: inputting the motion estimation parameters of each pixel in the current frame, the reconstructed motion vector of the reference frame, the latent features of each pixel in the current frame, and the reconstructed latent features of the reference frame into a motion encoder, and determining the motion vector of each pixel in the current frame output by the motion encoder.

[0242] S102, quantize the motion vectors of each pixel in the current frame to obtain the quantized motion vectors of each pixel; and encode the motion vectors of each pixel using a priori encoding to obtain the motion priori features of the current frame.

[0243] In some embodiments, determining the motion prior features of the current frame based on the motion vectors of each pixel in the current frame includes: inputting the motion vectors of each pixel in the current frame, the encoded motion vectors of each pixel in the reference frame, and the reconstructed residual features of each pixel in the reference frame into a motion prior coding network, and determining the motion prior features output by the motion prior coding network.

[0244] S103, perform hyperprior decoding on the hyperprior features of the current frame to obtain the hyperprior motion mean and hyperprior motion variance of each pixel in the current frame.

[0245] S104, determine the motion skip threshold for the current frame based on the first quality adjustment parameter.

[0246] The first quality adjustment parameter is a parameter used to adjust the coding quality level of the video sequence to which the current frame belongs.

[0247] In some embodiments, determining the motion skip threshold of the current frame based on a first quality adjustment parameter includes: determining a second quality adjustment parameter based on the position of the current frame in its respective image group, and determining the motion skip threshold of the current frame based on the first quality adjustment parameter and the second quality adjustment parameter. The second quality adjustment parameter is a parameter corresponding to the current frame used to adjust the coding quality of frames at different positions in the image group.

[0248] The method for determining the motion skip threshold of the current frame based on the first quality adjustment parameter and the second quality adjustment parameter can be the same as the method for determining the motion skip threshold of the current frame based on the first quality adjustment parameter and the second quality adjustment parameter in the video decoding method described above. To avoid redundancy, it will not be repeated here.

[0249] S105: Encode the quantized motion vectors of each pixel whose prior motion variance is greater than the motion skip threshold to obtain the motion information of the current frame.

[0250] S106. Determine the encoded data of the current frame based on the residual information, motion information, residual prior features, and motion prior features of the current frame.

[0251] In the above process, not only is an adaptive skipping mechanism based on prior statistics and quality adjustment parameters implemented for residual information, but this mechanism is also extended to the encoding process of motion information. Specifically, in this embodiment, a motion encoder is used to generate more accurate pixel-level motion vectors by combining the motion estimation parameters of the current frame, the reconstructed motion vector of the reference frame, and the latent features of the current frame and the reference frame. Subsequently, the motion vectors are quantized, and a motion prior coding network is used to construct motion prior features based on the motion vector of the current frame, the encoded motion vector of the reference frame, and the reconstructed residual features, thereby obtaining the prior motion mean and prior motion variance of each pixel at the decoding end. This modeling method explicitly utilizes the continuity and structural correlation of the motion field in the temporal domain, enabling the prior motion statistics to effectively reflect the distribution characteristics of the real motion vectors.

[0252] Based on this, the motion skip threshold for the current frame is dynamically determined according to the first quality adjustment parameter (representing the overall coding quality level) and an optional second quality adjustment parameter. Only pixels with a priori motion variance greater than this threshold have their quantized motion vectors written into the bitstream. For pixels with a priori motion variance less than or equal to the motion skip threshold, the priori motion mean is directly used as the reconstructed motion vector, without transmitting the corresponding motion data. Since the priori motion mean can be obtained by decoding the priori motion features at the decoding end, redundant motion information can be effectively eliminated. Therefore, without introducing additional signaling overhead, the overall bitstream size is significantly reduced while maintaining high-fidelity inter-frame prediction and reconstruction quality, thereby improving the rate-distortion performance and compression efficiency of video coding.

[0253] Referring to FIG7, some embodiments of this application provide a video decoding device 1100, which may include:

[0254] Decoding unit 111 is used to decode the encoded data of the current frame, determine the residual information of the current frame, and the mean and variance of the prior residuals corresponding to each pixel in the current frame.

[0255] The residual threshold parsing unit 112 is used to determine the residual skip threshold of the current frame according to the first quality adjustment parameter, wherein the first quality adjustment parameter is a parameter used to adjust the coding quality level of the video sequence to which the current frame belongs;

[0256] The residual reconstruction unit 113 is used to determine the reconstructed residual feature of the pixel as the mean of the prior residual if the variance of the prior residual of the pixel is less than or equal to the residual skip threshold of the current frame, and if the variance of the prior residual of the pixel is greater than the residual skip threshold of the current frame, then the reconstructed residual feature of the pixel is determined according to the residual information.

[0257] The frame reconstruction unit 114 is used to reconstruct the current frame based on the reconstruction residual features of each pixel to obtain the reconstructed frame of the current frame.

[0258] Referring to Figure 8, the video decoding device 1100 further includes:

[0259] The motion threshold parsing unit 115 is used to determine the motion skip threshold of the current frame based on the first quality adjustment parameter.

[0260] The motion vector reconstruction unit 116 is used to determine the reconstructed motion vector of the pixel as the prior motion mean of the pixel if the prior motion variance of the pixel is less than or equal to the motion skip threshold of the current frame, and if the prior residual variance of the pixel is greater than the motion skip threshold of the current frame, then the reconstructed motion vector of the pixel is determined according to the motion information.

[0261] The frame reconstruction unit 114 is also used to reconstruct the current frame based on the reconstruction residual features and reconstruction motion vectors of each pixel in the current frame, so as to obtain the reconstructed frame of the current frame.

[0262] This application embodiment also provides a video encoding device. Referring to FIG9, the video encoding device 1300 may include:

[0263] The residual feature determination unit 131 is used to determine the potential residual features of each pixel in the current frame;

[0264] The residual quantization unit 132 is used to quantize the potential residual features of each pixel in the current frame to obtain the quantized residual features of each pixel in the current frame.

[0265] The residual prior information determination unit 133 is used to determine the mean and variance of the prior residuals of each pixel based on the potential residual characteristics of each pixel.

[0266] The residual threshold processing unit 134 is used to determine the residual skip threshold of the current frame according to the first quality adjustment parameter, wherein the first quality adjustment parameter is a parameter used to adjust the coding quality level of the video sequence to which the current frame belongs;

[0267] The encoding information determination unit 135 is used to determine the encoding data of the current frame based on the residual information of the current frame; the residual information includes the quantization residual features of each pixel whose prior residual variance is greater than the residual skip threshold.

[0268] Referring to FIG10, the video encoding device 1300 further includes:

[0269] The motion vector determination unit 136 is used to determine the motion vector of each pixel in the current frame;

[0270] The motion vector quantization unit 137 is used to quantize the motion vectors of each pixel in the current frame to obtain the quantized motion vectors of each pixel in the current frame.

[0271] The motion prior information determination unit 138 is used to determine the prior motion mean and prior motion variance of each pixel in the current frame based on the motion vector of each pixel in the current frame.

[0272] The motion threshold processing unit 139 is used to determine the motion skip threshold of the current frame. When the timing of the current frame meets the preset conditions, the motion skip threshold of the current frame is the product of the third coefficient and the value of the motion skip threshold function. When the timing of the current frame does not meet the preset conditions, the motion skip threshold of the current frame is the product of the fourth coefficient and the value of the motion skip threshold function. The third coefficient is less than the fourth coefficient.

[0273] The motion information determination unit 1310 is used to determine the motion information of the current frame. The motion information is encoded data obtained by encoding the quantized motion vectors of each pixel whose prior motion variance is greater than the motion skip threshold.

[0274] The encoding information determination unit 135 is also used to determine the encoding data of the current frame based on the residual information and motion information of the current frame.

[0275] The video encoding apparatus provided in the above embodiments can execute the video encoding method provided in any of the above embodiments, and the implementation principle and the achieved technical effect are the same. To avoid redundancy, it will not be described in detail again.

[0276] Some embodiments of this application provide a video decoder, including: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to implement the video decoding method provided in any of the above embodiments when executing the computer program.

[0277] Some embodiments of this application provide a video encoder, which may include: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to implement the video encoding method provided in any of the above embodiments when executing the computer program.

[0278] Figure 11 illustrates an exemplary configuration block diagram of a video decoder or video encoder in some of the above embodiments. As shown in Figure 11, the video decoder or video encoder 1500 may include at least one of the following: a tuner / demodulator 151, a communicator 152, a detector 153, an external device interface 154, a processor 155, a display 156, an audio output interface 157, a memory 158, a power supply 159, and a user interface.

[0279] In some embodiments, processor 155 includes at least one of: a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (random access memory), ROM (read-only memory), a first to an nth interface for input / output, a communication bus, etc.

[0280] The display 156 includes a display screen assembly for presenting images, a driving assembly for driving image display, a component for receiving image signals from the processor output, and a user interface for displaying video content, image content, menu control interface, and user control UI.

[0281] The display 156 can be a liquid crystal display, an organic light-emitting diode (OLED) display, or a projection display, etc.

[0282] The communicator 152 is a component used to communicate with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of the following: a Wi-Fi module, a Bluetooth module, a wired Ethernet module, other network communication protocol chips or near-field communication protocol chips, and an infrared receiver. The video decoder or video encoder 1500 can use the communicator 152 to send and receive control signals and data signals with the control device or server.

[0283] The user interface can be used to receive control signals input by the user through a control device (such as an infrared remote control) or by touch or gesture.

[0284] Detector 153 can be used to acquire signals from the external environment or to interact with the external environment. For example, detector 153 may include a light receiver, a sensor for acquiring ambient light intensity; or, detector 153 may include an image acquisition device, such as a camera, for acquiring external environmental scenes, user attributes, or user interaction gestures; or, detector 153 may include a sound acquisition device, such as a microphone, for receiving external sounds.

[0285] External device interface 154 may include, but is not limited to, one or more of the following: High-Definition Multimedia Interface (HDMI), analog or high-definition component input interface (component), Composite Video Broadcast Signal (CVBS), Universal Serial Bus (USB), RGB (Red, Green, Blue) port, etc. It may also be a composite input / output interface formed by multiple of the above interfaces.

[0286] The tuner / demodulator 151 receives broadcast television signals via wired or wireless means, and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals. In some embodiments, the processor 155 and the tuner / demodulator 151 may be located in different separate devices, that is, the tuner / demodulator 151 may also be located in an external device of the main device where the processor 155 is located, such as an external set-top box.

[0287] Processor 155 controls the operation of electronic devices and responds to user operations through various software control programs stored in memory. Processor 155 controls the overall operation of video decoder or video encoder 1500. For example, in response to receiving a user command to select a UI object to be displayed on display 156, processor 155 can execute operations related to the object selected by the user command.

[0288] In some embodiments, processor 155 includes at least one of: a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (random access memory), ROM (read-only memory), a first to an nth interface for input / output, a communication bus, etc.

[0289] Some embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a computing device, causes the computing device to implement the video encoding method or the video decoding method described in any of the above embodiments.

[0290] Some embodiments of this application also provide a computer program product that, when run on a computer, enables the computer to implement the video encoding method or the video decoding method described in any of the above embodiments.

[0291] Some embodiments of this application provide a decoding chip, which includes a processor and a memory. The memory is used to store programs or instructions that can run on the processor, and the processor is used to execute the programs or instructions to perform the video decoding method described in any of the above embodiments.

[0292] Some embodiments of this application provide an encoding chip, the decoding chip including a processor and a memory, the memory for storing programs or instructions that can run on the processor, and the processor for executing the programs or instructions to perform the video encoding method described in any of the above embodiments.

[0293] Some embodiments of this application also provide a bitstream, including a bitstream generated according to any of the video encoding methods described above.

[0294] Some embodiments of this application also provide a method for storing a bitstream, including: generating a bitstream by performing any of the video encoding methods described above, and storing the bitstream.

[0295] Some embodiments of this application also provide a method for transmitting a bitstream, including: generating a bitstream by performing any of the video encoding methods described above, and transmitting the bitstream.

[0296] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A video decoding method, the method comprising: Decode the encoded data of the current frame to determine the residual information of the current frame, as well as the mean and variance of the prior residuals corresponding to each pixel in the current frame. The residual skip threshold for the current frame is determined based on a first quality adjustment parameter, wherein the first quality adjustment parameter is a parameter used to adjust the coding quality level of the video sequence to which the current frame belongs; For each pixel in the current frame, if the variance of the prior residual of the pixel is less than or equal to the residual skip threshold of the current frame, then the mean of the prior residual of the pixel is determined as the reconstruction residual feature of the pixel; if the variance of the prior residual of the pixel is greater than the residual skip threshold of the current frame, then the reconstruction residual feature of the pixel is determined according to the residual information. The current frame is reconstructed based on the reconstruction residual features of each pixel to obtain the reconstructed frame of the current frame.

2. The method as described in claim 1, wherein determining the residual skip threshold for the current frame based on the first quality adjustment parameter comprises: Based on the position of the current frame in its respective image group, a second quality adjustment parameter is determined. The second quality adjustment parameter is a parameter used to adjust the coding quality of frames at different positions in the image group corresponding to the current frame. The residual skip threshold for the current frame is determined based on the first quality adjustment parameter and the second quality adjustment parameter.

3. The method as described in claim 2, wherein determining the residual skip threshold for the current frame based on the first quality adjustment parameter and the second quality adjustment parameter comprises: The first coefficient is determined based on the first quality adjustment parameter and the first mapping function; The second coefficient is determined based on the second quality adjustment parameter and the second mapping function; Summing the first coefficient and the second coefficient, the residual threshold multiplier is determined; The residual skip threshold for the current frame is determined based on the residual threshold multiplier.

4. The method of claim 3, wherein determining the second quality adjustment parameter based on the position of the current frame in its respective image group includes: If the current frame is the first video frame in its respective image group, then the second quality adjustment parameter is determined to be the first value; If the current frame is not the first video frame in its respective image group, then the second quality adjustment parameter is determined to be a second value; wherein the first value is greater than the second value.

5. The method of claim 4, wherein determining the second coefficient based on the second quality adjustment parameter and the second mapping function comprises: If the current frame is the first video frame in its image group, then the first value is mapped to the third value through the second mapping function, and the third value is used as the second coefficient; If the current frame is not the first video frame in its image group, the second value is mapped to the fourth value through the second mapping function, and the fourth value is used as the second coefficient; wherein the third value is less than the fourth value.

6. The method of claim 4, wherein determining the second coefficient based on the second quality adjustment parameter and the second mapping function comprises: If the current frame is the first video frame in its image group, then the first value is mapped to the third value through the second mapping function, and the third value is used as the second coefficient; If the current frame is not the first video frame in the image group, the second coefficient is determined according to the second mapping function, the second value, and the sequence number of the current frame in the image group; wherein, when the current frame is not the first video frame in the image group, the sequence number of the current frame in the image group is positively correlated with the second coefficient.

7. The method of claim 3, wherein determining the residual skip threshold for the current frame based on the residual threshold multiplier comprises: The residual skip threshold for the current frame is determined by multiplying the residual threshold multiplier with the residual skip threshold function. The residual skip threshold function is a function with the frame index of the current frame as the independent variable, and the residual skip threshold function is monotonically increasing within the domain of the frame index of the video sequence to which the current frame belongs.

8. The method according to any one of claims 1-7, wherein reconstructing the current frame based on the reconstruction residual features of each pixel to obtain the reconstructed frame of the current frame includes: The reconstructed frame is obtained by reconstructing the current frame based on the reconstruction residual features and reconstruction motion vectors of each pixel in the current frame; The reconstructed motion vectors of each pixel in the current frame are determined in the following way: Decode the encoded data to determine the motion information of the current frame, as well as the mean and variance of the prior motion for each pixel in the current frame. The motion skip threshold for the current frame is determined based on the first quality adjustment parameter; For each pixel in the current frame, if the prior motion variance of the pixel is less than or equal to the motion skip threshold of the current frame, then the prior motion mean of the pixel is determined as the reconstructed motion vector of the pixel; if the prior residual variance of the pixel is greater than the motion skip threshold of the current frame, then the reconstructed motion vector of the pixel is determined according to the motion information.

9. The method of claim 8, wherein determining the motion skip threshold of the current frame based on the first quality adjustment parameter comprises: The second quality adjustment parameter is determined based on the position of the current frame in its respective picture group. The second quality adjustment parameter is a parameter used to adjust the coding quality of frames at different positions in the picture group corresponding to the current frame. The motion skip threshold for the current frame is determined based on the first quality adjustment parameter and the second quality adjustment parameter.

10. The method of claim 8, wherein reconstructing the current frame based on the reconstruction residual features and reconstruction motion vectors of each pixel in the current frame to obtain the reconstructed frame comprises: Based on the reconstructed motion vectors of each pixel in the current frame, determine the predicted features of each pixel in the current frame; Based on the reconstruction residual features of each pixel in the current frame and the prediction features of each pixel in the current frame, the potential reconstruction features of each pixel in the current frame are determined; and the current frame is reconstructed based on the potential reconstruction features of each pixel to obtain the reconstructed frame.

11. A video encoding method, comprising: Determine the potential residual features of each pixel in the current frame; The potential residual features of each pixel in the current frame are quantized to obtain the quantized residual features of each pixel in the current frame; Based on the potential residual characteristics of each pixel, determine the mean and variance of the prior residuals for each pixel. Based on the first quality adjustment parameter, the residual skip threshold of the current frame is determined, where the first quality adjustment parameter is a parameter used to adjust the coding quality level of the video sequence to which the current frame belongs; Based on the residual information of the current frame, the encoded data of the current frame is determined; the residual information includes the quantization residual features of each pixel whose super-prior residual variance is greater than the residual skip threshold.

12. The method of claim 11, wherein determining the residual skip threshold for the current frame based on the first quality adjustment parameter comprises: Based on the position of the current frame in its respective image group, a second quality adjustment parameter is determined. The second quality adjustment parameter is a parameter used to adjust the coding quality of frames at different positions in the image group corresponding to the current frame. The residual skip threshold for the current frame is determined based on the first quality adjustment parameter and the second quality adjustment parameter.

13. The method of claim 12, wherein determining the residual skip threshold for the current frame based on the first quality adjustment parameter and the second quality adjustment parameter comprises: The first coefficient is determined based on the first quality adjustment parameter and the first mapping function; The second coefficient is determined based on the second quality adjustment parameter and the second mapping function; Summing the first coefficient and the second coefficient, the residual threshold multiplier is determined; The residual skip threshold for the current frame is determined based on the residual threshold multiplier.

14. The method of claim 13, wherein determining the second quality adjustment parameter based on the position of the current frame in its respective image group comprises: If the current frame is the first video frame in its respective image group, then the second quality adjustment parameter is determined to be the first value; If the current frame is not the first video frame in its respective image group, then the second quality adjustment parameter is determined to be a second value; wherein the first value is greater than the second value.

15. The method of claim 14, wherein determining the second coefficient based on the second quality adjustment parameter and the second mapping function comprises: If the current frame is the first video frame in its image group, then the first value is mapped to the third value through the second mapping function, and the third value is used as the second coefficient; If the current frame is not the first video frame in its image group, the second value is mapped to the fourth value through the second mapping function, and the fourth value is used as the second coefficient; wherein the third value is less than the fourth value.

16. The method of claim 14, wherein determining the second coefficient based on the second quality adjustment parameter and the second mapping function comprises: If the current frame is the first video frame in its image group, then the first value is mapped to the third value through the second mapping function, and the third value is used as the second coefficient; If the current frame is not the first video frame in the image group, the second coefficient is determined according to the second mapping function, the second value, and the sequence number of the current frame in the image group; wherein, when the current frame is not the first video frame in the image group, the sequence number of the current frame in the image group is positively correlated with the second coefficient.

17. The method of claim 13, wherein determining the residual skip threshold for the current frame based on the residual threshold multiplier comprises: The residual skip threshold for the current frame is determined by multiplying the residual threshold multiplier with the residual skip threshold function. The residual skip threshold function is a function with the frame index of the current frame as the independent variable, and the residual skip threshold function is monotonically increasing within the domain of the frame index of the video sequence to which the current frame belongs.

18. The method of any one of claims 11-17, wherein after determining the encoded data of the current frame based on the residual information of the current frame, the method further comprises: The encoded data of the current frame is sent to the video decoder so that the video decoder can reconstruct the current frame based on the reconstruction residual features and reconstruction motion vectors of each pixel in the current frame, and obtain the reconstructed frame of the current frame. The reconstructed motion vectors of each pixel in the current frame are determined in the following way: Decode the encoded data to determine the motion information of the current frame, as well as the mean and variance of the prior motion for each pixel in the current frame. The motion skip threshold for the current frame is determined based on the first quality adjustment parameter; For each pixel in the current frame, if the prior motion variance of the pixel is less than or equal to the motion skip threshold of the current frame, then the prior motion mean of the pixel is determined as the reconstructed motion vector of the pixel; if the prior residual variance of the pixel is greater than the motion skip threshold of the current frame, then the reconstructed motion vector of the pixel is determined according to the motion information.

19. The method of claim 18, wherein the motion skipping threshold is determined by: The second quality adjustment parameter is determined based on the position of the current frame in its respective picture group. The second quality adjustment parameter is a parameter used to adjust the coding quality of frames at different positions in the picture group corresponding to the current frame. The motion skip threshold for the current frame is determined based on the first quality adjustment parameter and the second quality adjustment parameter.

20. The method of claim 18, wherein the reconstructed frame is determined by: Based on the reconstructed motion vectors of each pixel in the current frame, determine the predicted features of each pixel in the current frame; Based on the reconstruction residual features of each pixel in the current frame and the prediction features of each pixel in the current frame, the potential reconstruction features of each pixel in the current frame are determined; and the current frame is reconstructed based on the potential reconstruction features of each pixel to obtain the reconstructed frame.

21. A decoding chip comprising: A processor and a memory, the processor being configured to invoke a program or instructions from the memory to cause the method as described in any one of claims 1-10 to be executed.

22. A decoder, comprising: Memory, used to store computer programs; A processor, configured to perform the method as described in any one of claims 1-10 when running the computer program.

23. An encoding chip comprising: A processor and a memory, the processor being configured to invoke a program or instructions from the memory to cause the method as described in any one of claims 11-20 to be executed.

24. An encoder, comprising: Memory, used to store computer programs; A processor, configured to perform the method as described in any one of claims 11-20 when running the computer program.

25. A computer-readable storage medium having stored thereon a computer program / instructions and a bit stream, wherein the computer program / instructions, when executed by a processor, are capable of generating the bit stream according to any one of claims 11-20.

26. A bitstream comprising a bitstream generated according to any one of claims 11-20.

27. A method of storing a bitstream, comprising: The method described in any one of claims 11-20 is used to generate a bit stream and to store the bit stream.

28. A method of transmitting a bitstream, comprising: The method described in any one of claims 11-20 is used to generate a bit stream and to transmit the bit stream.

Images