High-level syntax for video coding using deep learning

Abstract

Methods and systems are described for end-to-end video coding and decoding using deep learning. Using a deep-learning video coding architecture, high-level syntax provides separate parameters for a decoder picture buffer and a separate decoder feature buffer. Syntax, semantics, and a decoding process for reference picture and features management are also provided.

Description

HIGH-LEVEL SYNTAX FOR VIDEO CODING USING DEEP LEARNINGCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of priority from U.S. Provisional Application No. 63 / 716,341, filed on 5 November 2024, which is incorporated by reference herein in its entirety.TECHNOLOGY

[0002] The present document relates generally to images and video coding. More particularly, embodiments of the present invention relate to high-level syntax for video coding using neural networks and deep learning.BACKGROUND

[0003] In 2020, the MPEG group in the International Standardization Organization (ISO), jointly with the International Telecommunications Union (ITU), released the first version of the Versatile Video Coding Standard (VVC), also known as H.266 (Ref. [8]). More recently, the same group has been working on the development of the next generation coding standard that provides improved coding performance over existing video coding technologies. As part of this investigation, new coding techniques are also examined.

[0004] As used herein the term “end-to-end image compression neural network” refers to a neural network that jointly optimizes all components of a video compression system, from the point of input of uncompressed input to the encoder, to the point of output of a reconstructed output image of the decoder, using a single loss function.

[0005] As used herein, the terms “latent features” or “latent variables” denote features or variables that are not directly observable but are rather inferred from other observable features or variables, e.g., by processing the directly observable variables. In image and video coding, the term ‘latent space’ may refer to a representation of the compressed data in which similar data points are closer together. In video coding, examples of latent features include the representation of the transform coefficients, the residuals, the motion representation, syntax elements, model information, and the like. In the context of neural networks, latent spaces are useful for learning data features and for finding simpler representations of the image data for analysis.

[0006] As used herein, the term high-level syntax (HLS) refers to syntax, syntax functions, and their semantics or descriptors that are part of a coded bitstream and are communicated from an encoder to decoder for a proper decoding of the coded bitstream.

[0007] As appreciated by the inventors here, improved techniques for video coding techniques using deep learning are desired, and they are described herein.

[0008] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section. Similarly, issues identified with respect to one or more approaches should not assume to have been recognized in any prior art on the basis of this section, unless otherwise indicated.BRIEF DESCRIPTION OF THE DRAWINGS

[0009] An embodiment of the present invention is illustrated by way of example, and not in way by limitation, in the figures of the accompanying drawings and in which like reference numerals refer to similar elements and in which:

[0010] FIG. 1 depicts an example framework for end-to-end video compression using a deep learning -based image and video compression (DLVC) architecture according to an embodiment of this invention; and

[0011] FIG. 2 depicts an example process for reference picture management in a DLVC architecture according to an embodiment of this invention.DESCRIPTION OF EXAMPLE EMBODIMENTS

[0012] Example embodiments that relate to high-level syntax in video coding using deep learning and neural networks are described herein. In the following description, for the purposes of explanation, numerous specific details are set forth to provide a thorough understanding of the various embodiments of present invention. It will be apparent, however, that the various embodiments of the present invention may be practiced without these specific details. In other instances, well-known structures and devices are not described in exhaustive detail to avoid unnecessarily occluding, obscuring, or obfuscating embodiments of the present invention.SUMMARY

[0013] Example embodiments described herein relate to high-level syntax in video coding techniques using deep learning. Using a deep-learning video coding architecture, high-level syntax provides separate parameters for a decoder picture buffer and a separate decoder feature buffer, which includes reference picture features, reference motion vector features, reference decoded latents of motion vectors, and reference decoded latents. Syntax, semantics, and a decoding process for reference picture and reference features management are also provided.DEEP LEARNING BASED VIDEO CODING (DLVC)

[0014] Deep learning-based image and video compression approaches (DLVC) are increasingly popular, and it is an area of active research. Early DLVC models followed a traditional coding approach using the residual coding-based framework (Ref. [1]). Lately, a conditional coding framework using Deep Contextual Video Compression (DCVC) was proposed (Ref. [2]). Improvements to this model include: DCVC-TCM (temporal context mining) (Ref. [3]), DCVC-HEM (hybrid spatial-temporal entropy modeling) (Ref. 4]), DCVC- DC (diverse contexts) (Ref. [5]), and DCVC-FM (feature modulation) (Ref. [6]). The DCVC series of models support a low delay P (LDP) coding structure as in the conventional codecs, such as AVC, HEVC, VVC, and the like. A common characteristic of all these approaches is that they focus only on improving coding efficiency without considering high-level syntax (HLS) design.

[0015] In video coding standards, such as HEVC, VVC, and the like, HLS is an integral part of a video codec. Since HLS is part of the coded bitstream, it adds overhead, thus an efficient design of HLS is highly important. The key purpose of HLS design is to provide an interface to the coded bitstream to various networks / systems where the video codec is used. For example, in VVC, HLS covers a variety of topics, such as: bitstream structure and coded data units structures, sequence and picture level parameters signaling at various syntax levels (sequence, picture, etc.), random access and stream adaptation, decoded picture management (including reference picture management), profile and level specification and signaling, buffering model, parallel processing, temporal scalability, byte stream format, and the like. Without losinggenerality, example embodiments may refer to an enhanced DLVC B -model architecture (Ref. [7]), which improves coding efficiency by supporting multiple reference frames.

[0016] In a conventional codec, such as VVC, a decoded picture buffer (DPB) is used to store decoded reference pictures for inter-prediction and to store out-of-order decoded pictures until they can be outputted. For example, VVC defines the DPB as:“decoded picture buffer (DPB): A buffer holding decoded pictures for reference, output reordering, or output delay specified for the hypothetical reference decoder. ”

[0017] VVC also supports temporal motion vector (MV) predictors. When enabled, it requires a decoder to store temporal motion information for a grid of 8x8 block areas in the picture. Because the amount of the data to be stored for MVs is relatively small, the specification does not emphasize the required buffer size when it discusses Profiles and Level requirements; however, in a deep learning-based codec, more information needs to be stored for interprediction. Before discussing example embodiments, it may be beneficial to discuss the processing pipeline and storage requirements for an example DLVC architecture, as depicted in FIG. 1.

[0018] As depicted in FIG. 1, traditional residual coding (where residuals are generated using motion estimation and motion compensation between the current frame and a reference frame) is being replaced by context modeling, based on feature extraction from decoded frames.

[0019] Given inputs xt(102), as in traditional encoding, motion estimation (142) plays a key role in compressing a bitstream, and a motion encoder (180) creates motion vector (MV) latents, which are encoded using a MV entropy model to generate MV latents (ymv), which may be stored in a MV latent buffer, and are part of the coded bitstream. Decoded motion vectors (vt) (174) and frame features (F) (147) from previously decoded frames (if any) are fed to a temporal context mining block (165) to generate a context stream (C) (167), which is fed to both the contextual encoder (105) and the contextual decoder (135).

[0020] Using the contextual encoder (105), input xt(102) is encoded into latent codes yt(106), which are quantized as yt(107) using a rounding operation before they are being compressed using a lossless encoder. Reconstructed data xt(137) and features Ftare generated using a frame and feature generator (140) which receives its data from the contextual decoder (135). Frames and features are also stored in a frame and features buffer (145). Latents may bestored in a latent buffer which is being used to update the entropy model for generating the compressed latent stream. As used herein, the term “fusion model” refers to NN blocks which fuse features of prior coded frames to allow efficient compression of B -frames (Ref. [7]).

[0021] As an example, without limitation, given RGB inputs, the input to this model is a sequence of video frames packed into an input tensor of size (1, GOPSIZE, 3, H, W). GOP_SIZE denotes the input group of pictures (GOP) size of the model (e.g., 8), 3 denotes the number of color channels, and H and W are the height and width respectively of pictures in the video sequence.

[0022] As an example, consider a group of pictures (GOP) with nine frames in display order denoted as:10, Bl, B2, B3, B4, B5, B6, B7, and P8 where 10 is coded on its own in intra mode, P8 uses 10 as a reference and is coded in P-mode, and each B frame uses two nearby frames as references for encoding of the B frame in B-mode. The order of the pictures in the bitstream is modified so that the pictures used as references are positioned earlier in the coding order (also known as decoding order or bitstream order) than the pictures that refer to them in their decoding process. For example, for a given inter-prediction process, to guarantee to a decoder that all required references are available for inter-prediction, a possible coding order may be:10, P8, B4, B2, Bl, B3, B6, B5, B7.

[0023] First, the initial frame in the GOP, e.g., frame 10, is encoded via a trained intra-frame model without information of future or previous frames. Next, one needs to code frame P8, the last frame in the GOP, with a trained P-frame encoder, e.g., one based on a DCVC-DC architecture (Ref. [5,7]). This P-frame model takes as input from the I-frame model its decoded 10 frame to be used as the reference frame for P8. This information, generated from the contextual decoder (135), is packed into the DPB buffer, which includes:1. The reconstructed reference 10 frame (%) (1,3, H, W).2. The reference frame feature (F) (1, D, H, W), where D (the size of a dimension in the tensor) is generally 48. An example of such a tensor could be (1,48,256,256). This would denote a 4D- tensor with 48 256x256 2D planes.3. The reference motion vector Feature (Fmv) (1,2, H, W), depicted in FIG. 1 as stored in a motion feature buffer (150).4. The reference quantized latents y (1, D, H / / 4, W / / 4) , depicted in FIG. 1 as stored in latent buffer (155), where D is generally 128, and “ / / ” denotes integer division, e.g., a / / b = int(a / b).5. The reference ymv(1, D, H / 4, W / A-), depicted in FIG. 1 as stored in motion vector latent buffer (160), where D is generally 64.Note, the following notation is being used:• x : input frames (102)• x : reconstructed frames (137), reference frames• F frame feature (147), also to be denoted as ref_feature, used to code latents (y) and generate context C (167), using temporal context mining (165)• Fmvmotion vector feature (172), also to be denoted as ref_mv_feature, used to encode motion vector (v). Fmvis generated from motion decoder (170)• C: context (167) used to encode latents (106) with contextual encoder (105)• y: decoded latent (107) of the input frame. Also to be denoted as ref_y_hat. Used with C (167) and contextual decoder (135) to generate reconstructed frames (%) and frame features (F)• ymvdecoded latent of a motion vector (185). Also to be denoted as ref_y_hat_mv. Used to generate decoded motion vectors (v) (174) and motion vector features (Fmv) (172).

[0024] These contents will then be used to help encode frame B4, which will have as input the reference frame from frame 10 and the reference frame from frame P8, as well as the contents of the whole DPB for frame P8. For inter prediction, the DPB may store multiple reference pictures and associated reference features, reference decoded latents, reference MV features, and reference decoded latents of motion vectors.HLS in deep learning architectures

[0025] In a first embodiment, without losing generality, it is proposed to split the reference data into two categories: 1) a decoded picture buffer, with same functionality as in traditionalvideo coding, 2) a decoded feature buffer (DFB) for storing reference features F, reference MV features Fmv, y, and ymv. This is based on two observations:- The feature tensor is only used for reference purposes. It is not needed for output / display purposes.- The feature tensor has a different size from that of a decoded picture. The feature tensor size needs to be explicitly specified. The buffer size required to hold a feature tensor is different from the DPB size and can be multiple times the size of the DPB size.

[0026] The sequence parameter set (SPS) is typically defined as a syntax structure containing syntax elements that apply to one or more entire coded video sequences (CVSs) as determined by the content of a syntax element found in the picture parameter set (PPS), referred to by a syntax element found in each picture header (PH). In an embodiment, in the SPS, it is proposed to add high-level syntax to specify the dimension of the feature tensor. A DLVC architecture can support compression with RGB or YUV color components (e.g. with 4:4:4 or 4:2:0 sampling) as the input to the NN. For the RGB case, typically, a single neural network (NN) is used. For the YUV case, two NN branches or a split NN can be used: a luma NN and a chroma NN. This is mainly to reduce complexity. Because the U and V components generally contain less information than the luma (Y) component, the chroma components can have a simplified NN compared to luma, such as using fewer layers and / or a smaller number of output channels from each NN layer. In an example embodiment, the dimension of each of the following feature tensors is specified: F, Fmv, y, and ymv. For the RGB case, because only one NN is needed, there is no need to split among the R, G, and B color components. For the YUV (or YCbCr) case, for the MV related features (Fmv, ymv) luma and chroma components generally share the same feature tensor. For F and y , luma and chroma components can have different dimension (other than height / width ratio of luma and chroma components itself, e.g., in the YUV 4:2:0 case, chroma components have half the width and half the height of the luma components). Table 1 shows an example of the related syntax of feature tensors in the SPS.Table 1. Example of specifying feature tensor in SPSSemantics: sps_feature_bitdepth_minus8 plus 8 specifies the bit depth of the decoded feature.The variable FeaBitDepth = sps_feature_bitdepth_minus8 + 8 sps_split_nn_flag equal to 1 specifies a split NN is applied for luma and chroma components. sps_split_nn_flag equal to 0 specifies a single NN is applied for all color components. sps_dim_ref_feature_same_as_pic_flag equal to 1 specifies sps_ref_feature_width and sps_ref_feature_height has the same value as sps_pic_width_max_in_luma_samples and sps_pic_height_max_in_luma_samples, respecitively, as well as sps_ref_mv_feature_width and sps_ref_mv_feature_height has the same value as sps_pic_width_max_in_luma_samples and sps_pic_height_max_in_luma_samples, respecitively. sps_dim_ref_feature_same_as_pic_flag equal to 0 specifies the syntax sps_ref_feature_width and sps_ref_feature_height follow next in SPS, as well as sps_ref_mv_feature_width and sps_ref_mv_feature_height follow next in SPS. sps_ref_mv_feature_width, sps_ref_mv_feature_height specify the width and height of tensor ref_mv_feature (Fmv), respectively. sps_ref_mv_y_hat_width, sps_ref_mv_y_hat_height, sps_ref_mv_y_hat_channel specify the width, height and channel number of tensor ref_mv_y_hat_feature ymv). respectively. sps_ref_feature_width_in_luma, sps_ref_feature_height_in_luma, sps_ref_feature_channel_in_luma specify the width, height and channel number of tensor ref_feature (F) for luma components, respectively. sps_ref_feature_width_in_chroma, sps_ref_feature_height_in_chroma, sps_ref_feature_channel_in_chroma specify the width, height and channel number of tensor ref_feature for chroma components, respectively. sps_ref_y_hat_width_in_luma, sps_ref_y_hat_height_in_luma, sps_ref_y_hat_channel_in_luma specify the width, height and channel number of tensor ref_y_hat (y) for luma components, respectively.sps_ref_y_hat_width_in_chroma, sps_ref_y_hat_height_in_chroma, sps_ref_y_hat_channel_in_chroma specify the width, height and channel number of tensor ref_y_hat for chroma components, respectively.The storage of feature tensor in feature units (FeaBitDepth) for one decoded picture is the summation of all the above feature storage.

[0027] For a single network case: the variable FeaSizelnFeaUnits = sps_ref_mv_feature_width * sps_ref_mv_feature_height * 2+ sps_ref_mv_y_hat_width * sps_ref_mv_y_hat_height * sps_ref_mv_y_hat_channel+ sps_ref_feature_width_in_luma * sps_ref_feature_height_in_luma * sps_ref_feature_channel_in_luma+ sps_ref_y_hat_width_in_luma * sps_ref_y_hat_height_in_luma * sps_ref_y_hat_channel_in_luma

[0028] For a split network case: the variable FeaSizelnFeaUnits = sps_ref_mv_feature_width * sps_ref_mv_feature_height * 2+ sps_ref_mv_y_hat_width * sps_ref_mv_y_hat_height * sps_ref_mv_y_hat_channel+ sps_ref_feature_width_in_luma * sps_ref_feature_height_in_luma * sps_ref_feature_channel_in_luma+ sps_ref_feature_width_in_chroma * sps_ref_feature_height_in_chroma * sps_ref_feature_channel_in_chroma+ sps_ref_y_hat_width_in_luma * sps_ref_y_hat_height_in_luma * sps_ref_y_hat_channel_in_luma+ sps_ref_y_hat_width_in_chroma * sps_ref_y_hat_height_in_chroma * sps_ref_y_hat_channel_in_chroma

[0029] In another embodiment, instead of specifying direct values of the feature tensor, one can alternatively specify the ratio of picture data versus feature tensor. In another embodiment,for the ref_feature related chroma tensor, one can also specify the ratio of luma and chroma tensor. In another embodiment, for the ref_feature tensor and the ref_feature_y_hat tensor, alternatively, one can specify the ratio of ref_feature tensor and ref_feature_y_hat tensor. The same can be done for the ref_mv_feature tensor and the ref_mv_y_hat feature tensor.

[0030] In VVC, the function dpb_parameters( sps_max_sublayers_minusl, sps_sublayer_dpb_params_flag ) is used to specify DPB-related parameters. In an embodiment, its functionality may be extended to cover both the DPB and DFB buffers. An example is shown in Table 2, where the new entry is shown in Italics. It allows the DPB and the DFB to have different maximum size numbers. The decoded picture in the DPB has to be stored until the picture is outputted or it is not used as a reference. The decoded feature in the DFB is only used for prediction. This allows the DFB to store a lower number of feature tensors.Table 2. Example of combined DPB and DFB parameter syntaxSemantics: dfb_max_dec_feature_buffering_minusl[ i ] plus 1 specifies the maximum required size of the DFB in units of feature storage buffers when Htid is equal to i. The value of dfb_max_dec_feature_buffering_minusl[ i ] shall be in the range of 0 to MaxDfbSize - 1, inclusive. When i is greater than 0, dfb_max_dec_feature_buffering_minusl[ i ] shall be greater than or equal to dfb_max_dec_feature_buffering_minusl[ i - 1 ]. When dfb_max_dec_feature_buffering_minusl[ i ] is not present for i in the range of 0 to MaxSubLayersMinusl - 1, inclusive, due to subLayerlnfoFlag being equal to 0, it is inferred to be equal to dfb_max_dec_feature_buffering_minusl[ MaxSubLayersMinusl ].Note: Htid refers to the highest temporal sublayer to be decoded.

[0031] In VVC, general tier and level limits specify the maximum luma picture size “MaxLumaPs (samples)”. In an embodiment, it is proposed to add “Max feature size MaxFs” in the general tier and level limits. One example is shown in Table 3. This can set a limit related to the DFB size for each level.Table 3. Example of Tier and Level Limits with feature size limitsNote: The ratio number r (e.g., r = 62) is computed for a single network case. Different levels may use different values of r, as determined by typical cases of interest and decoder limitations.

[0032] MaxDfbSize can be specified the same way as MaxDpbSize. if( 2 * FeaSizelnFeaUnits <= MaxFs )MaxDfbSize = 2 * maxDfbPicBuf else if( 3 * FeaSizelnFeaUnits <= 2 * MaxFs )MaxDfbSize = 3 * maxDfbPicBuf / 2 elseMaxDfbSize = maxDfbPicBufBecause the size of the feature tensor for one decoded picture, generally, is much larger than the size of the decoded picture, maxDfbPicBuf is set smaller than or equal to maxDpbPicBuf. For example, maxDfbPicBuf is equal to 5.Separate control of managing decoded pictures and decoded features

[0033] In this section it is assumed that inter prediction reference picture and feature management are coupled together, and that explicit signaling in HLS (e.g., in the SPS and the picture header (PH)) is used for reference control in the DPB and the DFB.

[0034] DLVC architectures tend to accumulate errors within their feature buffers for temporal prediction. In a conventional codec, such as VVC, intra frames are used to stop the error propagation by cleaning the DPB. In DLVC, it is observed that the main reason for temporal quality degradation is because the temporal feature tensor is either contaminated or does not have correlation for the current picture. In an embodiment, one can reduce the degradation on coding quality by selectively disabling certain feature tensor propagation.

[0035] In an embodiment, to ameliorate this issue and prevent further frames from performance degradation, an interval syntax parameter, e.g., reset_interval_minus2, in HLS, such as SPS, is specified. The variable Resetinterval = reset_interval_minus2 + 2. When picture order count POC % Resetinterval = 1, one or more features in the DFB (e.g., ref_feature) is reset to None or Null. Neural network blocks that receive feature vectors marked as “None” or “Null,” in effect, they are directed to use Lframe related features, thus removing any errors that are associated with those features. One can allow flexibility to disable any type of features. As described earlier, there are four distinct features: ref_feature, ref_mv_feature, ref_y_hat, ref_mv_y_hat. Though, as an example, one only resets the ref_feature, other features can be handled in a similar way.

[0036] An example HLS syntax using the proposed reset flag is described in Table 4 and Table 5. In SPS, one specifies reset_interval_minus2. In PH, one may specify which feature is reset in the DFB. Similar flags can be set in the picture parameter sets and slice headers too.Table 4. An example of HLS for feature refresh control in DLVC SPSSemantics: reset_interval_minus2 plus 2 specifies the reset interval in CVS. The variable Resetinterval is set equal to reset_interval_minus2 + 2.Table 5. Example of HLS for feature refresh control in the picture headerSemantics: dfb_ref_feature_disable_flag equal to 1 specifies the ref_feature stored in the DFB is reset and is calculated directly from decoded reference picture in the DPB. dfb_ref_feature_disable_flag equal to 0 specifies ref_feature stored in the DFB is not reset. dfb_ref_mv_feature_disable_flag equal to 1 specifies the ref_mv_feature stored in the DFB is set to NUEE. dfb_ref_mv_feature_disable_flag equal to 0 specifies ref_mv_feature stored in the DFB is not reset. dfb_ref_y_hat_disable_flag equal to 1 specifies the ref_y_hat stored in the DFB is set to NULL. dfb_ref_y_hat_disable_flag equal to 0 specifies ref_y_hat stored in the DFB is not reset. dfb_ref_mv_y_hat_disable_flag equal to 1 specifies the ref_mv_y_hat stored in the DFB is set to NULL. dfb_ref_mv_y_hat_disable_flag equal to 0 specifies ref_mv_y_hat stored in the DFB is not reset.

[0037] In an alternative method, a new network abstraction layer (NAL) unit of type FDR is proposed to indicate the coded slice which only resets the DFB and does not reset decoded pictures in the DPB. An example is given in Table 6. Table 6. An example of an FDR NAL typeThe benefit of having a new NAL type is to let the system layer know that different operations are to be performed on the DPB and the DFB. In one embodiment, one can use the FDR NAL type to specify that all feature tensors are disabled and the DFB is required to reset completely.Operations for the DPB and the DFBReference picture and reference feature management (RPFM)

[0038] Given that two separate buffers are being used, one for reference pictures and one for reference features, this section provides example embodiments for improved management of the content of the two buffers.

[0039] Reference picture management (RPM) is necessary for a video codec which uses inter prediction. It manages storage and removal of reference pictures into and from the DPB and puts reference pictures in a proper order in the reference picture lists (RPL). To mark which pictures are to be removed from the DPB, a reference picture marking process (making a reference picture as “used for short-term reference,” “used for long-term reference,” or “unused for reference”) is needed. RPM in VVC is based on direct signaling of RPLs. Two RPLs, list 0 and list 1 , are directly signaled and derived. When a picture is referenced for inter-picture prediction, an index into an PRL is used to select which picture in the DPB is being referenced. Reference picture marking is directly based on RPLs 0 and 1 , utilizing both active and inactive entries in the RPLs, while only active entries may be used as reference indices in inter prediction. Information for the derivation of the two RPLs is signaled by syntax elements and syntax structures in the SPS, the PPS, the PH, and the slice header (SH). Predefined RPL structures are signaled in the SPS, for use by referencing in the PH / SH. The two RPLs are generated for all types of slices, i.e., B, P, and I slices.

[0040] In an embodiment, POC (Picture Order Count) is used as a picture ID, for identification of pictures in the decoding process. Without limitation, it is also proposed to associate POC with both decoded pictures and decoded features.

[0041] VVC uses up to two reference picture list structures. In DL-based coding, there is no such requirement. Unlike traditional coding, there is no block-level switching between uniprediction (P) and bi-prediction (B) for inter-prediction. Instead, a picture-level decision is made on whether to use one reference picture and feature tensor for inter-prediction or to apply fusion (e.g., exercise the gray blocks in FIG.l) from multiple reference pictures and feature tensors for inter-prediction.

[0042] In an example embodiment:1) There is only one reference picture list (RPL)2) The number of reference entries in the RPL is used to specify the upper limit of references for inter-prediction, or to be more specific, the number of reference entries in RPL is used to specify the upper limit of the number of temporal fusion, i.e., how many fusions can be applied. In addition, the number of reference entries for decoded pictures and decoded feature tensors can be different. This provides two advantages: 1) the size of feature tensors is much larger than that of frame, thus, allowing a different number of reference entries for the DPB and the DFB, which can help provide a better balance on coding efficiency and memory requirement. 2) Some of the feature tensors may cause temporal error propagation and can be computed from a decoded picture directly, while other features tensors can be set to NULL, thus disabling error propagation and forcing to use features only from I-frames. This is different from traditional decoded picture behavior, where a decoded picture is required for output / display purposes and setting to NULL is not a choice.3) For simplicity, the reference picture marking contains only two states: “used for reference” and “not used for reference”; however, a three-state model, differentiating between short and long references, may be followed as well.

[0043] The following three Tables show examples of the proposed RPFM related syntax. The newly proposed syntax or current VVC syntax, but with different semantics, is denoted using an Italic font.Table 7. Example of RPL related syntax in DLVC SPSSemantics: sps_num_ref_lists specifies the number of the ref_pic_list_struct( rplsldx ) syntax structures included in the SPS.Table 8. Example of RPL related syntax in DLVC ref_pic_listSemantics: rpl_sps_flag equal to 1 specifies that RPL in ref_pic_lists( ) is derived based on one of the ref_pic_list_struct( rplsldx ) syntax structures in the SPS. rpl_sps_flag equal to 0 specifies that RPL of the picture is derived based on the ref_pic_list_struct( rplsldx ) syntax structure that is directly included in ref_pic_lists( ).When rpl_sps_flag is not present, if sps_num_ref_pic_lists is equal to 0, the value of rpl_sps_flag is inferred to be equal to 0.rpl_idx specifies the index, into the list of the ref_pic_list_struct( rplsldx ) syntax structures included in the SPS, of the ref_pic_list_struct( rplsldx ) syntax structure that is used for derivation of RPL of the current picture or slice. The syntax element rpl_idx is represented by Ceil( Log2( sps_num_ref_pic_lists ) ) bits.When rpl_sps_flag is equal to 1 and sps_num_ref_pic_lists is equal to 1, the value of rpl_idx is inferred to be equal to 0.The value of rpl_idx shall be in the range of 0 to sps_num_ref_pic_lists - 1 , inclusive.The variable Rplsldx is derived as follows:Rplsldx = rpl_sps_flag ? rpl_idx : sps_num_ref_pic_listsTable 9. Example of RPL related syntax in DLVC ref_pic_list_struct()Notes and semantics:In ref_pic_list_struct(), the syntax and semantics don’t not align with VVC anymore.1) num_ref_entries is used to specify the maximum number of reference pictures to be used for inter-prediction (to be more specific, maximum number of allowed temporal fusions)2) num_ref_feature_entries is added to allow the maximum number of references for feature tensors. Note that when the value of a feature tensor is set to None or NULL, then a decoder is not allowed to use any of these features. One may limit num_ref_feature_entries to be smaller than or equal num_ref_entries, and code the delta_num difference. In another embodiment one may have the limitation and allow num_ref_feature_entries to be larger than num_ref_entries. Then, setdelta_num_fea_ref_entries[ rplsldx ] to se(v). The loop is set to use max(num_ref_entries[ rplsldx ], num_fea_ref_entries[ rplsldx ])3) additional signaling related to short-term or long-term is removed. There is no need to signal which kind of reference the reference picture is. The active entries in RPL are marked as “used for reference” and the remaining references in the DPB / DFB are marked as “not used for reference.” The value of delta POC value may be coded as in VVC.

[0044] The ref_pic_list_struct( rplsldx ) syntax structure could be present in an SPS, in a PH syntax structure, or in a slice header. Depending on whether the syntax structure is included in an SPS, a PH syntax structure, or a slice header, the following applies:- If present in a PH syntax structure or slice header, the ref_pic_list_struct( rplsldx ) syntax structure specifies RPL listldx of the current picture (i.e., the coded picture containing the PH syntax structure or slice header).- Otherwise (present in an SPS), the ref_pic_list_struct( rplsldx ) syntax structure specifies a candidate for RPL listldx, and the term "the current picture" in the semantics specified in the remainder of this clause refers to each picture that 1) has a PH syntax structure or one or more slices containing rpl_idx equal to an index into the list of the ref_pic_list_struct( rplsldx ) syntax structures included in the SPS, and 2) is in a CLVS that refers to the SPS. num_ref_entries[ rplsldx ] specifies the number of entries in the ref_pic_list_struct( rplsldx ) syntax structure. This syntax indicates that the maximum number of references which are used for inter-prediction (temporal fusion). delta_num_fea_ref_entries[ rplsldx ] specifies the differece between num_ref_entries[ rplsldx ] and NumRefFeatureEntries[ rplsldx ]. num_ref_entries[ rplsldx ] minus delta_num_fea_ref_entries[ rplsldx ] specifies NumRefFeatureEntries[ rplsldx ] for a feature tensor. The value ofNumRefFeatureEntries[ rplsldx ] shall be in the range of 0 to num_ref_entries[ rplsldx ].abs_delta_poc[ rplsldx ] [ i ] specifies the value of the variable AbsDeltaPoc[ rplsldx ][ i ] as follows:AbsDeltaPocf rplsldx ][ i ] = abs_delta_poc[ rplsldx ][ i ] + 1 sign_flag[ rplsldx ][ i ] equal to 0 specifies that DeltaPocValf rplsldx ][ i ] is greater than or equal to 0. sign_flag[ rplsldx ][ i ] equal to 1 specifies that DeltaPocValf rplsldx ][ i ] is less than 0. When not present, the value of sign_flag[ rplsldx ][ i ] is inferred to be equal to 0.The list DeltaPocValf rplsldx ] is derived as follows: for( i = 0; i < num_ref_entries[ rplsldx ]; i++ )DeltaPocVal[ rplsldx ][ i ] = ( 1 - 2 * sign_flag[ rplsldx ][ i ] ) * AbsDeltaPoc[ rplsldx ][ i ]Table 10. Example of RPL-related syntax in DLVC SH

[0045] In the slice header, one can update the number of active references. This will allow the current picture to have only one reference, a two-reference fusion, or more than two-references fusion. In Table 10, the value of sh_num_ref_idx_active_minusl is used to derive parameter NumRefldxActive, which is used to decide how many decoded reference pictures are used for inter prediction. NumRefFealdxActive is also used to decide how many decoded reference feature tensors are used for temporal fusion.Other semantics: sh_num_ref_idx_active_override_flag equal to 1 specifies that the syntax element sh_num_ref_idx_active_minusl is present for inter slices when num_ref_entries[ Rplsldx ] is greater than 1. sh_num_ref_idx_active_override_flag equal to 0 specifies that the syntax element sh_num_ref_idx_active_minusl is not present. When not present, the value of sh_num_ref_idx_active_override_flag is inferred to be equal to 1. sh_num_ref_idx_active_minusl is used for the derivation of the variable NumRefldxActive.When the current slice is a inter slice, sh_num_ref_idx_active_override_flag is equal to 1 , and sh_num_ref_idx_active_minusl is not present, sh_num_ref_idx_active_minusl is inferred to be equal to 0. sh_num_ref_feature_idx_active_minusl is used for the derivation of the variable NumRefFealdxActive.When the current slice is an inter slice, sh_num_ref_feature_idx_active_override_flag is equal to 1, and sh_num_ref_feature_idx_active_minusl is not present, sh_num_ref_feature_idx_active_minusl is inferred to be equal to 0.The variable NumRefldxActive [ i ] is derived as follows: if( sh_slice_type = = inter slice ) { if( sh_num_ref_idx_active_override_flag ) { NumRefldxActive = sh_num_ref_idx_active_minusl + 1 NumRefFealdxActive = sh_num_ref_feature_idx_active_minusl + 1 } else {NumRefldxActive = num_ref_entries[ Rplsldx ] NumRefFealdxActive = NumRefFeatureEntries [ Rplsldx ] }} else { / * sh_slice_type = = I * / NumRefldxActive = 0NumRefFealdxActive = 0}The value of NumRefldxActive- 1 specifies the maximum reference index for RPL that may be used to decode the slice. When the value of NumRefldxActiveis equal to 0, no reference index for RPL is used to decode the slice.When the current slice is a inter slice, the value of NumRefldxActive shall be greater than 0.The value of NumRefFealdxActive- 1 specifies the maximum reference feature index for RPL that may be used to decode the slice. When the value of NumRefFealdxActiveis equal to 0, no reference index for RPL is used to decode the slice.Decoding process for reference picture lists construction

[0046] This process is invoked at the beginning of the decoding process for each slice of a picture.Reference pictures are addressed through reference indices. A reference index is an index into an RPL. When decoding an I slice, no RPL is used in decoding of the slice data. When decoding an inter slice, only one RPL is used in decoding of the slice data.At the beginning of the decoding process for each slice of a picture, the RPL RefPicList and RefFeaList are derived.The RPL RefPicList and RefFeaList are derived as follows for inter-pictures: for( j = 0, pocBase = PicOrderCntVal; j < num_ref_entries[ Rplsldx]; j++) {RefPicPocList[ j ] = pocBase + DeltaPocVal[ Rplsldx][ j ] if( there is a reference picture pic A in the DPB and PicOrderCntVal equal toRefPicPocList[ j ] )RefPicList! j ] = picA elseRefPicList[ j ] = "no reference picture" if( there is a reference feature feaA in the DFB and PicOrderCntVal equal to RefPicPocList[ j ] )RefFeaList[ j ] = feaA elseRefFeaListf j ] = "no reference feature" (note: for “no reference feature”, in an embodiment, ref_feature tensors can be computed directly from picA and other feature tensor values may be set to NULL) pocBase = RefPicPocListf j ]}

[0047] FIG. 2 depicts an example of the reference picture management during decoding in DLVC. As depicted in FIG. 2, first (205), the decoder constructs a a candidate list of ref_pic_list_struct() (RPLS) in SPS, where each RPLS contains a number of reference picture entries in the DPB and a number of reference feature entries in the DFB, respectively. Next, the decoder loops over all pictures and slices. During the loop, if it detects an inter-coded slice, then, in step 210, it checks the RPLS to get an active number of a reference picture and an active number of a refence fearure. Next, it checks if rpl_sps_flag = 1. If true (see Table 8), it reads rpl_idx, an index to candidate RPLS in the SPS (215), else it calls ref_pic_list_struct() to generate new RPLS (220). This is followed by inter coding (225), and decoded pictures and features are stored in the DPB and DFB buffers (235).

[0048] For intra-coded slices, a decoder proceeds directly to intra decoding (230), and decoded pictures and features are stored in the DPB and DFB buffers.Decoding process for reference feature marking

[0049] This process is invoked once per picture, after decoding of a slice header and the decoding process for RPL construction for the slice, but prior to the decoding of the slice data. This process might result in one or more reference features in the DFB being marked as "unused for reference."A decoded feature in the DFB can be marked as "unused for reference" and "used for reference", but only one among these two at any given moment during the operation of the decoding process.Assigning one of these markings to a picture implicitly removes another of these markings when applicable.

[0050] If the current picture is a CLVSS picture (a start picture), all reference features currently in the DFB (if any) are marked as "unused for reference". Otherwise, each reference feature in the DFB that is not referred to by any entry in RefPicList is marked as "unused for reference".Buffer Model - Hypothetical Reference Decoder (HRD) specification

[0051] In an embodiment, when specifying a hypothetical reference decoder (HRD), the same DPB management as in traditional coding (e.g., VVC) can be used. For DFB management, one only needs to specify 1) the current decoded feature tensor marking and storage, 2) removal of decoded feature tensors from the DFB. Note, that unlike the DPB, decoded feature tensors are not needed for display(output) purpose.Current decoded feature tensor marking and storage

[0052] The current decoded feature tensor is stored in the DFB in an empty feature storage buffer, the DFB fullness is incremented by one, and the current feature tensor is marked as "used for reference".Removal of pictures from the DFB

[0053] The output and removal of features from the DFB before the decoding of the current picture (but after parsing the slice header of the first slice of the current picture) happens instantaneously when the first decoding init (DU) of the access unit (AU) containing the current picture is removed from the CPB and proceeds as follows:- The decoding process for RPL construction and decoding process for reference feature marking are invoked.- If the current picture is the first picture of a coded video sequence start (CVSS) AU that is not AU 0, all feature storage buffers in the DFB are emptied and the DFB fullness is set equal to 0.Note: A CVS ends when a new CVSS AU is encountered, not just when the content of the SPS or the SPS ID changes.- Otherwise, when the current picture is not the first picture of AU 0, all feature storage buffers containing a picture feature which are marked as "unused for reference" are emptied. For each picture feature storage buffer that is emptied, the DFB fullness is decremented by one. When the number of picture features in the DFB is greater than or equal to dpb_max_dec_fea_buffering_minusl[ Htid ] + 1, the one having the smallest value of PicOrderCntVal is removed. The process is repeated until the number of picture features in the DFB is less than dpb_max_dec_fea_buffering_minusl[ Htid ] + 1.ReferencesEach one of the references listed herein is incorporated by reference in its entirety.[1] Guo Lu, et al. "DVC: An end-to-end deep video compression framework. " Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.[2] Jiahao Li, Bin Li, and Yan Lu, “Deep contextual video compression, ” Advances in Neural Information Processing Systems,” arXiv:2019.15047vl, 30 Sept. 2021.[3] Xihua Sheng, Jiahao Li, Bin Li, Li Li, Dong Liu, and Yan Lu, “Temporal Context Mining for Learned Video Compression,” IEEE Transactions on Multimedia, 2022, arXiv:2111.13850vl, 27 Nov. 2021.[4] Jiahao Li, Bin Li, and Yan Lu, “Hybrid Spatial-Temporal Entropy Modelling for Neural Video Compression,” ACM MM 2022, arXiv:2207.05894vl, 13 Jul. 2022.[5] Jiahao Li, Bin Li, and Yan Lu, “Neural Video Compression with Diverse Contexts,” CVPR 2023, arXiv:2302.14402vl, 28 Feb. 2023.[6] Jiahao Li, Bin Li, and Yan Lu, “Neural Video Compression with Feature Modulation,” CVPR 2024, arXiv:2402.17414vl, 27 Feb. 2024.[7] A. Arora, et al., “Inter-coding prediction tools in video coding using deep learning,” Indian Provisional Patent Application, Ser. No. 202411065408, filed on 29 Aug. 2024. (Dolby Ref. D24033).[8] ITU-T Rec. H.266, “Versatile video coding,” (v3), 09 / 2023, ITU.EXAMPLE COMPUTER SYSTEM IMPLEMENTATION

[0054] Embodiments of the present invention may be implemented with a computer system, systems configured in electronic circuitry and components, an integrated circuit (IC) device such as a microcontroller, a field programmable gate array (FPGA), or another configurable or programmable logic device (PLD), a discrete time or digital signal processor (DSP), an application specific IC (ASIC), and / or apparatus that includes one or more of such systems, devices or components. The computer and / or IC may perform, control, or execute instructions relating to applying video coding techniques using deep learning, such as those described herein. The computer and / or IC may compute any of a variety of parameters or values that relate to applying video coding techniques using deep learning as described herein. The image and video embodiments may be implemented in hardware, software, firmware and various combinations thereof.

[0055] Certain implementations of the invention comprise computer processors which execute software instructions which cause the processors to perform a method of the invention. For example, one or more processors in a display, an encoder, a set top box, a transcoder, or the like may implement methods related to applying video coding techniques using deep learning as described above by executing software instructions in a program memory accessible to the processors. Embodiments of the invention may also be provided in the form of a program product. The program product may comprise any non-transitory and tangible medium which carries a set of computer-readable signals comprising instructions which, when executed by a data processor, cause the data processor to execute a method of the invention. Program products according to the invention may be in any of a wide variety of non-transitory and tangible forms. The program product may comprise, for example, physical media such as magnetic data storage media including floppy diskettes, hard disk drives, optical data storage media including CD ROMs, DVDs, electronic data storage media including ROMs, flash RAM, or the like. The computer-readable signals on the program product may optionally be compressed or encrypted.

[0056] Where a component (e.g. a software module, processor, assembly, device, circuit, etc.) is referred to above, unless otherwise indicated, reference to that component (including a reference to a "means") should be interpreted as including as equivalents of that component any component which performs the function of the described component (e.g., that is functionallyequivalent), including components which are not structurally equivalent to the disclosed structure which performs the function in the illustrated example embodiments of the invention.EQUIVALENTS, EXTENSIONS, ALTERNATIVES AND MISCELLANEOUS

[0057] Example embodiments that relate to applying video coding techniques using deep learning are thus described. In the foregoing specification, embodiments of the present invention have been described with reference to numerous specific details that may vary from implementation to implementation. Thus, the sole and exclusive indicator of what is the invention, and what is intended by the applicants to be the invention, is the set of claims that issue from this application, in the specific form in which such claims issue, including any subsequent correction. Any definitions expressly set forth herein for terms contained in such claims shall govern the meaning of such terms as used in the claims. Hence, no limitation, element, property, feature, advantage or attribute that is not expressly recited in a claim should limit the scope of such claim in any way. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

[0058] Various aspects of the present disclosure may be appreciated from the following Enumerated Example Embodiments (EEEs):EEE1. A method to generate a coded video bitstream, the method comprising: receiving a sequence of input pictures to be coded; using a deep learning video coding (DLVC) network to generate the coded video bitstream with a coded representation of the input pictures; and generating a signaling section characterizing the coded video bitstream, wherein the signaling section comprises: parameters related to a decoder picture buffer (DPF); parameters related to a decoder feature buffer (DFB); and a sequence-split-NN flag indicating whether the bitstream was generated using a single neural network (NN) or two neural networks, one for luma and one for chroma.EEE2. The method of EEE1, wherein the parameters related to the DFB comprise one or more of: a bit depth parameter characterizing bit depth of reference decoded features;first width and height parameters of a reference motion vector (MV) feature (ref_mv_feature) ; second width, height, and channel parameters of reference decoded latents of motion vectors (ref_mv_y_hat); third width, height, and channel parameters of the reference features (ref_feature); and fourth width, height, and channel parameters of reference decoded latents (ref_y_hat).EEE3. The method of EEE1 or EEE2, wherein the signaling section further includes a parameter specifying a maximum size of the decoder feature buffer.EEE4. The method of EEE3, wherein for a Tier or Level characterizing the coded video bitstream, the signaling section further includes a parameter indicating a maximum feature size.EEE5. The method of EEE4, wherein the maximum feature size is expressed as a multiplication factor of a corresponding maximum picture size for the Tier or Level.EEE6. The method of any preceding EEE, wherein the signaling section further includes a reset interval parameter to reset a reference feature in the DFB to none or NULL.EEE7. The method of EEE6, wherein given a reset time (resetinterval) computed with the reset interval parameter, the reference feature in the DFB is reset when POC%restInterval = 1, where % indicates a modulus operation and POC denotes picture order count.EEE8. The method of EEE6, wherein the signaling section includes in a picture header of a coded picture one or more flags to reset to NULL or not one or more of stored features in the DFB.EEE9. The method of any preceding EEE, wherein the coded video bitstream comprises a new network abstraction layer (NAL) unit of type “FDR,” which when present resets the DFB but does not reset the DPB.EEE10. The method of any preceding EEE, wherein the signaling section further comprises reference-picture and reference-feature management (RPFM) parameters.EEE11. The method of EEE10, wherein the RPFM parameters include a number of reference entries parameter (num_ref_entries) which specifies the number of entries in a reference picture list (ref_pic_list_struct( rplsldx )) syntax structure, and indicates a maximum number of references being used for inter-prediction using temporal fusion.EEE12. The method of EEE11, wherein the RPFM parameters comprise: a total number of reference lists parameter (sps_num_ref_lists); a reference process list flag (rpl_sps_flag) to indicate whether a decoder needs to generate a new reference picture list or refer via an index (rpl_idx) to an existing reference picture list.EEE13. The method of EEE12, wherein decoding the coded video bitstream comprises: constructing a candidate list of reference picture list structures (RPLS), where each RPLS contains a number of reference picture entries in the DPB and a number of reference feature entries in the DFB, respectively; for a picture in the video bitstream: if detecting intra-coding, then decoding the picture using intra coding and storing the decoded picture and picture features in the DPB and DFB buffers, else, if detecting inter coding, then: determining based on the RPLS an active number of a reference pictures and an active number of a refence features; determining via the reference process list flag whether to generate a new reference picture list or refer to an existing reference picture list; anddecoding the picture using inter coding and storing the decoded picture and picture features in the DPB and DFB buffers.EEE14. A tangible computer-readable storage medium having stored thereon computerexecutable instructions for executing with one or more processors a method in accordance with any one of the methods recited in EEE1-EEE13.EEE15. An apparatus comprising a processor and configured to perform any one of the methods recited in EEE1-EEE13.

Claims

CLAIMSWhat is claimed is:

1. A method to generate a coded video bitstream, the method comprising: receiving a sequence of input pictures to be coded; using a deep learning video coding (DLVC) network to generate the coded video bitstream with a coded representation of the input pictures; and generating a signaling section characterizing the coded video bitstream, wherein the signaling section comprises: parameters related to a decoder picture buffer (DPF); parameters related to a decoder feature buffer (DFB); and a sequence-split-NN flag indicating whether the bitstream was generated using a single neural network (NN) or two neural networks, one for luma and one for chroma.

2. The method of claim 1, wherein the parameters related to the DFB comprise one or more of: a bit depth parameter characterizing bit depth of reference decoded features; first width and height parameters of a reference motion vector (MV) feature (ref_mv_feature) ; second width, height, and channel parameters of reference decoded latents of motion vectors (ref_mv_y_hat); third width, height, and channel parameters of the reference features (ref_feature); and fourth width, height, and channel parameters of reference decoded latents (ref_y_hat).

3. The method of claim 1 or 2, wherein the signaling section further includes a parameter specifying a maximum size of the decoder feature buffer.

4. The method of claim 3, wherein for a Tier or Level characterizing the coded video bitstream, the signaling section further includes a parameter indicating a maximum feature size.

5. The method of claim 4, wherein the maximum feature size is expressed as a multiplication factor of a corresponding maximum picture size for the Tier or Level.

6. The method of any preceding claim, wherein the signaling section further includes a reset interval parameter to reset a reference feature in the DFB to none or NULL.

7. The method of claim 6, wherein given a reset time (resetinterval) computed with the reset interval parameter, the reference feature in the DFB is reset when POC%restInterval = 1, where % indicates a modulus operation and POC denotes picture order count.

8. The method of claim 6, wherein the signaling section includes in a picture header of a coded picture one or more flags to reset to NULL or not one or more of stored features in the DFB.

9. The method of any preceding claim, wherein the coded video bitstream comprises a new network abstraction layer (NAL) unit of type “FDR,” which when present resets the DFB but does not reset the DPB.

10. The method of any preceding claim, wherein the signaling section further comprises reference-picture and reference-feature management (RPFM) parameters.

11. The method of claim 10, wherein the RPFM parameters include a number of reference entries parameter (num_ref_entries) which specifies the number of entries in a reference picture list (ref_pic_list_struct( rplsldx )) syntax structure, and indicates a maximum number of references being used for inter-prediction using temporal fusion.

12. The method of claim 11, wherein the RPFM parameters comprise: a total number of reference lists parameter (sps_num_ref_lists); a reference process list flag (rpl_sps_flag) to indicate whether a decoder needs to generate a new reference picture list or refer via an index (rpl_idx) to an existing reference picture list.

13. The method of claim 12, wherein decoding the coded video bitstream comprises: constructing a candidate list of reference picture list structures (RPLS), where each RPLS contains a number of reference picture entries in the DPB and a number of reference feature entries in the DFB, respectively; for a picture in the video bitstream: if detecting intra-coding, then decoding the picture using intra coding and storing the decoded picture and picture features in the DPB and DFB buffers, else, if detecting inter coding, then: determining based on the RPLS an active number of a reference pictures and an active number of a refence features; determining via the reference process list flag whether to generate a new reference picture list or refer to an existing reference picture list; and decoding the picture using inter coding and storing the decoded picture and picture features in the DPB and DFB buffers.

14. A tangible computer-readable storage medium having stored thereon computer-executable instructions for executing with one or more processors a method in accordance with any one of the methods recited in claims 1-13.

15. An apparatus comprising a processor and configured to perform any one of the methods recited in claims 1-13.

Images