Neural network filter complexity cap

Abstract

Methods, apparatuses, and systems are described for decoding and / or encoding one or more images from a coded video bitstream. Embodiments can determine a maximum allowed complexity from one or more syntax elements in the coded video bitstream, wherein the maximum allowed complexity corresponds to a neural network filtering process; and decoding the one or more images based on the maximum allowed complexity.

Description

[0001] NEURAL NETWORK FILTER COMPLEXITY CAP

[0002] TECHNICAL FIELD

[0003] The present disclosure generally relates to systems and methods for decoding and / or encoding one or more images from a coded video bitstream.

[0004] BACKGROUND

[0005] Video is the dominant form of data traffic in today’s networks and is projected to still increase its share. One way to reduce the data traffic from video is compression. In compression the source video is encoded to a bitstream, which then can be stored and transmitted to end users. Using a decoder, an end user can extract video data and display it on a screen. However, since an encoder may not know what kind of device an encoded bitstream is going to be sent to, it often has to compress the video to a predetermined format, such as the standardized format versatile video coding, VVC. This way, all devices that support the chosen standard format can decode the video. Compression can be lossless, i.e., a decoded video will be identical to a source given to the encoder, or lossy, where a certain degradation of content may be accepted. Using lossy compression often allows for significantly lower bit rates, i.e., the compression ratio can be much higher. This is because reproducing image noise perfectly can make lossless compression quite expensive.

[0006] A video sequence may contain a sequence of pictures. A color space commonly used in video sequences is YCbCr, where Y is the luma, or brightness, component and Cb and Cr are the chroma components. Sometimes the Cb and Cr components are called U and V. Other color spaces are also used, such as ICtCp, IPT, constant-luminance YCbCr, RGB, YCoCg etc., and the present disclosure is applicable in these cases as well. The terms luma and chroma channels are sometimes used instead of luma and chroma components and are used interchangeably in this disclosure. VVC, AV L and HEVC

[0007] Versatile video coding, VVC, and its predecessor high efficiency video coding, HEVC, are block-based video codecs standardized and developed jointly by International Telecommunication Union Telecommunication Standardization Sector, ITU-T, and Moving Pictures Experts Group, MPEG. AVI is another block-based video codec specified by the Alliance for Open Media (AOM). Codecs may utilize both temporal and spatial prediction. VVC, AVI, and HEVC are similar in many aspects. Spatial prediction may be achieved using intra, I, prediction from within a current picture. Temporal prediction may be achieved using uni-directional, P, or bi-directional inter, B, prediction on the block level from previously decoded reference pictures. In an encoder, the difference between original pixel data and predicted pixel data, referred to as the residual, may be transformed into a frequency domain, quantized and then entropy coded before being transmitted together with one or more prediction parameters which may include prediction mode and motion vectors, which may also be entropy coded. Pixels and pixel values are, in this disclosure, used interchangeably with samples and sample values, respectively. A decoder may perform entropy decoding, inverse quantization and / or inverse transformation to obtain the residual, and then may add any residual to an intra or inter prediction to reconstruct a picture. The VVC version 1 specification was published as Rec. ITU-T H.266 | ISO / IEC 23090-3, “Versatile Video Coding”, in 2020, and is hereby incorporated by reference in its entirety. Exploratory work is ongoing for the next generation video codec in the joint video experts team, JVET, collaboration between ITU-T and ISO / IEC, with one track for traditional coding tools in the enhance compression model (ECM) software and one track for NN coding tools. AOM is also working on a successor to the AVI video codec, likely to be called AV2.

[0008] Blocks and Units

[0009] In many video coding standards, such as HEVC, AVI, and VVC, each component is split into blocks and the coded video bitstream consists of a series of coded blocks. A block may be a two-dimensional array of samples. It is common in video coding that the picture is split into units that cover a specific area of the picture. Each unit often consists of all blocks from all components that make up that specific area and each block belongs fully to one unit. The macroblock in H.264 and the coding unit, CU, in HEVC and VVC are examples of units. A block can alternatively be defined as a two-dimensional array to which a transform used in coding may be applied. These blocks are known under the name “transform blocks.” Alternatively, a block can be defined as a two-dimensional array to which a single prediction mode is applied. These blocks can be called “prediction blocks.” Herein, the word block is not tied to one of these definitions but rather used broadly to refer to a variety of these “blocks.” VVC Block Structure

[0010] The VVC video coding standard uses a block structure referred to as quadtree plus binary tree plus ternary tree block structure, QTBT+TT, where each picture is first partitioned into square blocks called coding tree units, CTU. The size of all CTUs is identical and the partition is done without any syntax controlling it. Each CTU is further partitioned into coding units, CUs, that can have either square or rectangular shapes. The CTU is first partitioned by a quad tree structure, then it may be further partitioned with equally sized partitions either vertically or horizontally in a binary structure to form CUs. A block could thus have a square or rectangular shape. The depth of the quad tree and binary tree can be set by an encoder in the bitstream. An example of dividing a CTU 100 using QTBT is illustrated in Figure 1. A ternary tree, TT, part may add the possibility to divide a CU 102 into three partitions instead of two equally sized partitions; this may increase the possibilities to use a block structure that better fits the content structure in a picture. The CTU 100 may comprise one or three coding tree blocks 104, CTBs, where each of the CTBs 104 may contain an NxN block of samples for a channel. For mono coded video with only one channel each CTU 100 may comprise one CTB 104 whereas for a YCbCr coded video each CTU 100 may comprise one luma CTB and two chroma CTBs. The chroma CTBs may be spatially subsampled compared to the luma CTB, e.g. by a factor of two in vertical and horizontal directions. Herein, when referring to the size of a CTU, the size of a luma CTB is sometimes used, which is equivalent. Slices and Tiles

[0011] The concept of slices in HEVC divides a picture into independently coded slices, where decoding of one slice in a picture is independent of other slices of the same picture. Different coding types could be used for slices of the same picture, i.e. a slice could either be an I-slice, P- slice or B-slice. One purpose of slices is to enable resynchronization in case of data loss. In HEVC, a slice is a set of CTUs. The VVC and HEVC video coding standards include a tool called tiles that divides a picture into rectangular spatially independent regions. Tiles in VVC are often similar to the tiles used in HEVC. Using tiles, a picture in VVC can be partitioned into rows and columns of CTUs where a tile is an intersection of a row and a column.

[0012] In VVC, a slice can be defined as an integer number of complete tiles or an integer number of consecutive complete CTU rows within a tile of a picture that are exclusively contained in a single network abstract layer, NAL, unit. In VVC, a picture may be partitioned into either raster scan slices or rectangular slices. A raster scan slice may include a number of complete tiles in raster scan order. A rectangular slice may include a group of tiles that together occupy a rectangular region in a picture or a consecutive number of CTU rows inside one tile. Each slice may have a slice header comprising syntax elements. Decoded slice header values from these syntax elements are often used when decoding the slice. Each slice may be carried in one VCL NAL unit. NAL Units

[0013] Both VVC and HEVC define a Network Abstraction Layer (NAL). All the data, i.e. both Video Coding Layer (VCL) or non-VCL data in HEVC and VVC is encapsulated in NAL units. A VCL NAL unit contains data that represents picture sample values. A non-VCL NAL unit contains additional associated data such as parameter sets and supplemental enhancement information (SEI) messages. The NAL unit in VVC and HEVC begins with a header called the NAL unit header.

[0014] Temporal Sublayers

[0015] In HEVC and in VVC, all pictures are associated with a Temporalld value which specified what temporal sublayer the picture belongs to. Temporalld values are decoded from the nuh_temporal_id_plusl syntax element in the NAL unit header. The encoder is required to set Temporalld values such that pictures belonging to a lower temporal sublayer is perfectly decodable when higher temporal sublayers are discarded. Assume for instance that an encoder has output a bitstream using temporal sublayers 0, 1 and 2. Then removing all sublayer 2 NAL units or removing all sublayer 1 and 2 NAL units will result in bitstreams that can be decoded without problems. This is ensured by restrictions in the HEVC and VVC specifications that the encoder must comply with. For instance, it is not allowed for a picture of a temporal sublayer to reference a picture of a higher temporal sublayer.

[0016] Lavers and scalability

[0017] A layer in HEVC and VVC can be defined as a set of VCL NAL units that all have a particular value of nuh layer id and the associated non-VCL NAL units. The value of the nuh layer id syntax element can be signaled in the NAL unit header and specify the layer ID of the layer to which the NAL unit belongs to. In contrast to temporal sublayers, which may be used for temporal scalability, layers in HEVC and VVC are often used for spatial scalability coding, multiview coding or for carrying 3D and other data such as depth maps. An access unit, AU, may include multiple pictures with the same picture order count, POC, but that belongs to different layers and thus may have different values of nuh layer id. In HEVC and VVC, layers may be coded independently or dependently from each other. When the layers are coded independently, a layer with e.g. nuh layer id 0 may not predict video data from another layer with e.g. nuh layer id 1. In HEVC and VVC, dependent coding between layers may be used, which may enable support for scalable coding with SNR, spatial and view scalability.

[0018] Picture Unit and Access Unit

[0019] A picture unit, PU, in VVC may be defined as a set of NAL units for which the VCL NAL units all belong to the same layer, that are associated with each other according to a specified classification rule, that are consecutive in decoding order, and that contain exactly one coded picture. In HEVC, the PU is referred to as an access unit, AU. In VVC, an access unit may be a set of PUs that belong to different scalability layers and contain coded pictures associated with the same time for output from the decoded picture buffer, DPB, i.e. having the same picture order count, POC, value.

[0020] Picture Header

[0021] In VVC, a coded picture contains a picture header structure. The picture header structure contains syntax elements that are common for all slices of the associated picture. The picture header structure may be signaled in its own non-VCL NAL unit with NAL unit type PH_NUT or included in the slice header given that there is only one slice in the coded picture.

[0022] Parameter Sets

[0023] HEVC and VVC specify three types of parameter sets, the picture parameter set, PPS, the sequence parameter set, SPS, and the video parameter set, VPS. The PPS contains data that is common for a whole picture, the SPS contains data that is common for a coded video sequence, CVS, and the VPS contains data that is common for multiple CVSs, e.g. data for multiple scalability layers in the bitstream. VVC also specifies one additional parameter set, the adaptation parameter set, APS. The APS carries parameters needed for the adaptive loop filter, ALF, tool, the luma mapping and chroma scaling, LMCS, tool and the scaling list tool.

[0024] Both HEVC and VVC may allow certain information (e.g. parameter sets) to be provided by external means. “By external means” may mean the information is not provided in the coded video bitstream but by some other means not specified in the video codec, e.g. via metadata possibly provided in a different data channel, as a constant in the decoder, or provided through an API to the decoder.

[0025] Decoding Capability Information (PCI)

[0026] In VVC there may be a DCI NAL unit. The DCI specifies information that does not change during the decoding session and may be good for the decoder to know about early and upfront, such as profile and level information. The information in the DCI is not necessary for operation of the decoding process. Decoding capability information may also contain a set of general constraints for the bitstream, that gives the decoder information of what to expect from the bitstream, in terms of coding tools, types of NAL units, etc. In VVC version 1, general constraint information can be signaled in the DCI, VPS or SPS.

[0027] Profiles, Tiers and Levels A profile in HEVC and VVC is defined as a specified subset of the syntax of the specification. VVC version 1 includes a Main 10 profile and some profiles for still picture coding, support of 4:4:4 chroma sampled pixels and support for multilayer video. In version 2 of VVC, a number of range extension profiles were added for support of higher bit-depths. HEVC may include similar profiles as VVC. HEVC and VVC define a level as a defined set of constraints on values that may be taken by syntax elements and variables. The same set of levels is defined for all profiles, with most aspects of the definition of each level being common across different profiles. Individual implementations may, within specified constraints, support a different level for each supported profile.

[0028] HEVC and VVC define a tier as a specified category of level constraints imposed on values of syntax elements in the bitstream. The level constraints are nested within a tier. A bitstream in HEVC and VVC is said to conform to the specification at a specific profile, tier and level, where the profile, tier and level are signaled in the PTL structure in the bitstream. A decoder conforming to a certain tier and level can decode all bitstreams that conform to the same tier or the lower tier of that level or any level below it. The general tier and level limits of VVC is shown in Figure 2. A specific combination of tier and level may specify maximum allowed values for picture size, a maximum size of the coded picture buffer, CPB, i.e. the pictures that need to be stored in the encoder and decoder, the number of slices and the number of tiles. A decoder conforming to the specific combination of tier and level needs to handle that.

[0029] Loop Filtering in VVC

[0030] VVC contains three in-loop filters that are not based on neural networks: a deblocking filter, a sample adaptive offset, SAG, filter, and an adaptive loop filter, ALF. The deblocking filter is used to remove block artifacts by smoothening discontinuities in horizontal and vertical directions across block boundaries. The deblocking filter uses a block boundary strength, BS, parameter to determine the filtering strength. The BS can have values 0, 1, and 2, where a larger value indicates a stronger filtering. The output of the deblocking filter is further processed by the SAG filter, and the output of SAG is then processed by ALF. The output of ALF can then be put into the decoded picture buffer, DPB, which contains decoded pictures that may be used for prediction of subsequently encoded or decoded pictures. Since the deblocking filter, the SAG filter and ALF in this way influence the pictures in the DPB used for prediction, they are classified as in-loop filters, also known as loop filters. Changes done by the loop filter may influence not only the current picture but future pictures as well. It is possible for a decoder to further filter the picture in the DPB, but not store the filtered output in the DPB. In contrast to loop filters, such a filter does not influence future predictions and is therefore classified as a post-processing filter, also known as a postfilter. Postfiltering is generally optional for decoders and thereby not required to be performed for decoder implementations to conform to a standard specification.

[0031] In ECM, a bilateral filter, BIF, has also been added. The filter is carried out in a sample adaptive offset, SAG, loop-filter stage and uses samples from deblocking as input. In ECM, each of the BIF and SAG filters creates an offset per sample, and these are added to the input sample and then clipped.

[0032] Another loop-filter that was considered in the development of VVC is the Hadamard filter. Exploration Experiment on Neural Network based Video Coding (NNVC)

[0033] At the 20th JVET meeting it was decided to set up an exploration experiment, EE, on neural network-based, NN-based, video coding. The exploration experiment continued at the subsequent JVET meetings 21 through 36 and continued with many tests: NN-based in-loop filtering, NN- based post filtering, NN-based super resolution and NN-based intra prediction.

[0034] The contributions JVET-X0066 and JVET-Y0143 are two successive contributions that describe NN-based in-loop filtering. Both contributions use the same NN models for filtering. The NN-based in-loop filter is placed before SAG and ALF and the sample values before the deblocking filter are used as input to the filter. The output of the NN-based in-loop filter is mixed, or blended, with the output of the deblocking filter and forwarded as the input to SAG. The purpose of using the NN-based in-loop filter is to improve the quality of the reconstructed samples. It may be helpful that the NN model is non-linear. While deblocking, SAG and ALF all contain nonlinear elements such as conditions, and are thus not strictly linear. All three of them are based on linear filters. In contrast, a sufficiently large NN model can, in principle, leam any non-linear mapping and is therefore capable of representing a wider class of functions compared to deblocking, SAG and ALF. In JVET-X0066 and JVET-Y0143, there are four NN models, i.e., four NN-based in-loop filters. In a refined version of that work presented in the contribution JVET- AB0052, only two models are used: one model for luma samples and another model for chroma samples.

[0035] Contribution JVET-AD0380 proposed a new unified design for the NN-based in-loop filtering, which captures the benefits of previous NN structures. The unified filter has only one NN model, to filter luma and chroma samples and intra and inter pictures. Figure 3 illustrates a unified NN filter 300.

[0036] There are six inputs, the reconstructed samples of luma and chroma before deblocking, rec 302, the prediction samples of luma and chroma, pred 304, the deblocking block BS information of luma and chroma, bs 306, the quantization parameter for a sequence, QPbase 308, the quantization parameter for each slice, QPslice 310, as well as information on whether a particular sample was intra-predicted, uni-predicted or bi -predicted, IPB 312. These inputs first go through a convolutional layer 314, which may be 3x3 or 1x1, and a parametric rectified linear unit 316, PreLU, layer separately, then they are concatenated and fused together with a 1x1 convolutional layer 318.

[0037] A convolutional layer with a kernel size of 3x3 is denoted “conv 3x3” and a convolutional layer with a kernel size of 1x1 is denoted “conv 1x1”. The values dl, d2, ... are the numbers of the output channels 320. In this figure, ”J,2” denotes downsampling with a factor of 2 and ” j'2” denotes upsampling with a factor of 2. There are N backbone blocks 322. Inside a backbone block, a convolutional layer “conv 3x1 CxC21” 324 means the convolution kernel is of size 3x1, the number of input channels is C and the number of output channels is C21.

[0038] The NN-based in-loop filters presented in JVET-X0066, JVET-AB0053, JVET-AB0052 and JVET-AD0380 increase the compression efficiency of the codec substantially, i.e., they lower the bit rate substantially without lowering the objective quality as measured by mean squared error, MSE, based peak signal-to-noise ratio, PSNR. Increases in compression efficiency, typically referred to simply as “gain”, are often measured as the Bjontegaard-delta rate, BDR, against an anchor. As an example, a BDR of -1% means that the same PSNR distortion can be reached with 1% bitrate saving on average. As reported in JVET-AF0041, for the random access, RA, configuration, the BDR for the luma component, Y, is -10.27%, and for the all-intra, Al, configuration, the BDR for the luma component is -7.86%. The complexity of NN models used for compression are often measured by multiply-accumulate operations per pixel, MAC / pixel. The high bitrate savings of an NN model is typically directly related to the complexity of the NN model. The model described in JVET-AF0041 has a complexity of 477 kMAC / pixel, i.e., 477,000 multiply-accumulate operations per pixel. There are also other measures of complexity, such as total model size in terms of stored parameters.

[0039] To further investigate the tradeoff between the compression efficiency and complexity of the NN-based in-loop filters, new operation points have been introduced. The two new operation points are a low operation point, LOP, with a complexity of about 17 kMAC / pixel and a very low operation point, VLOP, with a complexity of about 5 kMAC / pixel. Figure 4 illustrates an example structure of a low operation point, LOP, architecture neural network, NN, loop filter. A low operation point, LOP, architecture NN loop filter used in a JVET EE test, EE1-1.0, is illustrated in Figure 4 from JVET-AG2023. In Figure 4, the input patch size is 144x144 and there is a final cropping step 402 that crops 8 pixels from each side of the output luma patch and 4 pixels from each side of each chroma patch, which has half the width and height of the luma patch. Hence the final patch size after cropping in the output is equal to 128x128 for luma and 64x64 for the chroma branch. The complexity of this current model is about 17 kMAC per sample. This means that about 17,000 multiplications are performed for calculating one sample value in the output of neural network loop filter architecture in Figure 4.

[0040] The architecture of VLOP is similar to LOP, but the complexity is only about 5 kMAC / pixel. Compared to LOP, VLOP has a smaller number of backbone blocks and channels. In JVET standardization, development and study of codecs is done using common test conditions, CTC. The CTC specifies how a codec under test should be configured and what test sequences should be used. Keeping the test conditions static enables apples-to-apples evaluations, but with the drawback that configurations outside the CTC are not tested much and the codec performance may become too optimized towards the CTC.

[0041] NNVC 10.0 Syntax and Semantics

[0042] The NN architecture in NNVC-10.0 is the current available version of the software. Note that there is no text containing the syntax and semantics shown in this section, this was all written by the authors of this disclosure from the NNVC software available on the following HHI repository https: / / vcgit.hhi.fraunhofer.de / jvet-ahg-nnvc / VVCSoftware_VTM where the NNVC- 10.0 software is tagged as vlOrc. A sequence parameter set 500 is shown in Figure 5, which is from NNVC-10.0 when NN_LF_UNIFIED=1. sps nnlf enabled flag equal to 1 specifies that NN in-loop filtering is enabled for the CLVS. sps_nnlf_enabled_flag equal to 0 specifies that NN inloop filtering is disabled for the CLVS. sps_nnlf_model_id equal to 0 specifies that NN in-loop filtering set 0 may be used. sps_nnlf_model_id equal to 1 specifies that NN in-loop filtering set 1 may be used, sps nnlf model id equal to 2 specifies that the LOP1 in-loop filtering may be used, sps nnlf set equal to 3 specifies that the HOP or LOP in-loop filtering may be used, sps nnlf model id equal to 4 specifies that LOP3 in-loop filtering may be used and sps nnlf model id equal to 5 specifies that HOP4 in-loop filtering may be used. sps_nnlf_unified_infer_size_base specifies the base inference size of NN-based in-loop filter. When not present in the bitstream, the value of sps_nnlf_unified_infer_size_base is inferred to be equal to 128. sps_nnlf_unified_inf_size_ext specifies the extension of inference size of NN-based in-loop filter. When not present in the bitstream, the value of sps_nnlf_unified_inf_size_ext is inferred to be equal to 8. sps_nnlf_unified_max_num_prms specifies the number of the conditional parameters of the NN-based in-loop filter. When not present in the bitstream, the value of sps_nnlf_unified_max_num_prms is inferred to be equal to 2. sps_nn_intra_pred_enabled_flag equal to 1 specifies that NN intra prediction tool is enabled for the CLVS. sps_nn_intra_pred_enabled_flag equal to 0 specifies that NN intra prediction tool is disabled for the CLVS. Further in NNVC-10.0 software, if the slice is an I slice, the patch size will be set equal to sps_nnlf_unified_infer_size_base«l where «1 is a left shift doubling the signalled value. Further in NNVC-10.0 software, if the slice is not an I-slice, for QP<29 the patch size will be set equal to sps nnlf unified infer size base, and if QP>29, the patch size will be set equal to sps_nnlf_unified_infer_size_base if the picture width is smaller than 823, and to sps_nnlf_unified_infer_size_base«l for wider pictures.

[0043] Figure 6 shows a slice header 600. slice_nnlf_unified_mode equal to 0 specifies that no filtering is done for the slice. slice_nnlf_unified_mode equal to 1 specifies that filtering is done with prmld equal to 0 for all NN blocks in the slice. slice_nnlf_unified_mode equal to 2 specifies that filtering is done with prmld equal to 1 for all NN blocks in the slice. slice_nnlf_unified_mode equal to 3 specifies that prmld is decoded for each NN block in the slice. slice_nnlf_unified_scale_flag equal to 1 specifies that there are scaling factors in the bitstream for NN in-loop filtering. slice_nnlf_unified_scale_flag equal to 0 specifies that no scaling factor for NN in-loop filtering is signalled in the bitstream. y_nnScale specifies the Y component of the NN in-loop filter scale factor. cb_nnScale specifies the Cb component of the NN in-loop filter scale factor. cr_nnScale specifies the Cr component of the NN in-loop filter scale factor.

[0044] Residue scaling is also used. In the existing code, NNVC-10.0, when the NN loop filter is applied to reconstructed pictures, a scaling factor is derived and signaled for each color component in the slice header. The derivation is based on least square method. The difference between the input samples and the NN filtered samples, or residues, are scaled by the scaling factors before being added to input samples.

[0045] Figure 7 illustrates an exemplary combination of reconstructed samples from a deblocking filter 702 and a NN in-loop filter 704. In the existing code, NNVC-10.0, the final filtered samples are generated by blending the result of the deblocking filter 702 and the NN in-loop filter 704. This is illustrated in Figure 7, which shows blending between the output of the deblocking filter 702 and the NN-based loop filter 704. The blending of the two results is expressed as:

[0046] R = w X RNN + ( 1 - w ) X RDB Where R is the result after blending, RNN and RDB are the results from the NN in-loop filter and the deblocking filter respectively, and w is a weight factor from a set of predefined weights. In the current code, the weight factor is signaled per NN block.

[0047] JVET-AF0193 (AHG11: Decoder complexity optimization for NNVC in-loop filter)

[0048] JVET-AF0193 (AHG11: Decoder complexity optimization for NNVC in-loop filter) proposes an encoder-only coding tool to optimize the decoder complexity of NNVC in-loop filter. In this method, the filter gain of each CTU is first calculated if the block-level NNLF on / off is enabled. Then, NNLF is disabled for the CTUs with small filter gains. Experimental results compared to the NNVC-6.0 HOP anchor show BDR results of Y: 0.05% Cb: 0.03% Cr: 0.01% with encoding time, EncT: 98% and decoding time, DecT: 91% in random access, RA, configuration, i.e. a test configuration used in JVET with periodic intra pictures. Also, the experimental results compared to the NNVC-6.0 LOP anchor show BDR results of Y : 0.06%, Cb: 0.01%, Cr: 0.04%, with EncT: 100%, and DecT: 87% for LOP in RA configuration. JVET-AJ0141 (AHG11: Adaptive skip of LOP filtering based on boundary strength)

[0049] JVET-AJ0141 (AHG11: Adaptive skip of LOP filtering based on boundary strength) proposes a method to reduce the decoding time of the LOP filter by adaptively skipping LOP filtering based on BS partitions. The proposed method determines the number of BS partitions from the BS map, which is used for an input to the LOP architecture and skips the LOP filtering if the number of BS partitions is lower than the BS partition threshold that is signaled in slice header. The experimental results of this method based onNNVC-10.0 show BDR results ofY: 0.10%, Cb: -0.11%, Cr: -0.15% with EncT: 100%, and DecT: 75% in RA configuration.

[0050] SUMMARY

[0051] One embodiment under the present disclosure comprises a method for decoding one or more pictures from a coded video bitstream, the method comprising determining a maximum allowed complexity from one or more syntax elements in the coded video bitstream, wherein the maximum allowed complexity corresponds to a neural network filtering process; decoding the one or more pictures based on the maximum allowed complexity.

[0052] Another embodiment of a method under the present disclosure is a method for encoding one or more pictures to a coded video bitstream, the method comprising obtaining a maximum allowed complexity for a neural network filter; determining, based on the maximum allowed complexity, one or more portions of each of the one or more pictures to encode using the neural network filter; filtering the one or more portions using the neural network filter, forming one or more filtered portions; and encoding each picture of the one or more pictures according to the one or more filtered portions and one or more remaining portions of the picture.

[0053] Another embodiment under the present disclosure comprises an apparatus for decoding a picture comprising a neural network filtering process from a coded video bitstream, comprising: processing circuitry configured to perform any of the steps disclosed herein; and power supply circuitry configured to supply power to the processing circuitry.

[0054] Another embodiment under the present disclosure comprises an apparatus for decoding a picture comprising a neural network filtering process from a coded video bitstream, comprising processing circuitry; and a memory storing instructions whereby the processing circuitry is operable to perform the steps of: determining a maximum allowed complexity from one or more syntax elements in the coded video bitstream, wherein the maximum allowed complexity corresponds to a neural network filtering process; decoding the one or more pictures based on the maximum allowed complexity.

[0055] Another embodiment under the present disclosure comprises an apparatus for encoding a picture comprising a neural network filtering process from a coded video bitstream, comprising processing circuitry; and a memory storing instructions whereby the processing circuitry is operable to perform the steps of: obtaining a maximum allowed complexity for a neural network filter; determining, based on the maximum allowed complexity, one or more portions of each of the one or more pictures to encode using the neural network filter; filtering the one or more portions using the neural network filter, forming one or more filtered portions; and encoding each picture of the one or more pictures according to the one or more filtered portions and one or more remaining portions of the picture.

[0056] This summary is provided to introduce a selection of concepts in a simplified form that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used as an indication of the scope of the claimed subject matter.

[0057] BRIEF DESCRIPTION OF THE DRAWINGS

[0058] For a more complete understanding of the present disclosure, reference is now made to the following descriptions taken in conjunction with the accompanying drawings, in which: Fig. 1 illustrates an example of dividing called coding tree unit, CTU, using quadtree plus binary tree, QTBT;

[0059] Fig. 2 illustrates a table showing general tier and level limits of versatile video coding, VVC;

[0060] Fig. 3 illustrates a unified NN filter.

[0061] Fig. 4 illustrates an example structure of a low operation point, LOP, architecture neural network, NN, loop filter;

[0062] Fig. 5 illustrates a table showing a sequence parameter set under the present disclosure;

[0063] Fig. 6 illustrates an exemplary slice header under the present disclosure;

[0064] Fig. 7 illustrates an exemplary combination of reconstructed samples from a deblocking filter and a NN in-loop filter;

[0065] Fig. 8 illustrates a flow-chart of a method embodiment under the present disclosure;

[0066] Fig. 9 illustrates a flow-chart of a method embodiment under the present disclosure;

[0067] Fig. 10 illustrates a flow-chart of a method embodiment under the present disclosure;

[0068] Fig. 11 illustrates an exemplary NN block and an exemplary input patch under the present disclosure;

[0069] Fig. 12 illustrates an exemplary image divided into NN blocks;

[0070] Fig. 13 illustrates exemplary syntax for signaling a number of NN blocks that use NN loop filtering in a picture parameter set, PPS;

[0071] Fig. 14 illustrates an exemplary slice header;

[0072] Fig. 15 illustrates exemplary slice data;

[0073] Fig. 16 illustrates exemplary syntax of signaling delta values for a cap for each temporal layer in a sequence parameter set;

[0074] Fig. 17 illustrates a table illustrating exemplary level and tier constraints;

[0075] Fig. 18 illustrates an exemplary hierarchical structure for transmitting images in a video sequence;

[0076] Fig. 19 illustrates a chart showing Bjontegaard-delta rate, BDR, as a function of decoding time, DecT;

[0077] Fig. 20 illustrates a flow-chart of a method embodiment under the present disclosure;

[0078] Fig. 21 illustrates a flow-chart of a method embodiment under the present disclosure;

[0079] Fig. 22 shows a schematic of a communication system embodiment under the present disclosure;

[0080] Fig. 23 shows a schematic of a user equipment embodiment under the present disclosure; Fig. 24 shows a schematic of a network node embodiment under the present disclosure;

[0081] Fig. 25 shows a schematic of a virtualization environment embodiment under the present disclosure; and

[0082] Fig. 26 shows a schematic of a wireless device embodiment under the present disclosure.

[0083] DETAILED DESCRIPTION

[0084] This application claims the benefit of United States of America priority application No. 63 / 715,381 filed on November 1, 2024, titled “NN Filter Complexity Cap,” and United States of America priority application No. 63 / 806,803 filed on May 16, 2025, titled “NN Filter Complexity Cap.”

[0085] Before describing various embodiments of the present disclosure in detail, it is to be understood that this disclosure is not limited to the parameters of the particularly exemplified systems, methods, apparatus, products, processes, and / or kits, which may, of course, vary. Thus, while certain embodiments of the present disclosure will be described in detail, with reference to specific configurations, parameters, components, elements, etc., the descriptions are illustrative and are not to be construed as limiting the scope of the claimed embodiments. In addition, the terminology used herein is for the purpose of describing the embodiments and is not necessarily intended to limit the scope of the claimed embodiments. Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art. Some of the embodiments contemplated herein will now be described more fully with reference to the accompanying drawings. Embodiments are provided by way of example to convey the scope of the subject matter to those skilled in the art.

[0086] The following terminology is defined as used in the present disclosure. Picture portion: Any portion of a picture including but not limited to a slice, a tile, or a subpicture in the picture. NN block: A block of samples (e.g. 128x128) in the picture used by the NN for the NN filtering. Border area: An area in the picture around the NN block. Patch: A block of elements with values that serve as input to the NN or output from the NN. The NN takes an input patch and produces an output patch. An input patch could be with or without a margin. An output patch could be with or without cropping. Central area: An input patch minus margin. The central area of the input patch may be the same size as the NN block. Margin: A part of the patch outside the central area that may include a border area sample values or non-sample values. Input block: An area in a picture having the same size as the input patch corresponding to the NN block. kMAC / pixel: The number of multiplication and accumulation operations required per one pixel. kMAC / pixel is a measure of complexity for a neural network when the pixel is a pixel in the output patch. Average kMAC / pixel: The number of multiplication and accumulation operations on average required per one pixel. If the average is computed over one picture, then the pixel can be any pixel in that picture. The average could be calculated over any picture portion or set of pictures. The relation between the terms is illustrated in Figure 11, which illustrates an NN block 1100 and input patch 1106. As can be seen, the left block is from the input picture, and the right block is the input patch that is input to the NN. The NN block 1100 is part of a picture and may include an input block 1102 and a border area 1104. The input patch 1106 may include a central area 1108 and a margin 1110. The picture may for example be divided into NN blocks 1200 as illustrated in Figure 12. In the picture the NN-blocks 1200 are marked with solid lines. A border area 1202 is marked with dotted lines. A picture 1204 is marked in dashed lines. There could also be other ways of dividing the picture and extracting NN blocks, e.g. the NN blocks may be sparsely distributed, or they may be overlapping.

[0087] An input block, an NN block, and / or an input patch may have multiple channels, e.g. one for luma and two for chroma. An input block, an NN block, and / or an input patch may alternatively comprise one of the channels. The size, or resolution, of the channels could be different. For example, the luma channel may have the size MxN but the chroma channels may have the size M / 2 x N / 2.

[0088] When an NN loop filter is applied to all NN blocks in a picture, the kMAC / pixel may be the same for all the pixels in that picture. When not all the blocks in the picture go through the NN loop filter, the average complexity changes. The new measure of complexity for the NN loop filtering of the picture could be defined as “average kMAC / pixel” for pixels in the picture, which may be defined to be the total number of multiplication-accumulation that is performed for NN loop filtering of the blocks in the picture going through the NN loop filter divided by the number of the pixels in the picture. “Average kMAC / pixel” can be similarly defined for a picture portion or set of pictures. When all pixels in a picture go through the same complexity NN loop filter, the average kMAC / pixel may be the same for the picture and any portion of the picture.

[0089] There currently exist certain challenges. Investigations in exploration experiment, EE, on NN-based video coding, JVET-EE1, show a trade-off between complexity and BDR gain for NN loop filters. Currently, several operation points for a NN loop filter are being investigated in JVET- EE1, namely HOP, LOP and VLOP. Among these operation points, VLOP has the lowest complexity with about 5 kMAC / pixel, yet even VLOP may still be computationally too expensive for some hardware. In this context, solutions that further reduce the computational complexity while providing reasonable BDR gain may be beneficial.

[0090] JVET-AF0193 provides another gain-complexity trade-off by introducing an encoder only tool to adaptively disable the NN loop filtering at the CTU level based on NN filter gain. In this method, CTUs with gain lower than a set threshold may not use NN loop filtering. One problem with this method is that in one picture there could be many CTUs that use NN loop filtering and in another picture this number can be only a few. This makes the decoding of some pictures more computationally expensive than other pictures. Another problem with this method is that this method does not guarantee the reduction of worst-case complexity. In an extreme example, the gain for all the CTUs in a picture may be equal or above the set threshold and so all the CTUs in that picture will use NN loop filtering. Hence, the worst-case complexity scenario is not improved. JVET-AJ0141 provides another gain complexity trade-off by adaptively skipping LOP filtering based on BS partitions. Although the complexity can be adjusted in this method by adjusting the threshold value signaled in the slice header, still both problems mentioned for the case of JVET- AF0193 exist.

[0091] Certain aspects of the disclosure and their embodiments may provide solutions to these or other challenges. Certain embodiments can set a limit on the usage of NN loop filtering for pictures or portions of the pictures in a video codec. For example, in one embodiment, a cap value, or a complexity budget, may be specified for usage of the NN loop filter which may limit the worstcase complexity of NN loop filtering process for decoding a picture, a portion of a picture, a set of pictures, or a coded video sequence, CVS.

[0092] In a second embodiment, the complexity budget may be used for specific blocks or areas in a picture. This may provide a means for potentially improving subjective quality or coding efficiency while staying within a specified complexity budget. In a third embodiment, per block signaling of NN loop filtering being on-off may happen only until a number of blocks in the picture that use NN loop filtering reaches a cap. This may save bits by omitting signaling when the cap is reached. In a fourth embodiment, a cap value and / or marking of on-off loop filtering per block may be inherited from another picture or a neighboring block. This may remove unnecessary signaling if the cap value and / or the marking of the on-off loop filtering per block is the same between pictures or can be inherited from another picture or a neighboring block.

[0093] In a fifth embodiment, methods such as a two-pass method may be used in an encoder to determine where to use a complexity budget. The encoder may be optimized towards higher coding performance or potentially provide another tradeoff between encoder complexity, or speed, and coding performance. In a sixth embodiment, a level and / or tier or a codec profile may specify a complexity cap for a picture or a set of pictures.

[0094] Embodiments may include methods and systems for decoding / decoders and / or encoding / encoders. For example, as shown in Figure 8, certain decoder embodiments can include a method 800 for decoding a picture which may include a neural network filtering process from a coded video bitstream, the method 800 comprising: determining a maximum allowed complexity from one or more syntax elements in the bitstream 802, wherein the maximum allowed complexity corresponds to the neural network filtering process; and decoding the picture using the maximum allowed complexity 804.

[0095] Certain alternative embodiments, as shown in Figure 9, can comprise a method 900 for decoding a picture comprising a neural network filtering process from a coded video bitstream, the method 900 comprising: obtaining an indication of a maximum allowed complexity 902, wherein the maximum allowed complexity corresponds to the neural network filtering process; and decoding the picture using the maximum allowed complexity 904. Some variations can be characterized in that the indication of the maximum allowed complexity is derived from one or more syntax elements in the bitstream. Some variations can be characterized in that the maximum allowed complexity is a constraint on the bitstream and / or the decoder.

[0096] In some variations, the maximum allowed complexity is specified for a picture, a portion of a picture, a set of pictures, or a coded video sequence, CVS. In some variations the maximum allowed complexity can be defined as one or more of: the maximum allowed MAC / pixel for NN filtering; the maximum allowed samples that use NN filtering; the maximum allowed percentage, or ratio, of samples that use NN filtering; the maximum allowed NN blocks or percentage, or ratio, of NN blocks that use NN filtering, and the maximum allowed CTUs or percentage, or ratio, of CTUs that use NN filtering

[0097] In some variations, decoding a picture using the maximum allowed complexity comprises: determining that a current portion of the picture is not using the neural network filtering process in response to a complexity value being larger than the maximum allowed complexity.

[0098] Some variations can be characterized such that decoding the picture using the maximum allowed complexity comprises: determining that a current portion of the picture is using the neural network filtering process in response to a complexity value being smaller than or equal to the maximum allowed complexity. This embodiment can be characterized such that determining that a current portion of the picture is using the neural network filtering process further comprises: decoding a first syntax element from the coded video bitstream into a first value, wherein the first value specifies that the current portion of the picture uses neural network filtering. The syntax element may be called, for example, ctu nn loop filter flag. These embodiments can be further characterized such that determining that a current portion of the picture is using the neural network filtering process may include considering one or more of: the scanning order of the portions in the picture; the current portion of the picture consists of a block which is marked by a block importance mapping, BIM, tool; the BS information of at least one block in the current portion of the picture; block type of at least one block in the current portion of the picture; QP of at least one block in the current portion of the picture; a picture type of the picture containing the current portion of the picture; a temporal ID of the picture containing the current portion of the picture; and / or layer ID of the picture containing the current portion of the picture.

[0099] In some variations of the preceding embodiments, the maximum allowed complexity from one or more syntax elements in the bitstream can be determined in response to a decoded value from a second syntax element in the bitstream being equal to a first value. The syntax element may be called, for example, sps nnlf enabled flag. In some variations of the preceding embodiments, the maximum allowed complexity may be decoded from a parameter set such as a PPS, SPS, VPS, or a header such as a picture header or a slice header, or a decoding capability information, DCI, NAL unit.

[0100] In some variations of the preceding embodiments, a tier or a codec profile may specify the maximum allowed complexity. In one variation of the preceding embodiment, two or more profiles may be specified where a first profile such as a main profile does not allow neural network filtering, and a second profile such as an extended profile allows neural network filtering. In one variation of the preceding embodiment, two or more profiles may be specified where a first profile may allow a first type of neural network filtering, and a second profile may allow a second type of neural network filtering.

[0101] In some variations of the preceding embodiments, at least one of the following applies: at least one block of a picture is using NN filtering and at least one block of the picture is not using NN-filtering; at least one portion of a picture is using NN filtering and at least one portion of the picture is not using NN-filtering; at least one picture in a set of pictures in the bitstream is using NN-filtering and at least one picture in a set of pictures in the bitstream is not using NN-filtering.

[0102] In other examples, certain embodiments can include encoder-based embodiments. For example, certain encoder embodiments can comprise a method for encoding a picture to a coded video bitstream, as illustrated in Figure 10, the method 1000 comprising a NN-filtering process, the method 1000 further comprising: obtaining a maximum allowed complexity for the NN- filtering 1002; determining, based on the obtained complexity cap N, which portions in the picture to encode using NN-filtering and which portions to not encode using NN-filtering 1004; and encoding the picture using NN-filtering for the portions determined to use NN-filtering 1006 and no NN-filtering for all other portions.

[0103] Some variations can further comprise encoding an indication of the maximum allowed complexity for the NN-filtering to the bitstream. In some variations of these embodiments, steps further comprise calculating or estimating performance gains of using NN-filtering for each portion in the picture. Some variations of the preceding embodiment can be characterized such that the determining is further based on the calculated or estimated performance gains for each portion in the picture.

[0104] Some variations of the preceding embodiments can be characterized such that calculating or estimating the performance gains comprises one or more of the following: subsample the portions in the picture; encode the picture using NN-filtering for all portions; encode the picture without using NN-filtering for any portions; calculate a performance metric for each portion encoded using NN-filtering; calculate a performance metric for each portion encoded not using NN-filtering; calculate a difference between the first performance metric and the second performance metric for each portion; prioritize to use NN-filtering for the portions for which the calculated difference has the highest performance gains of using NN-filtering; prioritize a portion based on the portion types; prioritize a portion based on the amount of boundary partitioning.

[0105] Some variations of the preceding embodiments can be characterized such that the determining that a current portion of the picture is to be encoded using NN-filtering further takes into account one or more of: the scanning order of the portions in the picture; the current portion of the picture consists of a block which is marked by block importance mapping, BIM, tool; the BS information of at least one block in the current portion of the picture; block type of at least one block in the current portion of the picture; QP of at least one block in the current portion of the picture; a picture type of the picture containing the current portion of the picture; a temporal ID of the picture containing the current portion of the picture; and / or a layer ID of the picture containing the current portion of the picture.

[0106] Any of the preceding embodiments can be characterized such that the maximum allowed complexity is a constraint on the bitstream and / or the decoder. Any of the preceding embodiments can be characterized such that the maximum allowed complexity is specified for a picture, a portion of a picture, a set of pictures, or a coded video sequence, CVS. Any of the preceding embodiments can be characterized such that the maximum allowed complexity is defined as one or more of: the maximum allowed MAC / pixel for NN filtering; the maximum allowed samples that use NN filtering; the maximum allowed percentage, or ratio, of samples that use NN filtering; and / or the maximum allowed NN blocks or percentage, or ratio, of NN blocks that use NN filtering, and the maximum allowed CTUs or percentage, or ratio, of CTUs that use NN filtering. Any of the preceding embodiments can be characterized in that the steps further comprise encoding a first value to a first syntax element in the bitstream, wherein the first value specifies that the current portion of the picture uses neural network filtering. The syntax element may be called, for example, ctu nn loop filter flag. Any of the preceding embodiments can be characterized such that the steps further comprise encoding a second value to a second syntax element in the bitstream, wherein the maximum allowed complexity may be obtained if the second value is equal to e.g. 1. The second syntax element may be called, for example, sps nnlf enabled flag.

[0107] Any of the preceding embodiments can be characterized such that the indication of the maximum allowed complexity is encoded to a parameter set such as a PPS, SPS, VPS, and / or a header such as a picture header or a slice header, or a decoding capability information, DCI, NAL unit. Any of the preceding embodiments can be characterized such that a level, a tier, and / or a codec profile specifies the maximum allowed complexity. Any of the preceding embodiments can be characterized such that at least one of the following applies: at least one block of a picture is using NN filtering and at least one block of the picture is not using NN-filtering; at least one portion of a picture is using NN filtering and at least one portion of the picture is not using NN- filtering; and / or at least one picture in a set of pictures in the bitstream is using NN-filtering and at least one picture in a set of pictures in the bitstream is not using NN-filtering.

[0108] Certain embodiments may provide one or more of the following technical advantage(s). One advantage of certain proposed embodiments may be that a worst-case decoder complexity for NN loop filtering per picture, portion of a picture or set of pictures can be guaranteed. Another advantage of the proposed embodiments may be that a guaranteed complexity can be achieved without a need for ultra-low complexity NN loop filters. When, for example, at most half of the CTUs in a picture may be allowed to use NN loop filtering. The average kMAC / pixel for the picture will be half of what could be with the same NN loop filter being applied to all the CTUs in the picture. Another advantage can be creating a new way to control a complexity which can be combined with other methods for adjusting a complexity / gain trade-off. For example, the complexity budget that may be set by a proposed method can be spent on areas of the image with higher potential for compression gain in order to maximize compression gain. In another example, a complexity budget set by the proposed method can be spent on region of interests, ROIs, in the picture to improve a subjective quality of the codec output. Another advantage can be that a lower worst-case memory requirement for NN loop filtering in the decoder can be guaranteed, without this, some applications run into a bottleneck and fail or experience significant latency. The guarantee of a worst-case memory requirement may also be specified for certain codec levels and / or coding profiles. Another advantage can be that a solution for responding to the power requirements of the decoder for NN loop filtering may be provided. Some aspects of the proposed embodiments may simplify an encoding process as well. For instance, an encoder may be simplified, when N blocks in the picture that use NN loop filtering may be chosen based on the BS values in the block rather than an exhaustive rate distortion optimization process. Certain embodiments may improve data rate, latency, and / or power consumption.

[0109] Several embodiments are described below. It should be understood that these examples are non-limiting and should not be taken to be an exhaustive list of all possible embodiments under the present disclosure.

[0110] Example Embodiment 1

[0111] One example embodiment can comprise specifying a cap on the usage of the NN loop filter. In this embodiment the usage of the NN loop filter can be confined, in a way that not more than a certain number of computations is allowed for NN loop filtering of a picture or a picture portion or a set of pictures or an access unit. In some versions, a cap may be specified for the number of pixels or number of blocks in the picture that can use NN loop filtering. In one example the cap may be specified for the number of CTU blocks in the picture that can use NN loop filtering. In another example the cap may be specified for a fixed size NxN block, where N may, for example, be 128.

[0112] In some versions, the cap may be specified for different block types. For example, blocks can have different prediction modes, i.e., intra-coded, uni-predicted, or bi-predicted, and different caps may be specified. The cap may be specified per picture, per picture portion (slice, tile, subpicture, etc.) or per set of pictures (pictures in one group of pictures, GOP, pictures in a temporal sublayer in a GOP, pictures in the same layer in a GOP, N subsequent pictures in decoding order, N subsequent pictures in output order, pictures referencing the same picture parameter set, etc.). Note that layer is not the same thing as temporal sublayer.

[0113] In some versions, the cap may be specified for a ratio, e.g. a percentage of pixels, a percentage of blocks, a percentage of CTUs that may use NN loop filtering for a picture, a picture portion or a set of pictures, etc. In one example, a 50% cap is specified for the percentage of the CTUs in the picture that may use NN loop filtering, which specifies that no more than half of the CTUs in the picture may use the NN loop filtering process. In this example, if there are 8 CTUs in a picture, only a maximum of 4 of them may use NN loop filtering, and if there are 32 CTUs in the picture, a maximum of 16 of them may use NN loop filtering. If the specified ratio is not giving a full number of CTUs considering the number of CTUs in a picture, for instance the cap is specified as a 10% ratio and there are 32 CTUs in the picture, obtaining the cap might include rounding towards the smaller, in this case 2, or larger number, in this case 3.

[0114] In a version of this embodiment, the cap may be specified based on the size of the margin of the input patch. In one example, if the margin is equal to 8 pixels, then a 50% cap may be specified for the number of CTUs that may use NN loop filtering and if the margin is equal to 16 pixels, then a 40% cap may be specified for the number of CTUs that may use NN loop filtering. This may be to compensate for any extra computation caused by the usage of a larger margin and in order to keep the worst-case computational complexity within a limit.

[0115] In some versions, the cap may be specified based on the size of the NN block or the size of the input block. In one example, if the size of the NN block is equal to 128x128 pixels then a cap of 40% may be specified for the number of the NN blocks that may use NN loop filtering, and if the size of the NN block is equal to 256x256 pixels then a cap of 50% may be specified for the number of NN blocks that may use NN loop filtering.

[0116] In another version of this embodiment, a complexity budget may be specified where the NN loop filtering of the picture is confined to this computation budget while the budget distribution may change. In one example of this version, the complexity budget C may be specified and there may be two NN loop filters, the first with complexity Cl and the second with complexity C2. In this example, any total complexity of (Nl x Cl) + (N2 x C2) < C may be allowed where N1 is the number of CTUs using the first NN loop filter and N2 is the number of CTUs using the second NN loop filter. The hard limit may be such that the total complexity does not exceed the complexity budget. The complexity budget C may, for example, be a limit of the maximum allowed average kMAC / pixel for NN filtering of the picture. For example, in a scenario with a picture with 10

[0117] CTUs where a complexity CO uses 0 kMAC / pixel (no filtering), Cl uses 5 kMAC / pixel, C2 uses

[0118] 20 kMAC / pixel and C is a limit of 3,5 kMAC / pixel for the picture, it would be sufficient to use 2

[0119] CTUs with Cl complexity and 1 CTU with C2 complexity, resulting in:

[0120] (AO x CO) + (All % Cl) + (A2 x C2) (7 % 0) + (2 x 5) + (1 x 20)

[0121] = 3 kMAC / pix

[0122] N1 + N2 + N3 7 + 2 + 1 but not 2 CTUs with C2 complexity, resulting in: ( / VO x CO) + (ATI % Cl) + ( / V2 x C2) (8 x 0) + (0 x 5) + (2 x 20)

[0123] N1 + N2 + N3= Q8 + 'n0 + ' 2Q = 4 kMAC / Plx

[0124] In another version of this embodiment with two NN loop filters of complexity Cl and C2, Cl > C2, two ratios can be used: rl and r2. For example, if rl=20% and r2=30%, then at most 20% of the CTUs / blocks / samples can be processed using the loop filter of complexity Cl, and at most 30% of the CTUs / blocks / samples can be processed using the loop filter of complexity C2, which means that at least 50% of the CTUs / blocks / samples may have little to no processing. A special case of this may be when rl+r2=100%, which means that every block can be processed using either the loop filter of complexity Cl or using the loop filter of complexity C2. In one version of this embodiment, it is possible to process the block with the lower complexity, like C2, even if it was designated as allowed for the higher complexity. As an example, if rl=10% and r2=80%, it is possible to process 90% of the blocks with the loop filter of lower complexity C2 if no blocks are processed with the loop filter of higher complexity Cl. These embodiments can be extended mutatis mutandis to three or more filters with three or more ratios.

[0125] In another example of this version, the total cap for the number of pixels or blocks that can use NN loop filtering is specified for more than one picture. In one example, a cap N is specified for CTUs in pictures Pl, P2 and P3 together, which means Nl + N2 + N3 < N CTUs may use NN loop filtering where Nl, N2 and N3 are the number of CTUs in Pl, P2 and P3 respectively that use NN loop filtering. In one example, N may be smaller than the total number of CTUs in one picture. In another example, N may be larger than the total number of CTUs in one picture. In another example, there is an additional per picture cap of Np, for example, Nl + N2 + N3 < N is allowed as long as Nl, N2, N3 < Np.

[0126] In another version of this embodiment, instead of a maximum number, an exact number, or an exact ratio, of the pixels or blocks that may use NN loop filtering is specified. In one example, there may be 8 CTUs per picture in a video sequence, and it may be specified that exactly 4 CTUs in every picture use NN loop filtering. The cap value may be signaled in a parameter set such as SPS, PPS, VPS, and / or a header such as a picture header or slice header. The cap value may alternatively be signaled in a decoding capability information, DCI, NAL unit.

[0127] Figure 13 shows one example for signaling a number of NN blocks that may use NN loop filtering in the PPS. In this example, sps_nnlf_enabled_flag 1300 equal to 1 specifies that NN inloop filtering is enabled for the CLVS. sps_nnlf_enabled_flag 1300 equal to 0 specifies that NN in-loop filtering is disabled for the CLVS; and pps max nn block cnt 1302 specifies the number of NN blocks in the picture that may use NN loop filtering. In one example, the cap may be signaled for the sequence and may guarantee a worst case per picture NN loop filter complexity for all the pictures in the sequence.

[0128] An apparatus, e.g., a decoder, may perform all or a subset of the following steps for this embodiment to decode a picture: determine from one or more syntax elements in the bitstream a maximum allowed complexity; determine that decoding a current portion of the picture is not using NN loop filtering if the maximum allowed complexity is reached; determine that decoding a current portion of the picture may use NN loop filtering if maximum allowed complexity is not reached; and / or decode the picture not using NN loop filtering if the maximum allowed complexity is reached, or decode the picture using NN loop filtering if the maximum allowed complexity is not reached.

[0129] Example Embodiment 2

[0130] Other example embodiments can comprise spending the complexity budget of NN filtering for particular areas in the picture or for particular pictures in a set of pictures. In such embodiments some areas in a picture or some pictures in a set of pictures are given a higher priority for using the NN loop filtering budget. The prioritized areas could be the ones located in a particular position in the picture or have a particular content or been marked as a region of interest, ROI. In one example, blocks in the picture that are located closer to the center of the picture may get a higher priority for being considered for NN loop filtering. In another example, blocks of the picture that contain a human face may be prioritized. In yet another example, one or more specific portions of a picture such as a specific subpicture may be given higher priority than other subpictures.

[0131] In a version of this embodiment, a subset of the blocks in the picture that are marked for other purposes may be prioritized for using NN loop filtering. In one example, blocks of the picture that are specified by block importance mapping, BIM, tool may be prioritized for using NN loop filtering. In another example, blocks in the picture with the highest sum of BS values may be prioritized for using NN loop filtering. In this example, if maximum N blocks in the picture may use NN loop filtering, the N blocks with the highest sum of BS values may use NN loop filtering. If the sum of BS values is equal for several blocks in the picture, another rule, like for example the scanning order, may be used along with the sum of the BS values to determine which of the blocks will use NN loop filtering.

[0132] In another version, the per block signaling of NN loop filter on / off may be replaced by a set of one or more conditions, where the decoder may use the set of one or more conditions for determining which blocks use NN loop filtering (e.g. based on BS values, block type, QP, etc.) and optionally in combination with the block scanning order in the picture, where the first N blocks that satisfy the set of one or more conditions use NN loop filtering, where N is the cap for the number of blocks in the picture that may use NN loop filtering.

[0133] In another version of this embodiment, one or more pictures in a set of pictures may be prioritized for the NN loop filtering budget. In one version, prioritization may be based on the picture type of the picture, i.e. whether the picture is an I-, P- or B-picture. For instance, an I- picture may be prioritized over a P- or B-picture. In another version, prioritization may be based on the temporal ID of the picture, e.g. such that a picture with a lower temporal ID may be prioritized over a picture with a higher temporal ID. In some versions, pictures that are going to be used for reference may be prioritized over pictures that are not going to be used for reference. Pictures in VVC and HEVC may, for example, be marked as non-reference pictures. In some versions, pictures of different layers within an access unit, or the like, are given different priorities. For example, a picture in a layer with lower layer ID may be given higher priority than a picture in a layer with a higher layer ID. Prioritization may further be determined based on the type of content or the auxiliary type of a picture, e.g. a picture comprising a natural image may be prioritized over a depth map picture or an alpha map picture.

[0134] Example Embodiment 3

[0135] Other example embodiments can comprise no signaling for the rest of the blocks in the picture when the cap of the block count for using NN loop filtering is reached. In such embodiments, it may be signaled in the bitstream that the current block is using NN loop filtering or not until the number of the blocks in the picture that use NN loop filtering reaches a cap. After that, and for the rest of the blocks in the picture, there may be no signaling specifying whether the block is using NN loop filtering or not, but it may be inferred that the block is not using NN loop filtering. This may save bits by omitting the signaling when the cap is reached.

[0136] In the following example, a maximum number of CTUs that may use NN loop filtering may be signaled in the slice header and a per CTU flag may be signaled in the slice data to specify if the current CTU is using NN loop filtering or not until the number of CTUs in the slice that use NN loop filtering reaches the max value. Figure 14 shows a slice header 1400 of this example, and Figure 15 shows slice data 1500. For this example, sps_nnlf_enabled_flag 1402, as shown in Figure 14, equal to 1 may specify that NN in-loop filtering is enabled for the CLVS. sps_nnlf_enabled_flag 1402 equal to 0 may specify that NN in-loop filtering is disabled for the CLVS. In addition, max_nn_loop_filter_ctu_cnt 1502 may specify the number of NN blocks in the picture that may use NN loop filtering. When not present, the value of max_nn_loop_filter_ctu_cnt 1502 is inferred to be equal to 0. Also, ctu_nn_loop_filter_flag 1504 equal to 1 may specify that the CTU uses NN loop filtering. ctu_nn_loop_filter_flag 1504 equal to 0 may specify that the CTU does not use NN loop filtering. When not present, the value of ctu_nn_loop_filter_flag 1504 may be inferred to be equal to 0. Variable NumCtusInCurrSlice 1506 may specify the number of CTUs in the current slice.

[0137] An apparatus, e.g., a decoder, may perform all or a subset of the following steps for an embodiment to decode a picture. First, the apparatus may determine from one or more syntax elements in a bitstream a maximum allowed complexity for the picture. Then for every current portion of the picture, the decoder may a) check whether an existing complexity has reached the maximum allowed complexity; b) if the maximum allowed complexity for the picture is not reached, decode a value from the code word and if the value is equal to a first value use NN loop filtering for decoding the current portion of the picture and if the value is equal to a second value skip NN loop filtering for decoding the current portion of the picture; and / or c) if NN loop filtering is used for decoding the current portion of the picture, update the existing complexity.

[0138] In some versions, rather than performing step (b) as described above, if the maximum allowed complexity for the picture is not reached, the apparatus may check a set of one or more rules that specify whether the current portion uses NN loop filtering. Use NN loop filtering for decoding the current portion of the picture if the set of one or more rules specify that the current portion uses NN loop filtering and skip NN loop filtering for decoding the current portion of the picture if the set of one or more rules do not specify that the current portion uses NN loop filtering.

[0139] If there are two complexity ratios, rl and r2, the first block of the image may signal to use either NN loop filter 1 , NN loop filter 2, or no NN loop filtering. When one of the loop filters, such as filter 1 or filter 2, has used up its entire budget, the following blocks may signal to use either NN loop filter 2 or filter 1 or no loop filtering. This signaling may thus become less expensive since only signal one of two options may be needed instead of one of three options. Once filter 2 or filter 1 has also used up its budget, no signaling is needed, and the remaining CTUs may default to no NN loop filtering.

[0140] Example Embodiment 4

[0141] Other example embodiments can comprise inheriting a cap from another picture or portion of picture. In such embodiments, the complexity cap for the NN-filter may be inherited from a previous picture or portion of a picture preceding the current picture or current portion in decoding order. In one version, the previous picture may be a picture directly preceding the current picture in decoding order. In another version, the previous picture may be the closest previous picture within the same temporal layer as the current picture. In another version, the POC of the previous picture to inherit the cap from is indicated. In another version, the cap may be signaled in PPS, such that the pictures referencing the PPS may have the same complexity cap for the NN-filter. In another version, the control of the block-based or frame-based NN filter may be based on spatially neighboring blocks or collocated blocks in one or more reference frames, or other past filtered blocks. In another version, instead of just inheriting the complexity cap from a previous picture, a delta to a cap for the previous picture may be signaled to set the cap for the current picture. The delta value may be, for example, in absolute number of blocks that may use NN-filtering or be a difference in percentage units compared to the previous picture, for example, the cap may be lowered by 20% units for higher temporal layers.

[0142] The indication of inheriting the cap, or the delta signaling of the cap, may be signaled in a parameter set, on a picture level, on a slice level, and / or on a block level in the bitstream. Example syntax and semantics of signaling delta values for the cap for each temporal layer in SPS is given in Figure 16. In this example, sps max nn block cnt layerO 1600 specifies the maximum number of blocks that may use NN-filtering in a picture with Temporalld equal to 0. Also, sps max nn block cnt delta [ i ] 1602 specifies a delta for the maximum number of blocks that may use NN-filtering in a picture with Temporalld equal to i. The maximum number of blocks that may use NN-filtering in a picture having a Temporalld equal to i, maxNumNNBlockCnt [ i ] is derived as: if i is equal to 0, then maxNumNNBlockCnt [ i ] = sps_max_nn_block_cnt_layerO; otherwise, maxNumNNBlockCnt [ i ] = maxNumNNBlockCnt [ i - 1 ] + sps max nn block cnt delta [ i ].

[0143] Example Embodiment 5

[0144] Other example embodiments can be characterized in that an apparatus, e.g., an encoder, may select which blocks to use NN-filtering for based on the cap and performance gains. In such embodiment, the NN loop filtering is used for at most N, or N%, of the blocks in a picture for which the NN-filtering provides, or is estimated to provide, the highest performance gain. A block may here for instance mean a CU, a CTU, a tile, a subpicture. In some versions, the encoder may first encode all blocks using NN-filtering and then may encode all blocks again but this time without using NN-filtering. This may be done in a 2-pass encoding scheme. The performance gains of using NN-filtering may then be calculated for each block. The performance gains may, for example, be calculated as a difference in mean squared error, MSE, towards the input picture between a block that is not using NN-filtering and the corresponding block using NN-filtering. In some versions, the performance gains may be estimated. Based on the performance gains and the complexity cap of using maximum N blocks or N% of the blocks for NN-filtering, it may then be determined which blocks to use NN-filtering and which NN-blocks to not use NN-filtering. In some versions, the N blocks, or N% of blocks, with the highest performance gains may be selected to use NN-filtering. The bitstream may then be encoded accordingly. In some versions, other factors may come in play too, e.g. the position of the block, etc., which may weigh in on the decision of whether or not to use NN-filtering for a particular block.

[0145] The encoder may avoid doing exhaustive NN-filtering performance gain calculations for all the blocks in a picture by instead estimating the performance gains. In some versions, blocks may be prioritized based on the block types or amount of boundary partitioning, e.g. do NN- filtering if the sum of BS values for the block is more than a threshold. In some versions, the encoding and / or NN-filtering may be done on subsampled versions of the block, which may give a good estimate of how the NN-filtering would perform on a full-size block. An encoder may have a fast mode where the performance gains are estimated and a slower mode where the performance gains are calculated.

[0146] An apparatus, e.g., an encoder, may perform all or a subset of the following steps according to this embodiment: obtain a picture to be encoded; encode the picture using NN-filtering for all blocks; encode the picture without using NN-filtering for any block; calculate or estimate one or more performance gains of using NN-filtering for each block in the picture; obtain a complexity cap N for the picture, wherein the complexity cap N may, for example, be the number of the blocks, or a percentage of the blocks in the picture that may be encoded with NN-filtering; determine which blocks in the picture to encode using NN-filtering based on the obtained complexity cap N and the calculated or estimated one or more performance gains of using NN-filtering for each block; and / or encode the picture using NN-filtering for the determined blocks and no NN-filtering for all other blocks.

[0147] Example Embodiment 6

[0148] Other example embodiments can link complexity cap to tier / level / profile. In one version of such embodiments, the complexity cap may be used for specifying a particular tier or level. The tier and level may be signaled in the bitstream to indicate that the bitstream conforms to the tier and level. A decoder conforming to a specific tier and specific level may be able to decode all bitstream for which the bitstream is indicated to conform to a tier that is lower than or equal to the specified tier and is indicated to conform to a level that is lower or equal to the specified level.

[0149] In this version, a level and / or tier may specify for a picture or a set of pictures one or more of: a maximum allowed MAC / pixel for NN filtering; a maximum allowed samples that use NN filtering; a maximum allowed percentage, or ratio, of samples that use NN filtering; a maximum number of allowed NN blocks that use NN filtering; a maximum allowed percentage, or ratio, of NN blocks that use NN filtering; a maximum number of allowed CTUs that use NN filtering; and / or a maximum allowed percentage, or ratio, of CTUs that use NN filtering. The maximum allowed samples or the maximum number of allowed NN blocks that use NN filtering may be expressed as a number N for each level, where N may specify a maximum number of samples or NN blocks that may be output from NN filtering for one picture or a set of pictures. In one version, the value N may specify a maximum number of samples or NN blocks that may be output from NN filtering during a certain time. As an example, the certain time may be a time window of 1 second, such that there is no time window of the length of 1 second where more than N samples or NN blocks are output from NN filtering. The time may be any time, such as picture output time, decoding time, CPB removal time, etc. When the encoder produces a bitstream of a certain level, it may be the responsibility of the encoder to ensure that NN filtering is not done more than the limit N for that level. This may mean that when the decoder decodes a bitstream of a level, the decoder will know that the amount of NN filtering is capped such that the limit N for that level is not exceeded. An example of where the tier and level specify the maximum allowed ratio of luma samples in a picture that uses NN filtering is depicted in Figure 17 on top of the general tier and level limits of VVC.

[0150] In another version, the codec profile may determine the complexity cap. In one version, a codec may specify two or more profiles where a first profile does not allow NN filtering, and a second profile allows NN filtering. The first profile may be a main profile, and the second profile may be an extended profile. In one version, a codec may specify two or more profiles where at least one profile may allow NN filtering but does not allow NN filtering to be used for a whole picture. In one version, a codec may specify two or more profiles where at least one profile may allow NN filtering but does not allow NN filtering to be used for all pictures in a set of pictures. In one version, a codec may specify two or more profiles where a first profile may allow NN filtering of a first type to be used but does not allow NN filtering of a second type to be used and a second profile may allow NN filtering of both the first and second types to be used. Example Embodiment 7

[0151] In one embodiment, the picture order count, POC, can be used to determine whether a picture can be filtered or not. A common way to transmit pictures in a video sequence is to use a hierarchical structure 1800 such as the one shown in Figure 18. The figure shows 33 pictures 1802 that are being received by the decoder, each picture 1802 represented by a parallelogram. Inside each parallelogram, the picture order count 1804 is marked. After decoding, the pictures 1802 are displayed in increasing POC order. Hence, in Figure 18, picture 0 1806 is first displayed, then picture 1 1808, then picture 2 1810, etc. until picture 32 1812. The GOP structure 1800 also has six hierarchical layers 1814, denoted HID 0 through HID 5. A picture in a hierarchical layer can only predict from (e.g., copy samples from) a picture in a hierarchical layer equal to or lower than its own hierarchical layer. For example, the picture with POC 16 1816 is situated in HID 1, which means that it can only predict from the pictures in HID 0 or 1, namely the pictures with POC 0 1806 and POC 32 1812. The picture with POC 8 1818 belongs to HID 2 and can therefore predict from all pictures in HID 0-2, such as POC 0 1806, POC 16 1816 and POC 32 1812. Pictures of lower hierarchical layers can thus be made available for prediction to more pictures than pictures of higher hierarchical layers. It is therefore often economical to spend more bits on pictures belonging to a lower hierarchical layer such as HID 0 or HID 1, rather than spend bits on the top hierarchical layer. This may be because the increased quality from the extra bits will improve not only the picture itself, but the many pictures that are predicted from them. The same goes for complexity spent on filtering a picture with a neural network. If there are enough computational resources to filter all pictures in the video sequence, it may be best to filter all pictures. However, if resources in a decoder are to be saved, it may be better to spend the computational resources on filtering pictures belonging to a lower hierarchical layer, such as HID 0 and HID 1, rather than spending them on pictures in HID 5. This may be because filtering may produce pictures that are better for prediction, since the filtered pictures may be closer to the original. Further, spending computational resources on pictures in HID 0 and HID 1, which will be used more for prediction, may be often advantageous compared to spending them on pictures in HID 5, or even on pictures in HID 4.

[0152] Therefore, in some versions, the decoder can infer from the POC whether or not any blocks in the picture are allowed to undergo filtering. As an example, it is possible to have the following rule: if (POC mod 2 == 0),

[0153] # Blocks in this picture are allowed to be filtered using NN loop filter, else

[0154] # No blocks in this picture are allowed to be filtered using NN loop filter. Here mod is the modulos operator, meaning that for the integers A and B, the expression A mod B gives the remainder of the integer division A divided by B, so that A = (A / B) * B + (A mod B) holds. For pictures with an odd POC, the expression POC mod 2 will always equal 1, and therefore no blocks may be allowed to be filtered by the decoder. As seen in Figure 18, all pictures belonging to the top most hierarchical layer 5, HID5, have odd POC numbers, and hence the if- statement would prevent the filtering of pictures in HID5, if the GOP structure is hierarchical in a manner such as in Figure 18. Also, no other hierarchical layer has odd-numbered POC pictures, so filtering would still be allowed for HID 0-4. This may have four advantages. One advantage may be that the decoder may save computational resources since pictures with an odd POC may not need neural network processing. Another advantage may be that since neural network filtering is not done for half of the pictures, not only the decoder but also the encoder may save computational resources. Another advantage may be that since the decoder knows that it does not need to filter every picture, but only every second picture, it may easily amortize the filtering cost over time. For example, filtering half of the picture with POC 0 in one time slot, and half of that picture in the next time slot, each time slot being the time difference between the display of two decoded pictures. Put another way, a decoder designed to be capable of a certain resolution and frame rate, such as 3840x2160@60 fps, can rely on the fact that, since only half of the frames need to be filtered, the hardware can be dimensioned to filter only 3840x2160@30 fps, which may save resources such as silicon surface area. Another advantage may be that for pictures with an odd POC, no signaling may be needed to say whether any blocks in this picture should be filtered, since this may be derived implicitly from the POC. For pictures with an even POC, signaling may be needed to say whether any blocks in the picture are filtered or not, since it is sometimes advantageous to not filter certain blocks / pictures. By filtering only half of the pictures, image quality may go down compared to filtering all pictures. However, since the filtering is denied only to the pictures of the highest hierarchical layer(s), this quality degradation may be limited. Avoiding the filtering of odd-numbered pictures may save about 50% of the computational burden compared to filtering all pictures. If it is desirable to save more computational resources, it is possible to instead use: if (POC mod 4 == 0)

[0155] # Blocks in this picture are allowed to be filtered using NN loop filter, else

[0156] # No blocks in this picture are allowed to be filtered using NN loop filter.

[0157] This may prohibit the neural network filtering of pictures in both HID 4 and HID 5. Similarly, using: if (POC mod 32 == 0)

[0158] # Blocks in this picture are allowed to be filtered using NN loop filter, else

[0159] # No blocks in this picture are allowed to be filtered using NN loop filter, may prohibit filtering of all pictures that do not belong to the lowest hierarchical layer, HID 0.

[0160] In another version, the decoder may not be prohibited from filtering a picture no matter what POC the picture has. However, it may still be desirable to ease the burden of the decoder for some video sequences. In such cases, the encoder can choose to restrict itself from using neural network loop filtering of a picture, if the POC of the picture is odd, using: if (POC mod 2 == 0)

[0161] # Encode this image with NN loop filter allowed, else

[0162] # Encode this image with NN loop filter turned off.

[0163] In this version, the decoder and encoder respectively both may save computational effort. When implementing a hardware decoder that should be able to run at a certain resolution and frame rate, it may not be possible for the hardware design to rely on filtering being turned off for every second picture, since another encoder may decide to filter every image in the sequence. The encoder may signal that the loopfilter is turned off for frames with an odd-numbered POC. In some cases, the solution where the encoder decides can be advantageous. As an example, in a point-to- point video conferencing scenario, the encoder may get an out-of-band distress signal from the decoder that it has a hard time processing so much neural network filtering, in which case the encoder can use this approach to ease the burden for the decoder.

[0164] Figure 19 shows an example of how these computational resource savings can be in practice. The benchmark is a decoder that does not do any neural network filtering at all 1900. The x-axis is decoding time 1902, and the decoder that does not do any neural network filtering at all 1900 thus sits at 100% in the x-direction. The y-axis is bit rate change at constant quality compared to the benchmark, called Bjontegaard-delta rate 1904, BDR, and a negative BDR is desired. The decoder that does not do any neural network filtering at all 1900 therefore sits at 0% BDR change. If 100% of the frames are allowed to be filtered, then the decoder takes 2220%, so 22.2 times slower than the benchmark, shown by a filled circle 1906. However, neural network filtering may also mean that the bit rate is almost 3% smaller, BDR = -2.85%, a substantial change. The asterisk 1908 illustrates what may happen if one uses the test (POC mod 2 == 0) in the if-statement, i.e., avoid filtering any pictures with odd POCs. The decoder time goes down from 2220% to 1133%, but the BDR only goes down to 2.56%. Hence around 90% of the BDR change may be preserved, while avoiding around half of the computational burden in the decoder and encoder. The square 1910 shows what may happen if (POC mod 4 == 0) is used, which may be equivalent of only filtering pictures with a POC divisible by 4. The BDR may go down to -2.19%, or about 75% of the maximum, but the decoder run time may go down to 661%, or about 30% of the maximum. Finally, the triangle 1912 shows the result for (POC mod 8 == 0), which has a BDR of -1.78%, or 60% of the maximum, but the decoder run time is now 387%, or 17% of the maximum. Thus with 17% of the computational resources, 60% of the benefit may be obtained.

[0165] POC does not necessarily need to increase by one for each picture in a video bitstream, it may be allowed to increase by any number, e.g. by 5 or 24. In order to compensate for that, a factor j may be introduced for how much the POC increases. In general, the rule for hierarchical GOP structures can be: if ((POC / j) mod M == k)

[0166] # Blocks in this picture are allowed to be filtered using NN loop filter, else # No blocks in this picture are allowed to be filtered using NN loop filter.

[0167] So far, M has had powers of two, i.e., M = 2N where N is an integer. It is also possible to use an M that is not a power of two, for instance if the GOP structure deviates from that in Figure 18. So far, k = 0 was also used, i.e., allowing filtering picture where POC is divisible by M has been prioritized, which in Figure 18 means pictures under a certain HID hierarchical layer. However, in other circumstances, a k not equal to 0 may instead be used, for instance if the GOP structure deviates from that in Figure 18.

[0168] In one version of the embodiment, the hierarchical layers 1814 described above may be temporal sublayers and the HIDs may be TemporallDs. In another version of the embodiment, the hierarchical layers 1814 described above may be layers, such as scalable or multiview layers, and the HIDs may be layer IDs. In another version, the hierarchical layers 1814 described above may not necessarily be temporal sublayers or layers. For example, multiple hierarchical layers with HIDs 0-5 could be within one temporal sublayer with TemporallD 0, or hierarchical layers with HIDs 4-5 could be within one temporal sublayer with TemporallD 3.

[0169] In some versions, only pictures belonging to a specific hierarchical layer, e.g., HID 0, may be allowed to be filtered using NN loop filter. In some versions, only pictures with a HID below a certain value, e.g. HIDs < 4, i.e. HIDs 0 - 3, may be allowed to be filtered using NN loop filter.

[0170] In some versions, one or more syntax elements in the bitstream may be used to specify which HIDs that may be allowed to be filtered using NN loop filter. These one or more syntax elements may be signaled in a parameter set, such as a VPS, an SPS, a PPS or an APS, in a DCI, or in a header such as a picture header or a slice header.

[0171] In some versions, different NN filters of different complexity may be used, wherein pictures with a certain set of HIDs may be allowed to be filtered using a first NN loop filter of a first complexity, and a second set of HIDs may be allowed to be filtered using a second NN loop filter of a second complexity that may be different from the first complexity. For example, pictures with HID 0 may be allowed to be filtered using a first NN loop filter of a first complexity, while pictures with HIDs 1-3 may only be allowed to be filtered using a second NN loop filter of a second complexity that is lower than the first complexity, and pictures with HIDs 4-5 may not be allowed to be filtered using an NN loop filter at all.

[0172] In one embodiment, such determination of filter may be done using the POC of a picture according to: if (POC mod 2 == 0) if (POC mod 4 == 0)

[0173] # Encode this image with NN loop filter of first complexity allowed, else

[0174] # Encode this image with NN loop filter of second complexity allowed, else

[0175] # Encode this image without any NN loop filtering.

[0176] Another way to state the same is to use the following logic: if (POC mod 2 == 0) and (POC mod 4 == 0)

[0177] # Encode this image with NN loop filter of first complexity allowed, if (POC mod 2 == 0) and (POC mod 4 ! = 0)

[0178] # Encode this image with NN loop filter of second complexity allowed, if (POC mod 2 ! = 0) and (POC mod 4 != 0)

[0179] # Encode this image without any NN loop filtering.

[0180] Where == denotes ‘is equal to’ and != denotes ‘is not equal to’.

[0181] If the GOP structure is according to Figure 18, the if-statements above may allow NN filtering of a first complexity for HID 0-3, NN-filtering of a second complexity for HID 4 and no filtering for HID 5. In another version of this embodiment, only pictures that are or may be used for reference by other pictures may be allowed to be filtered using NN loop filter. Pictures of the highest hierarchical layer may typically not used for reference. In some versions, the non-reference pictures may belong to the highest hierarchical layer. VVC may have a flag in the picture header, ph_non_ref_pic_flag, to indicate that a picture is not used for reference. In some versions, a non- ref flag may be used to indicate whether a picture may be allowed to be or will be filtered using an NN loop filter or not. As an example, NN loop-filtering may be disabled when a picture is specified to be a non-reference picture. In this example, when a picture is not specified to be a non-reference picture, NN loop-filtering may be done.

[0182] In some versions, there may be a picture enable syntax element or a slice enable syntax element that can specify that NN in-loop filtering is not done for the picture or slice. The syntax element may be a flag. In this version, the encoder can use the syntax element to disable NN inloop filtering for a picture or slice which means that NN-inloop filtering may not be done by the decoder when decoding the picture or slice. An encoder may use any POC-based method previously described to set the value of the syntax element to reduce the overall NN complexity. For example, the encoder may set the syntax element to disable NN inloop filtering for any of the following cases: disable filtering for pictures with an odd POC value; disable filtering for any picture with a POC value P not evenly divisible by 4, i.e. do not filter if ((P mod 4) != 0); disable filtering for pictures belonging to one or more specific layer or temporal layer; disable filtering for picture belonging to the N highest hierarchical levels in a hierarchical bi-directional structure of picture referencing; and / or disable filtering for non-reference pictures. In this version, the value of the enable syntax elements for a set of pictures may be restricted such that it may not be allowed for an encoder to enable NN inloop filtering for all pictures in the set of pictures.

[0183] In some versions, an apparatus for decoding a picture comprising a neural network filtering process from a coded video bitstream, comprising: processing circuitry; and a memory storing instructions whereby the processing circuitry may be operable to perform, as shown in Figure 20, the steps 2000 of: determining a maximum allowed complexity from one or more syntax elements in the coded video bitstream 2002, wherein the maximum allowed complexity corresponds to a neural network filtering process; and decoding the one or more images based on the maximum allowed complexity 2004.

[0184] In some versions, an apparatus for decoding a picture comprising a neural network filtering process from a coded video bitstream, comprising: processing circuitry; and a memory storing instructions whereby the processing circuitry may be operable to perform, as shown in Figure 21, the steps 2100 of: obtaining a maximum allowed complexity for a neural network filter 2102; determining, based on the maximum allowed complexity, one or more portions of each of the one or more images to encode by the neural network filter 2104; filtering the one or more portions by the neural network filter, forming one or more filtered portions 2106; and encoding each image of the one or more images according to the one or more filtered portions and one or more remaining portions of the image 2108.

[0185] Additional Embodiments

[0186] One example of a method embodiment under the present disclosure is shown in Figure 20. Method 2000 comprises a method performed by a decoder for decoding one or more pictures from a coded video bitstream. Step 2002 is determining a maximum allowed complexity from one or more syntax elements in the coded video bitstream, wherein the maximum allowed complexity corresponds to a neural network filtering process. Step 2004 is decoding the one or more images based on the maximum allowed complexity. Method 2000 can comprise a variety of additional, alternative, or optional steps, or other variations. For example, in some embodiments the maximum allowed complexity is determined for at least one of the following: one or more pictures, one or more portions of a picture, one or more sets of pictures, or one or more coded video sequences, CVS. In another example, the maximum allowed complexity is defined as one or more of: the maximum allowed MAC for NN filtering; the maximum allowed samples that use NN filtering; the maximum allowed percentage of samples that use NN filtering; the maximum allowed NN blocks or percentage of NN blocks that use NN filtering; the maximum allowed CTUs that use NN filtering; or a percentage of CTUs that use NN filtering. Another example is the method of any of the previous examples, wherein the maximum allowed complexity is a constraint on the coded video bitstream. In another example, decoding (2004) the one or more pictures based on the maximum allowed complexity, comprises selectively filtering the one or more pictures using a neural network filtering process. In another example, decoding the one or more pictures based on the maximum allowed complexity comprises determining whether a current portion of a picture of the one or more pictures is filtered using a neural network filtering process in response to a comparison of a complexity value and the maximum allowed complexity. This embodiment may be configured wherein determining whether a current portion of the picture is filtered using a neural network filtering process in response to a comparison of a complexity value and the maximum allowed complexity comprises decoding a first syntax element from the coded video bitstream. In some versions, determining whether a current portion of the picture is filtered using a neural network filtering process is based on at least one of the scanning order of the portions in the picture; the current portion of the picture consists of a block which is marked by a block importance mapping, BIM, tool; the boundary strength, BS, information of at least one block in the current portion of the picture; a block type of at least one block in the current portion of the picture; a quantization parameter, QP, value of at least one block in the current portion of the picture; a picture type of the picture containing the current portion of the picture; a temporal ID of the picture containing the current portion of the picture; and / or a layer ID of the picture containing the current portion of the picture. In some variations, the maximum allowed complexity may be decoded from a parameter set comprising at least one of: a PPS; a SPS; a VPS; a header; a picture header; a slice header; a decoding capability information, DCI, NAL unit. In some embodiments, selectively filtering the one or more pictures is based on a picture order count, POC. In some embodiments, selectively filtering the one or more pictures is based on a hierarchical layer of the one or more pictures. In some embodiments, the hierarchical layer comprises at least one of: a temporal sublayer; a layer; a scalable layer; or a multiview layer.

[0187] Another example of a method embodiment under the present disclosure is shown in Figure 21. Method 2100 comprises a method performed by an encoder for encoding one or more pictures to a coded video bitstream. Step 2102 is obtaining a maximum allowed complexity for a neural network filter. Step 2104 is determining, based on the maximum allowed complexity, one or more portions of each of the one or more pictures to encode using the neural network filter. Step 2106 is filtering the one or more portions using the neural network filter, forming one or more filtered portions. Step 2108 is encoding each picture of the one or more pictures according to the one or more filtered portions and one or more remaining portions of the picture. Method 2100 can comprise a variety of additional, alternative, or optional steps, or other variations. For example, some embodiments may further comprise encoding an indication of the maximum allowed complexity for the neural network filter to the coded video bitstream. In some embodiments, determining one or more portions of each picture of the one or more pictures to encode using the NN filter comprises calculating one or more performance gains of filtering using the neural network filter for at least one portion of each picture. In some embodiments, the maximum allowed complexity is a constraint on the coded video bitstream. In some embodiments, determining (2104) one or more portions of each picture of the one or more pictures to encode using the NN filter is based on at least one of the scanning order of the portions in the picture; the current portion of the picture consists of a block which is marked by a block importance mapping, BIM, tool; the boundary strength, BS, information of at least one block in the current portion of the picture; a block type of at least one block in the current portion of the picture; a quantization parameter, QP, value of at least one block in the current portion of the picture; a picture type of the picture containing the current portion of the picture; a temporal ID of the picture containing the current portion of the picture; and / or a layer ID of the picture containing the current portion of the picture. In another example embodiment, the maximum allowed complexity is determined for at least one of the following: a picture, a portion of a picture, a set of pictures, or a coded video sequence, CVS. In another example, the maximum allowed complexity is defined as one or more of: the maximum allowed MAC / pixel for NN filtering; the maximum allowed samples that use NN filtering; the maximum allowed percentage of samples that use NN filtering; the maximum allowed NN blocks or percentage of NN blocks that use NN filtering; the maximum allowed CTUs or percentage of CTUs that use NN filtering. In some embodiments, determining one or more portions of each picture of the one or more pictures to encode using the NN filter is further based on a picture order count, POC. In some embodiments, determining (2104) one or more portions of each picture of the one or more pictures to encode is based on a hierarchical layer of the one or more pictures. In some embodiments of the method, the hierarchical layer comprising at least one of: a temporal sublayer; a layer; a scalable layer; or a multiview layer.

[0188] Figure 22 shows an example of a communication system 22100 in accordance with some embodiments. In the example, the communication system 22100 includes a telecommunication network 22102 that includes an access network 22104, such as a radio access network, RAN, and a core network 22106, which includes one or more core network nodes 22108. The access network 22104 includes one or more access network nodes, such as network nodes 22110a and 22110b (one or more of which may be generally referred to as network nodes 22110), or any other similar 3rd Generation Partnership Project, 3GPP, access nodes or non-3GPP access points. Moreover, as will be appreciated by those of skill in the art, a network node is not necessarily limited to an implementation in which a radio portion and a baseband portion are supplied and integrated by a single vendor. Thus, it will be understood that network nodes include disaggregated implementations or portions thereof. For example, in some embodiments, the telecommunication network 22102 includes one or more Open-RAN, ORAN, network nodes. An ORAN network node is anode in the telecommunication network 22102 that supports an ORAN specification (e.g., a specification published by the O-RAN Alliance, or any similar organization) and may operate alone or together with other nodes to implement one or more functionalities of any node in the telecommunication network 22102, including one or more network nodes 22110 and / or core network nodes 22108.

[0189] Examples of an ORAN network node include an open radio unit (O-RU), an open distributed unit (O-DU), an open central unit (O-CU), including an O-CU control plane (O-CU- CP) or an O-CU user plane (O-CU-UP), a RAN intelligent controller (near-real time or non-real time) hosting software or software plug-ins, such as a near-real time control application (e.g., xApp) or anon-real time control application (e.g., rApp), or any combination thereof (the adjective “open” designating support of an ORAN specification). The network node may support a specification by, for example, supporting an interface defined by the ORAN specification, such as an Al, Fl, Wl, El, E2, X2, Xn interface, an open fronthaul user plane interface, or an open fronthaul management plane interface. Moreover, an ORAN access node may be a logical node in a physical node. Furthermore, an ORAN network node may be implemented in a virtualization environment (described further below) in which one or more network functions are virtualized. For example, the virtualization environment may include an O-Cloud computing platform orchestrated by a Service Management and Orchestration Framework via an 0-2 interface defined by the O-RAN Alliance or comparable technologies. The network nodes 22110 facilitate direct or indirect connection of user equipment (UE), such as by connecting UEs 22112a, 22112b, 22112c, and 22112d (one or more of which may be generally referred to as UEs 22112) to the core network 22106 over one or more wireless connections.

[0190] Example wireless communications over a wireless connection include transmitting and / or receiving wireless signals using electromagnetic waves, radio waves, infrared waves, and / or other types of signals suitable for conveying information without the use of wires, cables, or other material conductors. Moreover, in different embodiments, the communication system 22100 may include any number of wired or wireless networks, network nodes, UEs, and / or any other components or systems that may facilitate or participate in the communication of data and / or signals whether via wired or wireless connections. The communication system 22100 may include and / or interface with any type of communication, telecommunication, data, cellular, radio network, and / or other similar type of system.

[0191] The UEs 22112 may be any of a wide variety of communication devices, including wireless devices arranged, configured, and / or operable to communicate wirelessly with the network nodes 22110 and other communication devices. Similarly, the network nodes 22110 are arranged, capable, configured, and / or operable to communicate directly or indirectly with the UEs 22112 and / or with other network nodes or equipment in the telecommunication network 22102 to enable and / or provide network access, such as wireless network access, and / or to perform other functions, such as administration in the telecommunication network 22102.

[0192] In the depicted example, the core network 22106 connects the network nodes 22110 to one or more host computing systems, such as host 22116. These connections may be direct or indirect via one or more intermediary networks or devices. In other examples, network nodes may be directly coupled to hosts. The core network 22106 includes one more core network nodes (e.g., core network node 22108) that are structured with hardware and software components. Features of these components may be substantially similar to those described with respect to the UEs, network nodes, and / or hosts, such that the descriptions thereof are generally applicable to the corresponding components of the core network node 22108. Example core network nodes include functions of one or more of a Mobile Switching Center (MSC), Mobility Management Entity (MME), Home Subscriber Server (HSS), Access and Mobility Management Function (AMF), Session Management Function (SMF), Authentication Server Function (AUSF), Subscription Identifier De-concealing function (SIDF), Unified Data Management (UDM), Security Edge Protection Proxy (SEPP), Network Exposure Function (NEF), and / or a User Plane Function (UPF).

[0193] The host 22116 may be under the ownership or control of a service provider other than an operator or provider of the access network 22104 and / or the telecommunication network 22102. The host 22116 may host a variety of applications to provide one or more service. Examples of such applications include live and pre-recorded audio / video content, data collection services such as retrieving and compiling data on various ambient conditions detected by a plurality of UEs, analytics functionality, social media, functions for controlling or otherwise interacting with remote devices, functions for an alarm and surveillance center, or any other such function performed by a server.

[0194] As a whole, the communication system 22100 of Figure 22 enables connectivity between the UEs, network nodes, and hosts. In that sense, the communication system may be configured to operate according to predefined rules or procedures, such as specific standards that include, but are not limited to: Global System for Mobile Communications (GSM); Universal Mobile Telecommunications System (UMTS); Long Term Evolution (LTE), and / or other suitable 2G, 3G, 4G, 5G standards, or any applicable future generation standard (e.g., 6G); wireless local area network (WLAN) standards, such as the Institute of Electrical and Electronics Engineers (IEEE) 802.11 standards (WiFi); and / or any other appropriate wireless communication standard, such as the Worldwide Interoperability for Microwave Access (WiMax), Bluetooth, Z-Wave, Near Field Communication (NFC) ZigBee, LiFi, and / or any low-power wide-area network (LPWAN) standards such as LoRa and Sigfox.

[0195] In some examples, the telecommunication network 22102 is a cellular network that implements 3GPP standardized features. Accordingly, the telecommunications network 22102 may support network slicing to provide different logical networks to different devices that are connected to the telecommunication network 22102. For example, the telecommunications network 22102 may provide Ultra Reliable Low Latency Communication (URLLC) services to some UEs, while providing Enhanced Mobile Broadband (eMBB) services to other UEs, and / or Massive Machine Type Communication (mMTC)ZMassive loT services to yet further UEs.

[0196] In some examples, the UEs 22112 are configured to transmit and / or receive information without direct human interaction. For instance, a UE may be designed to transmit information to the access network 22104 on a predetermined schedule, when triggered by an internal or external event, or in response to requests from the access network 22104. Additionally, a UE may be configured for operating in single- or multi-RAT or multi-standard mode. For example, a UE may operate with any one or combination of Wi-Fi, NR (New Radio) and LTE, i.e. being configured for multi-radio dual connectivity (MR-DC), such as E-UTRAN (Evolved-UMTS Terrestrial Radio Access Network) New Radio - Dual Connectivity (EN-DC).

[0197] In the example, the hub 22114 communicates with the access network 22104 to facilitate indirect communication between one or more UEs (e.g., UE 22112c and / or 22112d) and network nodes (e.g., network node 22110b). In some examples, the hub 22114 may be a controller, router, content source and analytics, or any of the other communication devices described herein regarding UEs. For example, the hub 22114 may be a broadband router enabling access to the core network 22106 for the UEs. As another example, the hub 22114 may be a controller that sends commands or instructions to one or more actuators in the UEs. Commands or instructions may be received from the UEs, network nodes 22110, or by executable code, script, process, or other instructions in the hub 22114. As another example, the hub 22114 may be a data collector that acts as temporary storage for UE data and, in some embodiments, may perform analysis or other processing of the data. As another example, the hub 22114 may be a content source. For example, for a UE that is a VR device, display, loudspeaker, or other media delivery device, the hub 22114 may retrieve VR assets, video, audio, or other media or data related to sensory information via a network node, which the hub 22114 then provides to the UE either directly, after performing local processing, and / or after adding additional local content. In still another example, the hub 22114 acts as a proxy server or orchestrator for the UEs, in particular if one or more of the UEs are low energy loT devices.

[0198] The hub 22114 may have a constant / persistent or intermittent connection to the network node 22110b. The hub 22114 may also allow for a different communication scheme and / or schedule between the hub 22114 and UEs (e.g., UE 22112c and / or 22112d), and between the hub 22114 and the core network 22106. In other examples, the hub 22114 is connected to the core network 22106 and / or one or more UEs via a wired connection. Moreover, the hub 22114 may be configured to connect to an M2M service provider over the access network 22104 and / or to another UE over a direct connection. In some scenarios, UEs may establish a wireless connection with the network nodes 22110 while still connected via the hub 22114 via a wired or wireless connection. In some embodiments, the hub 22114 may be a dedicated hub - that is, a hub whose primary function is to route communications to / from the UEs from / to the network node 22110b. In other embodiments, the hub 22114 may be a non-dedicated hub - that is, a device which is capable of operating to route communications between the UEs and network node 22110b, but which is additionally capable of operating as a communication start and / or end point for certain data channels.

[0199] Figure 23 shows a UE 23200 in accordance with some embodiments. The UE 23200 presents additional details of some embodiments of the UE 23112 of Figure 1. As used herein, a UE refers to a device capable, configured, arranged and / or operable to communicate wirelessly with network nodes and / or other UEs. Examples of a UE include, but are not limited to, a smart phone, mobile phone, cell phone, voice over IP (VoIP) phone, wireless local loop phone, desktop computer, personal digital assistant (PDA), wireless cameras, gaming console or device, music storage / playback device, wearable terminal device, wireless endpoint, mobile station, tablet, laptop, laptop-embedded equipment (LEE), laptop-mounted equipment (LME), an Augmented Reality (AR) or Virtual Reality (VR) device, wireless customer-premise equipment (CPE), vehicle, vehicle-mounted or vehicle embedded / integrated wireless device, etc. Other examples include any UE identified by the 3rd Generation Partnership Project (3GPP), including a narrow band internet of things (NB-IoT) UE, a machine type communication (MTC) UE, and / or an enhanced MTC (eMTC) UE.

[0200] A UE may support device-to-device (D2D) communication, for example by implementing a 3GPP standard for sidelink communication, Dedicated Short-Range Communication (DSRC), vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), or vehicle-to-everything (V2X). In other examples, a UE may not necessarily have a user in the sense of a human user who owns and / or operates the relevant device. Instead, a UE may represent a device that is intended for sale to, or operation by, a human user but which may not, or which may not initially, be associated with a specific human user (e.g., a smart sprinkler controller). Alternatively, a UE may represent a device that is not intended for sale to, or operation by, an end user but which may be associated with or operated for the benefit of a user (e.g., a smart power meter).

[0201] The UE 23200 includes processing circuitry 23202 that is operatively coupled via a bus 23204 to an input / output interface 23206, a power source 23208, a memory 23210, a communication interface 23212, and / or any other component, or any combination thereof. Certain UEs may utilize all or a subset of the components shown in Figure 23. The level of integration between the components may vary from one UE to another UE. Further, certain UEs may contain multiple instances of a component, such as multiple processors, memories, transceivers, transmitters, receivers, etc.

[0202] The processing circuitry 23202 is configured to process instructions and data and may be configured to implement any sequential state machine operative to execute instructions stored as machine-readable computer programs in the memory 23210. The processing circuitry 23202 may be implemented as one or more hardware-implemented state machines (e.g., in discrete logic, field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.); programmable logic together with appropriate firmware; one or more stored computer programs, general-purpose processors, such as a microprocessor or digital signal processor (DSP), together with appropriate software; or any combination of the above. For example, the processing circuitry 23202 may include multiple central processing units (CPUs).

[0203] In the example, the input / output interface 23206 may be configured to provide an interface or interfaces to an input device, output device, or one or more input and / or output devices. Examples of an output device include a speaker, a sound card, a video card, a display, a monitor, a printer, an actuator, an emitter, a smartcard, another output device, or any combination thereof. An input device may allow a user to capture information into the UE 23200. Examples of an input device include a touch-sensitive or presence-sensitive display, a camera (e.g., a digital camera, a digital video camera, a web camera, etc.), a microphone, a sensor, a mouse, a trackball, a directional pad, a trackpad, a scroll wheel, a smartcard, and the like. The presence-sensitive display may include a capacitive or resistive touch sensor to sense input from a user. A sensor may be, for instance, an accelerometer, a gyroscope, a tilt sensor, a force sensor, a magnetometer, an optical sensor, a proximity sensor, a biometric sensor, etc., or any combination thereof. An output device may use the same type of interface port as an input device. For example, a Universal Serial Bus (USB) port may be used to provide an input device and an output device.

[0204] In some embodiments, the power source 23208 is structured as a battery or battery pack. Other types of power sources, such as an external power source (e.g., an electricity outlet), photovoltaic device, or power cell, may be used. The power source 23208 may further include power circuitry for delivering power from the power source 23208 itself, and / or an external power source, to the various parts of the UE 23200 via input circuitry or an interface such as an electrical power cable. Delivering power may be, for example, for charging of the power source 23208. Power circuitry may perform any formatting, converting, or other modification to the power from the power source 23208 to make the power suitable for the respective components of the UE 23200 to which power is supplied.

[0205] The memory 23210 may be or be configured to include memory such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic disks, optical disks, hard disks, removable cartridges, flash drives, and so forth. In one example, the memory 23210 includes one or more application programs 23214, such as an operating system, web browser application, a widget, gadget engine, or other application, and corresponding data 23216. The memory 23210 may store, for use by the UE 23200, any of a variety of various operating systems or combinations of operating systems.

[0206] The memory 23210 may be configured to include a number of physical drive units, such as redundant array of independent disks (RAID), flash memory, USB flash drive, external hard disk drive, thumb drive, pen drive, key drive, high-density digital versatile disc (HD-DVD) optical disc drive, internal hard disk drive, Blu-Ray optical disc drive, holographic digital data storage (HDDS) optical disc drive, external mini-dual in-line memory module (DIMM), synchronous dynamic random access memory (SDRAM), external micro-DIMM SDRAM, smartcard memory such as tamper resistant module in the form of a universal integrated circuit card (UICC) including one or more subscriber identity modules (SIMs), such as a USIM and / or ISIM, other memory, or any combination thereof. The UICC may for example be an embedded UICC (eUICC), integrated UICC (iUICC) or a removable UICC commonly known as ‘SIM card.’ The memory 23210 may allow the UE 23200 to access instructions, application programs and the like, stored on transitory or non-transitory memory media, to off-load data, or to upload data. An article of manufacture, such as one utilizing a communication system may be tangibly embodied as or in the memory 23210, which may be or comprise a device-readable storage medium.

[0207] The processing circuitry 23202 may be configured to communicate with an access network or other network using the communication interface 23212. The communication interface 23212 may comprise one or more communication subsystems and may include or be communicatively coupled to an antenna 23222. The communication interface 23212 may include one or more transceivers used to communicate, such as by communicating with one or more remote transceivers of another device capable of wireless communication (e.g., another UE or a network node in an access network). Each transceiver may include a transmitter 23218 and / or a receiver 23220 appropriate to provide network communications (e.g., optical, electrical, frequency allocations, and so forth). Moreover, the transmitter 23218 and receiver 23220 may be coupled to one or more antennas (e.g., antenna 23222) and may share circuit components, software or firmware, or alternatively be implemented separately.

[0208] In the illustrated embodiment, communication functions of the communication interface 23212 may include cellular communication, Wi-Fi communication, LPWAN communication, data communication, voice communication, multimedia communication, short-range communications such as Bluetooth, near-field communication, location-based communication such as the use of the global positioning system (GPS) to determine a location, another like communication function, or any combination thereof. Communications may be implemented in according to one or more communication protocols and / or standards, such as IEEE 802.11, Code Division Multiplexing Access (CDMA), Wideband Code Division Multiple Access (WCDMA), GSM, LTE, New Radio (NR), UMTS, WiMax, Ethernet, transmission control protocol / intemet protocol (TCP / IP), synchronous optical networking (SONET), Asynchronous Transfer Mode (ATM), QUIC, Hypertext Transfer Protocol (HTTP), and so forth.

[0209] Regardless of the type of sensor, a UE may provide an output of data captured by its sensors, through its communication interface 23212, via a wireless connection to a network node. Data captured by sensors of a UE can be communicated through a wireless connection to a network node via another UE. The output may be periodic (e.g., once every 15 minutes if it reports the sensed temperature), random (e.g., to even out the load from reporting from several sensors), in response to a triggering event (e.g., when moisture is detected an alert is sent), in response to a request (e.g., a user initiated request), or a continuous stream (e.g., a live video feed of a patient).

[0210] As another example, a UE comprises an actuator, a motor, or a switch, related to a communication interface configured to receive wireless input from a network node via a wireless connection. In response to the received wireless input the states of the actuator, the motor, or the switch may change. For example, the UE may comprise a motor that adjusts the control surfaces or rotors of a drone in flight according to the received input or to a robotic arm performing a medical procedure according to the received input.

[0211] A UE, when in the form of an Internet of Things (loT) device, may be a device for use in one or more application domains, these domains comprising, but not limited to, city wearable technology, extended industrial application and healthcare. Non-limiting examples of such an loT device are a device which is or which is embedded in: a connected refrigerator or freezer, a TV, a connected lighting device, an electricity meter, a robot vacuum cleaner, a voice controlled smart speaker, a home security camera, a motion detector, a thermostat, a smoke detector, a door / window sensor, a flood / moisture sensor, an electrical door lock, a connected doorbell, an air conditioning system like a heat pump, an autonomous vehicle, a surveillance system, a weather monitoring device, a vehicle parking monitoring device, an electric vehicle charging station, a smart watch, a fitness tracker, a wearable for tactile augmentation or sensory enhancement, a water sprinkler, an animal- or item-tracking device, a sensor for monitoring a plant or animal, an industrial robot, an Unmanned Aerial Vehicle (UAV), and any kind of medical device, like a heart rate monitor or a remote controlled surgical robot. A UE in the form of an loT device comprises circuitry and / or software in dependence of the intended application of the loT device in addition to other components as described in relation to the UE 23200 shown in Figure 23.

[0212] As yet another specific example, in an loT scenario, a UE may represent a machine or other device that performs monitoring and / or measurements, and transmits the results of such monitoring and / or measurements to another UE and / or a network node. The UE may in this case be an M2M device, which may in a 3GPP context be referred to as an MTC device. As one particular example, the UE may implement the 3GPP NB-IoT standard. In other scenarios, a UE may represent a vehicle, such as a car, a bus, a truck, a ship and an airplane, or other equipment that is capable of monitoring and / or reporting on its operational status or other functions associated with its operation.

[0213] In practice, any number of UEs may be used together with respect to a single use case. For example, a first UE might be or be integrated in a drone and provide the drone’s speed information (obtained through a speed sensor) to a second UE that is a remote controller operating the drone. When the user makes changes from the remote controller, the first UE may adjust the throttle on the drone (e.g. by controlling an actuator) to increase or decrease the drone’s speed. The first and / or the second UE can also include more than one of the functionalities described above. For example, a UE might comprise the sensor and the actuator, and handle communication of data for both the speed sensor and the actuators.

[0214] Figure 24 shows a network node 24300 in accordance with some embodiments. As used herein, network node refers to equipment capable, configured, arranged and / or operable to communicate directly or indirectly with a UE and / or with other network nodes or equipment, in a telecommunication network. Examples of network nodes include, but are not limited to, access points (APs) (e.g., radio access points), base stations (BSs) (e.g., radio base stations, Node Bs, evolved Node Bs (eNBs) and NR NodeBs (gNBs)), O-RAN nodes or components of an O-RAN node (e.g., O-RU, O-DU, O-CU). Base stations may be categorized based on the amount of coverage they provide (or, stated differently, their transmit power level) and so, depending on the provided amount of coverage, may be referred to as femto base stations, pico base stations, micro base stations, or macro base stations. A base station may be a relay node or a relay donor node controlling a relay. A network node may also include one or more (or all) parts of a distributed radio base station such as centralized digital units, distributed units (e.g., in an O-RAN access node) and / or remote radio units (RRUs), sometimes referred to as Remote Radio Heads (RRHs). Such remote radio units may or may not be integrated with an antenna as an antenna integrated radio. Parts of a distributed radio base station may also be referred to as nodes in a distributed antenna system (DAS).

[0215] Other examples of network nodes include multiple transmission point (multi-TRP) 5G access nodes, multi-standard radio (MSR) equipment such as MSR BSs, network controllers such as radio network controllers (RNCs) or base station controllers (BSCs), base transceiver stations (BTSs), transmission points, transmission nodes, multi-cell / multicast coordination entities (MCEs), Operation and Maintenance (O&M) nodes, Operations Support System (OSS) nodes, Self-Organizing Network (SON) nodes, positioning nodes (e.g., Evolved Serving Mobile Location Centers (E-SMLCs)), and / or Minimization of Drive Tests (MDTs).

[0216] The network node 24300 includes a processing circuitry 24302, a memory 24304, a communication interface 24306, and a power source 24308. The network node 24300 may be composed of multiple physically separate components (e.g., a NodeB component and a RNC component, or a BTS component and a BSC component, etc.), which may each have their own respective components. In certain scenarios in which the network node 24300 comprises multiple separate components (e.g., BTS and BSC components), one or more of the separate components may be shared among several network nodes. For example, a single RNC may control multiple NodeBs. In such a scenario, each unique NodeB and RNC pair, may in some instances be considered a single separate network node. In some embodiments, the network node 24300 may be configured to support multiple radio access technologies (RATs). In such embodiments, some components may be duplicated (e.g., separate memory 24304 for different RATs) and some components may be reused (e.g., a same antenna 24310 may be shared by different RATs). The network node 24300 may also include multiple sets of the various illustrated components for different wireless technologies integrated into network node 24300, for example GSM, WCDMA, LTE, NR, WiFi, Zigbee, Z-wave, LoRaWAN, Radio Frequency Identification (RFID) or Bluetooth wireless technologies. These wireless technologies may be integrated into the same or different chip or set of chips and other components within network node 24300. The processing circuitry 24302 may comprise a combination of one or more of a microprocessor, controller, microcontroller, central processing unit, digital signal processor, application-specific integrated circuit, field programmable gate array, or any other suitable computing device, resource, or combination of hardware, software and / or encoded logic operable to provide, either alone or in conjunction with other network node 24300 components, such as the memory 24304, to provide network node 24300 functionality.

[0217] In some embodiments, the processing circuitry 24302 includes a system on a chip (SOC). In some embodiments, the processing circuitry 24302 includes one or more of radio frequency (RF) transceiver circuitry 24312 and baseband processing circuitry 24314. In some embodiments, the radio frequency (RF) transceiver circuitry 24312 and the baseband processing circuitry 24314 may be on separate chips (or sets of chips), boards, or units, such as radio units and digital units. In alternative embodiments, part or all of RF transceiver circuitry 24312 and baseband processing circuitry 24314 may be on the same chip or set of chips, boards, or units.

[0218] The memory 24304 may comprise any form of volatile or non-volatile computer-readable memory including, without limitation, persistent storage, solid-state memory, remotely mounted memory, magnetic media, optical media, random access memory (RAM), read-only memory (ROM), mass storage media (for example, a hard disk), removable storage media (for example, a flash drive, a Compact Disk (CD) or a Digital Video Disk (DVD)), and / or any other volatile or non-volatile, non-transitory device-readable and / or computer-executable memory devices that store information, data, and / or instructions that may be used by the processing circuitry 24302. The memory 24304 may store any suitable instructions, data, or information, including a computer program, software, an application including one or more of logic, rules, code, tables, and / or other instructions capable of being executed by the processing circuitry 24302 and utilized by the network node 24300. The memory 24304 may be used to store any calculations made by the processing circuitry 24302 and / or any data received via the communication interface 24306. In some embodiments, the processing circuitry 24302 and memory 24304 is integrated.

[0219] The communication interface 24306 is used in wired or wireless communication of signaling and / or data between a network node, access network, and / or UE. As illustrated, the communication interface 24306 comprises port(s) / terminal(s) 24316 to send and receive data, for example to and from a network over a wired connection. The communication interface 24306 also includes radio front-end circuitry 24318 that may be coupled to, or in certain embodiments a part of, the antenna 24310. Radio front-end circuitry 24318 comprises filters 24320 and amplifiers 24322. The radio front-end circuitry 24318 may be connected to an antenna 24310 and processing circuitry 24302. The radio front-end circuitry may be configured to condition signals communicated between antenna 24310 and processing circuitry 24302. The radio front-end circuitry 24318 may receive digital data that is to be sent out to other network nodes or UEs via a wireless connection. The radio front-end circuitry 24318 may convert the digital data into a radio signal having the appropriate channel and bandwidth parameters using a combination of filters 24320 and / or amplifiers 24322. The radio signal may then be transmitted via the antenna 24310. Similarly, when receiving data, the antenna 24310 may collect radio signals which are then converted into digital data by the radio front-end circuitry 24318. The digital data may be passed to the processing circuitry 24302. In other embodiments, the communication interface may comprise different components and / or different combinations of components.

[0220] In certain alternative embodiments, the network node 24300 does not include separate radio front-end circuitry 24318, instead, the processing circuitry 24302 includes radio front-end circuitry and is connected to the antenna 24310. Similarly, in some embodiments, all or some of the RF transceiver circuitry 24312 is part of the communication interface 24306. In still other embodiments, the communication interface 24306 includes one or more ports or terminals 24316, the radio front-end circuitry 24318, and the RF transceiver circuitry 24312, as part of a radio unit (not shown), and the communication interface 24306 communicates with the baseband processing circuitry 24314, which is part of a digital unit (not shown).

[0221] The antenna 24310 may include one or more antennas, or antenna arrays, configured to send and / or receive wireless signals. The antenna 24310 may be coupled to the radio front-end circuitry 24318 and may be any type of antenna capable of transmitting and receiving data and / or signals wirelessly. In certain embodiments, the antenna 24310 is separate from the network node 24300 and connectable to the network node 24300 through an interface or port.

[0222] The antenna 24310, communication interface 24306, and / or the processing circuitry 24302 may be configured to perform any receiving operations and / or certain obtaining operations described herein as being performed by the network node. Any information, data and / or signals may be received from a UE, another network node and / or any other network equipment. Similarly, the antenna 24310, the communication interface 24306, and / or the processing circuitry 24302 may be configured to perform any transmitting operations described herein as being performed by the network node. Any information, data and / or signals may be transmitted to a UE, another network node and / or any other network equipment.

[0223] The power source 24308 provides power to the various components of network node 24300 in a form suitable for the respective components (e.g., at a voltage and current level needed for each respective component). The power source 24308 may further comprise, or be coupled to, power management circuitry to supply the components of the network node 24300 with power for performing the functionality described herein. For example, the network node 24300 may be connectable to an external power source (e.g., the power grid, an electricity outlet) via an input circuitry or interface such as an electrical cable, whereby the external power source supplies power to power circuitry of the power source 24308. As a further example, the power source 24308 may comprise a source of power in the form of a battery or battery pack which is connected to, or integrated in, power circuitry. The battery may provide backup power should the external power source fail.

[0224] Embodiments of the network node 24300 may include additional components beyond those shown in Figure 24 for providing certain aspects of the network node’s functionality, including any of the functionality described herein and / or any functionality necessary to support the subject matter described herein. For example, the network node 24300 may include user interface equipment to allow input of information into the network node 24300 and to allow output of information from the network node 24300. This may allow a user to perform diagnostic, maintenance, repair, and other administrative functions for the network node 24300. In some embodiments providing a core network node, such as core network node 108 of FIG. 241, some components, such as the radio front-end circuitry 24318 and the RF transceiver circuitry 24312 may be omitted.

[0225] Figure 25 is a block diagram illustrating a virtualization environment 25400 in which functions implemented by some embodiments may be virtualized. In the present context, virtualizing means creating virtual versions of apparatuses or devices which may include virtualizing hardware platforms, storage devices and networking resources. As used herein, virtualization can be applied to any device described herein, or components thereof, and relates to an implementation in which at least a portion of the functionality is implemented as one or more virtual components. Some or all of the functions described herein may be implemented as virtual components executed by one or more virtual machines (VMs) implemented in one or more virtual environments 25400 hosted by one or more of hardware nodes, such as a hardware computing device that operates as a network node, UE, core network node, or host. Further, in embodiments in which the virtual node does not require radio connectivity (e.g., a core network node or host), then the node may be entirely virtualized. In some embodiments, the virtualization environment 25400 includes components defined by the O-RAN Alliance, such as an O-Cloud environment orchestrated by a Service Management and Orchestration Framework via an O-2 interface. Virtualization may facilitate distributed implementations of a network node, UE, core network node, or host.

[0226] Applications 25402 (which may alternatively be called software instances, virtual appliances, network functions, virtual nodes, virtual network functions, etc.) are run in the virtualization environment Q400 to implement some of the features, functions, and / or benefits of some of the embodiments disclosed herein. Hardware 25404 includes processing circuitry, memory that stores software and / or instructions executable by hardware processing circuitry, and / or other hardware devices as described herein, such as a network interface, input / output interface, and so forth. Software may be executed by the processing circuitry to instantiate one or more virtualization layers 25406 (also referred to as hypervisors or virtual machine monitors (VMMs)), provide VMs 25408a and 25408b (one or more of which may be generally referred to as VMs 25408), and / or perform any of the functions, features and / or benefits described in relation with some embodiments described herein. The virtualization layer 25406 may present a virtual operating platform that appears like networking hardware to the VMs 25408.

[0227] The VMs 25408 comprise virtual processing, virtual memory, virtual networking or interface and virtual storage, and may be run by a corresponding virtualization layer 25406. Different embodiments of the instance of a virtual appliance 25402 may be implemented on one or more of VMs 25408, and the implementations may be made in different ways. Virtualization of the hardware is in some contexts referred to as network function virtualization (NFV). NFV may be used to consolidate many network equipment types onto industry standard high volume server hardware, physical switches, and physical storage, which can be located in data centers, and customer premise equipment.

[0228] In the context of NFV, a VM 25408 may be a software implementation of a physical machine that runs programs as if they were executing on a physical, non-virtualized machine. Each of the VMs 25408, and that part of hardware 25404 that executes that VM, be it hardware dedicated to that VM and / or hardware shared by that VM with others of the VMs, forms separate virtual network elements. Still in the context of NFV, a virtual network function is responsible for handling specific network functions that run in one or more VMs 25408 on top of the hardware 25404 and corresponds to the application 25402.

[0229] Hardware 25404 may be implemented in a standalone network node with generic or specific components. Hardware 25404 may implement some functions via virtualization. Alternatively, hardware 25404 may be part of a larger cluster of hardware (e.g. such as in a data center or CPE) where many hardware nodes work together and are managed via management and orchestration 25410, which, among others, oversees lifecycle management of applications 25402. In some embodiments, hardware 25404 is coupled to one or more radio units that each include one or more transmitters and one or more receivers that may be coupled to one or more antennas. Radio units may communicate directly with other hardware nodes via one or more appropriate network interfaces and may be used in combination with the virtual components to provide a virtual node with radio capabilities, such as a radio access node or a base station. In some embodiments, some signaling can be provided with the use of a control system 25412 which may alternatively be used for communication between hardware nodes and radio units.

[0230] Although the computing devices described herein (e.g., UEs, network nodes) may include the illustrated combination of hardware components, other embodiments may comprise computing devices with different combinations of components. It is to be understood that these computing devices may comprise any suitable combination of hardware and / or software needed to perform the tasks, features, functions and methods disclosed herein. Determining, calculating, obtaining or similar operations described herein may be performed by processing circuitry, which may process information by, for example, converting the obtained information into other information, comparing the obtained information or converted information to information stored in the network node, and / or performing one or more operations based on the obtained information or converted information, and as a result of said processing making a determination. Moreover, while components are depicted as single boxes located within a larger box, or nested within multiple boxes, in practice, computing devices may comprise multiple different physical components that make up a single illustrated component, and functionality may be partitioned between separate components. For example, a communication interface may be configured to include any of the components described herein, and / or the functionality of the components may be partitioned between the processing circuitry and the communication interface. In another example, non- computationally intensive functions of any of such components may be implemented in software or firmware and computationally intensive functions may be implemented in hardware.

[0231] In certain embodiments, some or all of the functionality described herein may be provided by processing circuitry executing instructions stored on in memory, which in certain embodiments may be a computer program product in the form of a non-transitory computer-readable storage medium. In alternative embodiments, some or all of the functionality may be provided by the processing circuitry without executing instructions stored on a separate or discrete device-readable storage medium, such as in a hard-wired manner. In any of those particular embodiments, whether executing instructions stored on a non-transitory computer-readable storage medium or not, the processing circuitry can be configured to perform the described functionality. The benefits provided by such functionality are not limited to the processing circuitry alone or to other components of the computing device, but are enjoyed by the computing device as a whole, and / or by end users and a wireless network generally.

[0232] Figure 26 shows a wireless device 2600, which may be configured to operate in communication system 4100 of Fig. 13 or in communication system 4200 of Fig. 14. The wireless device 2600 may be alternatively referred to as a UE 2600, like a UE 4112 within the context of communication system 4100, or as a station (STA) 2600 or as a non-access-point station (non-AP STA) 2600, like a STA 4212 within the context of the communication system 4200, in accordance with respective embodiments. As used herein, a wireless device refers to a device capable, configured, arranged and / or operable to communicate wirelessly with network nodes and / or other wireless devices. Examples of a wireless device include, but are not limited to, a smart phone, mobile phone, cell phone, voice over IP (VoIP) phone, wireless local loop phone, desktop computer, personal digital assistant (PDA), wireless cameras, gaming console or device, music storage device, playback appliance, wearable terminal device, wireless endpoint, mobile station, tablet, laptop, laptop-embedded equipment (LEE), laptop-mounted equipment (LME), smart device, wireless customer-premise equipment (CPE), vehicle, vehicle-mounted or vehicle embedded / integrated wireless device, and wireless terminal. Other examples include any type of UE identified by the 3rd Generation Partnership Project (3GPP), including a narrow band internet of things (NB-IoT) UE, a machine type communication (MTC) UE, and / or an enhanced MTC (eMTC) UE.

[0233] A wireless device 2600 may support device-to-device (D2D) communication, for example by implementing a 3 GPP standard for sidelink communication, Dedicated Short-Range Communication (DSRC), vehicle-to-vehicle (V2V), vehicle-to-infrastructure (V2I), or vehicle-to- everything (V2X). In other examples, wireless device 2600 may not necessarily have a user in the sense of a human user who owns and / or operates the relevant device. Instead, wireless device 2600 may represent a device that is intended for sale to, or operation by, a human user but which may not, or which may not initially, be associated with a specific human user (e.g., a smart sprinkler controller). Alternatively, wireless device 2600 may represent a device that is not intended for sale to, or operation by, an end user but which may be associated with or operated for the benefit of a user (e.g., a smart power meter).

[0234] In particular embodiments, wireless device 2600 includes processing circuitry 2602 that is operatively coupled via a bus 2604 to an input / output interface 2606, a power source 2608, a memory 2610, a communication interface 2612, and / or any other component, or any combination thereof. Certain embodiments of wireless device 2600 may include all or a subset of the components shown in Figure 26. The level of integration between the components may vary from one embodiment of wireless device 2600 to another. In general, in a particular embodiment of wireless device 2600, processing circuitry 2602, input / output interface 2606, power source 2608, memory 2610, and communication interface 2612 may, in whole or in part, represent or include physical components common to or shared by one or more of the other elements of wireless device 2600. Further, certain embodiments of wireless devices 2600 may contain multiple instances of a component, such as multiple processors, memories, transceivers, transmitters, receivers, etc.

[0235] The processing circuitry 2602 is configured to process instructions and data and may be configured to implement any sequential state machine operative to execute instructions stored as machine-readable computer programs in the memory 2610. The processing circuitry 2602 may be implemented as one or more hardware-implemented state machines (e.g., in discrete logic, field- programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.); programmable logic together with appropriate firmware; one or more stored computer programs, general-purpose processors, such as a microprocessor or digital signal processor (DSP), together with appropriate software; or any combination of the above. For example, the processing circuitry 2602 may include multiple central processing units (CPUs).

[0236] In the example, the input / output interface 2606 may be configured to provide an interface or interfaces to an input device, output device, or one or more input and / or output devices. Examples of an output device include a speaker, a sound card, a video card, a display, a monitor, a printer, an actuator, an emitter, a smartcard, another output device, or any combination thereof. An input device may allow a user to capture information into wireless device 2600. Examples of an input device include a touch-sensitive or presence-sensitive display, a camera (e.g., a digital camera, a digital video camera, a web camera, etc.), a microphone, a sensor, a mouse, a trackball, a directional pad, a trackpad, a scroll wheel, a smartcard, and the like. The presence-sensitive display may include a capacitive or resistive touch sensor to sense input from a user. A sensor may be, for instance, an accelerometer, a gyroscope, a tilt sensor, a force sensor, a magnetometer, an optical sensor, a proximity sensor, a biometric sensor, etc., or any combination thereof. An output device may use the same type of interface port as an input device. For example, a Universal Serial Bus (USB) port may be used to provide an input device and an output device.

[0237] In some embodiments, the power source 2608 is structured as a battery or battery pack. Other types of power sources, such as an external power source (e.g., an electricity outlet), photovoltaic device, or power cell, may be used to supply power to circuitry or to charge an associated battery. The power source 2608 may further include power circuitry for delivering power from the power source 2608 itself, and / or an external power source, to the various parts of wireless device 2600 via input circuitry or an interface such as an electrical power cable. Power source 2608 may perform any formatting, converting, or other modification to make accessible power suitable for the respective components of the wireless device 2600 to which power is supplied.

[0238] The memory 2610 may be or be configured to include memory such as random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic disks, optical disks, hard disks, removable cartridges, flash drives, and so forth. In one example, the memory 2610 includes one or more programs 2614, such as an operating system, web browser application, a widget, gadget engine, or other application, and corresponding data 2616. The memory 2610 may store, for use by wireless device 2600, any of a variety of various operating systems or combinations of operating systems.

[0239] The memory 2610 may be configured to include a number of physical drive units, such as redundant array of independent disks (RAID), flash memory, USB flash drive, external hard disk drive, thumb drive, pen drive, key drive, high-density digital versatile disc (HD-DVD) optical disc drive, internal hard disk drive, Blu-Ray optical disc drive, holographic digital data storage (HDDS) optical disc drive, external mini-dual in-line memory module (DIMM), synchronous dynamic random access memory (SDRAM), external micro-DIMM SDRAM, smartcard memory such as tamper resistant module in the form of a universal integrated circuit card (UICC) including one or more subscriber identity modules (SIMs), such as a USIM and / or ISIM, other memory, or any combination thereof. The UICC may for example be an embedded UICC (eUICC), integrated UICC (iUICC) or a removable UICC commonly known as ‘SIM card.’ The memory 2610 may allow wireless device 2600 to access instructions, programs and the like, stored on transitory or non-transitory memory media, to off-load data, or to upload data. An article of manufacture, such as one utilizing a communication system may be tangibly embodied as or in the memory 2610, which may be or comprise a device-readable storage medium.

[0240] The processing circuitry 2602 may be configured to communicate with an access network or other network via or using the communication interface 2612. The communication interface 2612 may comprise one or more communication subsystems and may include or be communicatively coupled to an antenna 2622. The communication interface 2612 may include one or more transceivers used to communicate, such as by communicating with one or more remote transceivers of another device capable of wireless communication (e.g., another wireless device or a network node in an access network). Each transceiver may include a transmitter 2618 and / or a receiver 2620 appropriate to provide network communications (e.g., optical, electrical, frequency allocations, and so forth). Moreover, the transmitter 2618 and receiver 2620 may be coupled to one or more antennas (e.g., antenna 2622) and may share circuit components, software or firmware, or alternatively be implemented separately.

[0241] In the illustrated embodiment, communication functions of the communication interface 2612 may include cellular communication, Wi-Fi communication (e.g., according to an IEEE 802.11 family standard), LPWAN communication, data communication, voice communication, multimedia communication, short-range communications such as Bluetooth, near-field communication, location-based communication such as the use of the global positioning system (GPS) to determine a location, another like communication function, or any combination thereof. Communications may be implemented according to one or more communication protocols and / or standards, such as IEEE 802.11, Code Division Multiplexing Access (CDMA), Wideband Code Division Multiple Access (WCDMA), GSM, LTE, New Radio (NR), UMTS, WiMax, Ethernet, transmission control protocol / intemet protocol (TCP / IP), synchronous optical networking (SONET), Asynchronous Transfer Mode (ATM), QUIC, Hypertext Transfer Protocol (HTTP), and so forth.

[0242] In particular embodiments, wireless device 2600 may provide an output of data captured via a sensor, through its communication interface 2612, via a wireless connection to a network node, and / or in any appropriate manner. Data captured by sensors of a wireless device 2600 can be communicated through a wireless connection to a network node via another wireless device 2600. In particular embodiments, such output may be periodic (e.g., once every 15 minutes if it reports the sensed temperature), random (e.g., to even out the load from reporting from several sensors), in response to a triggering event (e.g., when moisture is detected an alert is sent), in response to a request (e.g., a user initiated request), or a continuous stream (e.g., a live video feed of a patient).

[0243] As another example, wireless device 2600 comprises an actuator, a motor, or a switch, related to a communication interface configured to receive wireless input from a network node via a wireless connection. In response to the received wireless input the states of the actuator, the motor, or the switch may change. For example, wireless device 2600 may comprise a motor that adjusts the control surfaces or rotors of a drone in flight according to the received input or to a robotic arm performing a medical procedure according to the received input.

[0244] Wireless device 2600, when in the form of an Internet of Things (loT) device, may be a device for use in one or more application domains, these domains comprising, but not limited to, wearable technology, extended industrial application and healthcare. Non-limiting examples of such an loT device are a device which is or which is embedded in: a connected refrigerator or freezer, a TV, a connected lighting device, an electricity meter, a robot vacuum cleaner, a voice controlled smart speaker, a home security camera, a motion detector, a thermostat, a smoke detector, a door / window sensor, a flood / moisture sensor, an electrical door lock, a connected doorbell, an air conditioning system like a heat pump, an autonomous vehicle, a surveillance system, a weather monitoring device, a vehicle parking monitoring device, an electric vehicle charging station, a smart watch, a fitness tracker, a wearable for tactile augmentation or sensory enhancement, a water sprinkler, an animal- or item-tracking device, a sensor for monitoring a plant or animal, an industrial robot, an Unmanned Aerial Vehicle (UAV), and any kind of medical device, like a heart rate monitor or a remote controlled surgical robot. In particular embodiments, wireless device 2600 represents an loT device that comprises circuitry and / or software in dependence of the intended application of the loT device in addition to other components as described in relation to the example embodiment of wireless device 2600 shown in Figure 26.

[0245] As yet another specific example, in an loT scenario, wireless device 2600 may represent a machine or other device that performs monitoring and / or measurements, and transmits the results of such monitoring and / or measurements to another wireless device and / or a network node. Wireless device 2600 may in this case be an M2M device, which may in a 3GPP context be referred to as an MTC device. As one particular example, wireless device 2600 may implement the 3GPP NB-IoT standard. In other scenarios, wireless device 2600 may represent a vehicle, such as a car, a bus, a truck, a ship and an airplane, or other equipment that is capable of monitoring and / or reporting on its operational status or other functions associated with its operation.

[0246] In practice, any number of wireless devices 2600 may be used together with respect to a single use case. For example, a first wireless device 2600 might be or be integrated in a drone and provide the drone’s speed information (obtained through a speed sensor) to a second wireless device 2600 that is a remote controller operating the drone. When a user makes changes from the remote controller, the first wireless device 2600 may adjust the throttle on the drone (e.g. by controlling an actuator) to increase or decrease the drone’s speed. The first and / or the second wireless device 2600 can also include more than one of the functionalities described above. For example, wireless device 2600 might comprise the sensor and the actuator, and handle communication of data for both the speed sensor and the actuators.

[0247] In certain embodiments, some or all of the functionality described herein may be provided by processing circuitry executing instructions stored on in memory, which in certain embodiments may be a computer program product in the form of a non-transitory computer-readable storage medium. In alternative embodiments, some or all of the functionality may be provided by the processing circuitry without executing instructions stored on a separate or discrete device-readable storage medium, such as in a hard-wired manner. In any of those particular embodiments, whether executing instructions stored on a non-transitory computer-readable storage medium or not, the processing circuitry can be configured to perform the described functionality. The benefits provided by such functionality are not limited to the processing circuitry alone or to other components of the computing device, but are enjoyed by the computing device as a whole, and / or by end users and a wireless network generally.

[0248] Example Embodiments

[0249] Below are provided a number of non-limiting example embodiments under the present disclosure.

[0250] A first embodiment comprises a method for decoding a picture comprising a neural network filtering process from a coded video bitstream, the method comprising determining a maximum allowed complexity from one or more syntax elements in the bitstream, wherein the maximum allowed complexity corresponds to the neural network filtering process.

[0251] A second embodiment comprises the method of the first embodiment, further comprising decoding the picture using the maximum allowed complexity.

[0252] A third embodiment comprises the method for decoding a picture comprising a neural network filtering process from a coded video bitstream, the method comprising obtaining an indication of a maximum allowed complexity, wherein the maximum allowed complexity corresponds to the neural network filtering process.

[0253] A fourth embodiment comprises the method of the third embodiment, further comprising decoding the picture using the maximum allowed complexity.

[0254] A fifth embodiment comprises the method of the third or fourth embodiment, wherein the indication of the maximum allowed complexity is derived from one or more syntax elements in the bitstream. A sixth embodiment comprises the method of any of the third through fifth embodiments, wherein the maximum allowed complexity is a constraint on the bitstream and / or the decoder.

[0255] A seventh embodiment comprises the method of any of the previous embodiments, wherein the maximum allowed complexity is specified for a picture, a portion of a picture, a set of pictures, or a coded video sequence (CVS).

[0256] An eighth embodiment comprises the method of any of the previous embodiments, wherein the maximum allowed complexity is defined as one or more of: the maximum allowed MAC / pixel for NN filtering; the maximum allowed samples that use NN filtering; the maximum allowed percentage (or ratio) of samples that use NN filtering; the maximum allowed NN blocks or percentage (ratio) of NN blocks that use NN filtering; the maximum allowed CTUs or percentage (ratio) of CTUs that use NN filtering.

[0257] A nineth embodiment comprises the method of any of the previous embodiments, wherein decoding the picture using the maximum allowed complexity comprises determining that a current portion of the picture is not using the neural network filtering process in response to a complexity value being larger than the maximum allowed complexity.

[0258] A tenth embodiment comprises the method of any of the previous embodiments, wherein decoding the picture using the maximum allowed complexity comprises determining that a current portion of the picture is using the neural network filtering process in response to a complexity value being smaller than or equal to the maximum allowed complexity.

[0259] An eleventh embodiment comprises the method of the tenth embodiment, wherein determining that a current portion of the picture is using the neural network filtering process further comprises decoding a first syntax element (e.g. named ctu_nn_loop_filter_flag) from the coded video bitstream into a first value, wherein the first value specifies that the current portion of the picture uses neural network filtering.

[0260] A twelfth embodiment comprises the method of the tenth or eleventh embodiment, wherein determining that a current portion of the picture is using the neural network filtering process further takes into account one or more of: the scanning order of the portions in the picture; the current portion of the picture consists of a block which is marked by block importance mapping (BIM) tool; the boundary strength (BS) information of at least one block in the current portion of the picture; block type of at least one block in the current portion of the picture; QP of at least one block in the current portion of the picture; a picture type of the picture containing the current portion of the picture; a temporal ID of the picture containing the current portion of the picture; a layer ID of the picture containing the current portion of the picture. A thirteenth embodiment comprises the method of any of the previous embodiments, wherein the maximum allowed complexity from one or more syntax elements in the bitstream is determined in response to a decoded value from a second syntax element (e.g. named sps nnlf enabled flag) in the bitstream being equal to a second value.

[0261] A fourteenth embodiment comprises the method of any of the previous embodiments, wherein the maximum allowed complexity is decoded from a parameter set comprising at least one of: a PPS; a SPS; a VPS; a header; a picture header; a slice header; a decoding capability information, DCI, NAL unit.

[0262] A fifteenth embodiment comprises the method of any of the previous embodiments, wherein a level, a tier or a codec profile specifies the maximum allowed complexity.

[0263] A sixteenth embodiment comprises the method of the fifteenth embodiment, wherein two or more profiles are specified where a first profile such as a main profile does not allow neural network filtering, and a second profile such as an extended profile allows neural network filtering.

[0264] A seventeenth embodiment comprises the method of the fifteenth embodiment, wherein two or more profiles are specified where a first profile does allow a first type of neural network filtering, and a second profile allows a second type of neural network filtering.

[0265] An eighteenth embodiment comprises the method of any of the previous embodiments, wherein at least one of the following applies: at least one block of a picture is using NN filtering and at least one block of the picture is not using NN-filtering; at least one portion of a picture is using NN filtering and at least one portion of the picture is not using NN-filtering; at least one picture in a set of pictures in the bitstream is using NN-filtering and at least one picture in a set of pictures in the bitstream is not using NN-filtering.

[0266] A nineteenth embodiment comprises the method of any of the previous embodiments, wherein the picture order count (POC) can be used to determine whether a picture can be filtered or not.

[0267] A twentieth embodiment comprises the method of the nineteenth embodiment, wherein after decoding, the pictures are displayed in increasing POC order.

[0268] A twenty-first embodiment comprises the method of the nineteenth or twentieth embodiment, wherein a picture in a hierarchical layer can only predict from (e.g., copy samples from) a picture in a hierarchical layer equal to or lower than its own hierarchical layer.

[0269] A twenty-second embodiment comprises the method of any of the nineteenth through twenty-first embodiment, wherein the decoder can infer from the POC whether or not any blocks in the picture are allowed to undergo filtering. A twenty -third embodiment comprises the method of any of the nineteenth through twenty- second embodiments, wherein if (POC mod 2 == 0), then # Blocks in this picture are allowed to be filtered using NN loop filter, else # No blocks in this picture are allowed to be filtered using NN loop filter; wherein mod is the modulos operator, meaning that for the integers A and B, the expression A mod B gives the remainder of the integer division A div B, so that A = (A div B) * B + (A mod B) holds.

[0270] A twenty-fourth embodiment comprises the method of any of the nineteenth through twenty-second embodiments, wherein if (POC mod 4 == 0), then # Blocks in this picture are allowed to be filtered using NN loop filter; else # No blocks in this picture are allowed to be filtered using NN loop filter.

[0271] A twenty-fifth embodiment comprises the method of any of the nineteenth through twenty- second embodiments, wherein: if (POC mod 32 == 0), then # Blocks in this picture are allowed to be filtered using NN loop filter; else # No blocks in this picture are allowed to be filtered using NN loop filter.

[0272] A twenty-sixth embodiment comprises the method of any of the nineteenth through twenty- second embodiments, wherein, if (POC mod 2 == 0), then # Encode this image with NN loop filter allowed; else # Encode this image with NN loop filter turned off.

[0273] A twenty -seventh embodiment comprises the method of any of the nineteenth through twenty-second embodiments, wherein, if ((POC / j) mod M == k), then # Blocks in this picture are allowed to be filtered using NN loop filter; else # No blocks in this picture are allowed to be filtered using NN loop filter.

[0274] A twenty-eighth embodiment comprises the method of any of the previous embodiments, wherein hierarchical layers are temporal sublayers and HIDs are TemporallDs.

[0275] A twenty-nineth embodiment comprises the method of any of the previous embodiments, wherein hierarchical layers are layers (such as scalable or multiview layers) and HIDs are layer IDs.

[0276] A thirtieth embodiment comprises the method of any of the previous embodiments, wherein hierarchical layers are not necessarily temporal sublayers or layers. For instance, multiple hierarchical layers with HIDs 0-5 could be within one temporal sublayer with TemporallD 0, or hierarchical layers with HIDs 4-5 could be within one temporal sublayer with TemporallD 3.

[0277] A thirty-first embodiment comprises the method of any of the previous embodiments, wherein only pictures belonging to a specific hierarchical layer, e.g.. HID 0, are allowed to be filtered using NN loop filter. A thirty-second embodiment comprises the method of any of the previous embodiments, wherein only pictures with aHID below a certain value, e.g. HIDs < 4, i.e. HIDs 0 - 3, are allowed to be filtered using NN loop filter.

[0278] A thirty-third embodiment comprises the method of any of the previous embodiments, wherein one or more syntax elements in the bitstream are used to specify which HIDs that may be allowed to be filtered using NN loop filter. These one or more syntax elements may be signaled in a parameter set, such as a VPS, an SPS, a PPS or an APS, in a DCI, or in a header such as a picture header or a slice header.

[0279] A thirty-fourth embodiment comprises the method of any of the previous embodiments, wherein different NN filters of different complexity are used, wherein pictures with a certain set of HIDs are allowed to be filtered using a first NN loop filter of a first complexity, and a second set of HIDs are allowed to be filtered using a second NN loop filter of a second complexity that is different from the first complexity. For instance, pictures with HID 0 may be allowed to be filtered using a first NN loop filter of a first complexity, while pictures with HIDs 1 -3 may only be allowed to be filtered using a second NN loop filter of a second complexity that is lower than the first complexity, and pictures with HIDs 4-5 may not be allowed to be filtered using an NN loop filter at all.

[0280] A thirty-fifth embodiment comprises the method of any of the previous embodiments, wherein such determination of filter maybe done using the POC of a picture according to: if (POC mod 2 == 0) and if (POC mod 4 == 0), then # Encode this image with NN loop filter of first complexity allowed; if (POC mod 2 == 0) and if (POC mod 4 != 0) # Encode this image with NN loop filter of second complexity allowed; if (POC mod 2 != 0) # Encode this image without any NN loop filtering.

[0281] A thirty-sixth embodiment comprises the method of any of the previous embodiments, wherein only pictures that are (or may) be used for reference by other pictures are allowed to be filtered using NN loop filter.

[0282] A thirty-seventh embodiment comprises the method of any of the previous embodiments, wherein there is a picture enable syntax element or a slice enable syntax element that can specify that NN in-loop filtering is not done for the picture or slice.

[0283] A thirty-eighth embodiment comprises the method of any of the previous embodiments, wherein the encoder may set the syntax element to disable NN inloop filtering for any of the following cases: disable filtering for pictures with an odd POC value; disable filtering for any picture with a POC value P not evenly divisible by 4 (i.e. do not filter if ((P mod 4) != 0); disable filtering for pictures belonging to one or more specific layer or temporal layer; disable filtering for picture belonging to the N highest hierarchical levels in a hierarchical bi-directional structure of picture referencing; and / or disable filtering for non-reference pictures

[0284] A thirty-ninth embodiment comprises a method for encoding a picture to a coded video bitstream, the method comprising a neural network (NN) filtering process, the method further comprising: obtaining a maximum allowed complexity (complexity cap N) for the NN-filtering; determining, based on the obtained complexity cap N, which portions in the picture to encode using NN-filtering and / or which portions to not encode using NN-filtering; and encoding the picture using NN-filtering for the portions determined to use NN-filtering and no NN-filtering for all other portions.

[0285] A fortieth embodiment comprises the method of the thirty-nineth embodiment, further comprising encoding an indication of the maximum allowed complexity for the NN-filtering to the bitstream.

[0286] A forty-first embodiment comprises the method of the thirty-nineth or fortieth embodiment, further comprising calculating or estimating performance gains of using NN-filtering for at least one portion in the picture.

[0287] A forty-second embodiment comprises the method of the forty-first embodiment, wherein the determining is further based on the calculated or estimated performance gains of using NN- filtering for at least one portion in the picture.

[0288] A forty-third embodiment comprises the method of the forty-first or forty-second embodiment, wherein calculating or estimating the performance gains comprises one or more of the following: subsample the portions in the picture; encode the picture using NN-filtering for all portions; encode the picture without using NN-filtering for any portions; calculate a performance metric for each portion encoded using NN-filtering; calculate a performance metric for each portion encoded not using NN-filtering; calculate a difference between the first performance metric and the second performance metric for each portion; prioritize to use NN-filtering for the portions for which the calculated difference has the highest performance gains of using NN- filtering; prioritize a portion based on the portion types; prioritize a portion based on the amount of boundary partitioning.

[0289] A forty -fourth embodiment comprises the method of any of the thirty-nineth through forty- third embodiments, wherein the determining that a current portion of the picture is to be encoded using NN-filtering further takes into account one or more of: the scanning order of the portions in the picture; the current portion of the picture consists of a block which is marked by block importance mapping (BIM) tool; the boundary strength (BS) information of at least one block in the current portion of the picture; block type of at least one block in the current portion of the picture; QP of at least one block in the current portion of the picture; a picture type of the picture containing the current portion of the picture; a temporal ID of the picture containing the current portion of the picture; a layer ID of the picture containing the current portion of the picture.

[0290] A forty-fifth embodiment comprises the method of any of the thirty-nineth through fortyfourth embodiments, wherein the maximum allowed complexity is a constraint on the bitstream and / or the decoder.

[0291] A forty-sixth embodiment comprises the method of any of the thirty-nineth through fortyfifth embodiments, wherein the maximum allowed complexity is specified for a picture, a portion of a picture, a set of pictures, or a coded video sequence (CVS).

[0292] A forty-seventh embodiment comprises the method of any of thirty-nineth through fortysixth embodiment, wherein the maximum allowed complexity is defined as one or more of: the maximum allowed MAC / pixel for NN filtering; the maximum allowed samples that use NN filtering; the maximum allowed percentage (or ratio) of samples that use NN filtering; the maximum allowed NN blocks or percentage (ratio) of NN blocks that use NN filtering; the maximum allowed CTUs or percentage (ratio) of CTUs that use NN filtering.

[0293] A forty-eighth embodiment comprises the method of any of thirty-nineth through fortyseventh embodiments, further comprising encoding a first value to a first syntax element (e.g. named ctu nn loop filter flag) in the bitstream, wherein the first value specifies that the current portion of the picture uses neural network filtering.

[0294] A forty-nineth embodiment comprises the method of any of embodiments 39 to 48, further comprising encoding a second value to a second syntax element (e.g. named sps nnlf enabled flag) in the bitstream, wherein the maximum allowed complexity is obtained based on the second value.

[0295] A fiftieth embodiment comprises the method of any of thirty-nineth through forty-nineth embodiments, wherein the indication of the maximum allowed complexity is encoded to a parameter set such as a PPS, SPS, VPS, or a header such as a picture header or a slice header, or a decoding capability information (DCI) NAL unit.

[0296] A fifty-first embodiment comprises the method of any of the thirty-nineth through fiftieth embodiments, wherein a level, a tier or a codec profile specifies the maximum allowed complexity.

[0297] A fifty-second embodiment comprises the method of any of the thirty-nineth through fifty- first embodiments, wherein at least one of the following applies: at least one block of a picture is using NN filtering and at least one block of the picture is not using NN-filtering; at least one portion of a picture is using NN filtering and at least one portion of the picture is not using NN- filtering; at least one picture in a set of pictures in the bitstream is using NN-filtering and at least one picture in a set of pictures in the bitstream is not using NN-filtering.

[0298] A fifty-third embodiment comprises the method of any of the thirty-nineth through fifty- second embodiments, wherein the picture order count (POC) can be used to determine whether a picture can be filtered or not.

[0299] A fifty-fourth embodiment comprises the method of the fifty-third embodiment, wherein after decoding, the pictures are displayed in increasing POC order.

[0300] A fifty-fifth embodiment comprises the method of the fifty-third or fifty-fourth embodiment, wherein a picture in a hierarchical layer can only predict from (e.g., copy samples from) a picture in a hierarchical layer equal to or lower than its own hierarchical layer.

[0301] A fifty-sixth embodiment comprises the method of any of the fifty-third through fifty -fifth embodiments, wherein the decoder can infer from the POC whether or not any blocks in the picture are allowed to undergo filtering.

[0302] A fifty-seventh embodiment comprises the method of any of the fifty-third to fifty-sixth embodiment, wherein if (POC mod 2 == 0), then # Blocks in this picture are allowed to be filtered using NN loop filter, else # No blocks in this picture are allowed to be filtered using NN loop filter; wherein mod is the modulos operator, meaning that for the integers A and B, the expression A mod B gives the remainder of the integer division A div B, so that A = (A div B) * B + (A mod B) holds.

[0303] A fifty-eighth embodiment comprises the method of any of the fifty-third through fiftysixth embodiments, wherein if (POC mod 4 == 0), then # Blocks in this picture are allowed to be filtered using NN loop filter; else # No blocks in this picture are allowed to be filtered using NN loop filter.

[0304] A fifty-nineth embodiment comprises the method of any of the fifty-third through fiftysixth embodiments, wherein if (POC mod 32 == 0), then # Blocks in this picture are allowed to be filtered using NN loop filter; else # No blocks in this picture are allowed to be filtered using NN loop filter.

[0305] A sixtieth embodiment comprises the method of any of the fifty-third through fifty-sixth embodiments, wherein if (POC mod 2 == 0), then # Encode this image with NN loop filter allowed; else # Encode this image with NN loop filter turned off. A sixty -first embodiment comprises the method of any of the fifty-third through fifty-sixth embodiment, wherein: if ((POC / j) mod M == k), then # Blocks in this picture are allowed to be filtered using NN loop filter; else # No blocks in this picture are allowed to be filtered using NN loop filter.

[0306] A sixty-second embodiment comprises the method of any of the thirty-nineth through sixty-first embodiments, wherein hierarchical layers are temporal sublayers and HIDs are TemporallDs.

[0307] A sixty-third embodiment comprises the method of any of the thirty-nineth through sixty- second embodiments, wherein hierarchical layers are layers (such as scalable or multiview layers) and HIDs are layer IDs.

[0308] A sixty-fourth embodiment comprises the method of any of the thirty-nineth through sixty- third embodiments, wherein hierarchical layers are not necessarily temporal sublayers or layers. For instance, multiple hierarchical layers with HIDs 0-5 could be within one temporal sublayer with TemporallD 0, or hierarchical layers with HIDs 4-5 could be within one temporal sublayer with TemporallD 3.

[0309] A sixty -fifth embodiment comprises the method of any of the thirty-nineth through sixtyfourth embodiments, wherein only pictures belonging to a specific hierarchical layer, e.g. HID 0, are allowed to be filtered using NN loop filter.

[0310] A sixty-sixth embodiment comprises the method of any of the thirty-nineth through sixtyfifth embodiments, wherein only pictures with a HID below a certain value, e.g. HIDs < 4, i.e. HIDs 0 - 3, are allowed to be filtered using NN loop filter.

[0311] A sixty-seventh embodiment comprises the method of any of the thirty-nineth through sixty-sixth embodiments, wherein one or more syntax elements in the bitstream are used to specify which HIDs that may be allowed to be filtered using NN loop filter. These one or more syntax elements may be signaled in a parameter set, such as a VPS, an SPS, a PPS or an APS, in a DCI, or in a header such as a picture header or a slice header.

[0312] A sixty-eighth embodiment comprises the method of any of the thirty-nineth through sixtyseventh embodiments, wherein different NN filters of different complexity are used, wherein pictures with a certain set of HIDs are allowed to be filtered using a first NN loop filter of a first complexity, and a second set of HIDs are allowed to be filtered using a second NN loop filter of a second complexity that is different from the first complexity. For instance, pictures with HID 0 may be allowed to be filtered using a first NN loop filter of a first complexity, while pictures with HIDs 1-3 may only be allowed to be filtered using a second NN loop filter of a second complexity that is lower than the first complexity, and pictures with HIDs 4-5 may not be allowed to be filtered using an NN loop filter at all.

[0313] A sixty -nineth embodiment comprises the method of any of the thirty -nineth through sixty eighth embodiments, wherein such determination of filter maybe done using the POC of a picture according to: if (POC mod 2 == 0) and if (POC mod 4 == 0), then # Encode this image with NN loop filter of first complexity allowed; else # Encode this image with NN loop filter of second complexity allowed; else # Encode this image without any NN loop filtering.

[0314] A seventieth embodiment comprises the method of any of the thirty -nineth through sixty - nineth embodiments, wherein only pictures that are (or may) be used for reference by other pictures are allowed to be filtered using NN loop filter.

[0315] A seventy-first embodiment comprises the method of any of the thirty-nineth through seventieth embodiments, wherein there is a picture enable syntax element or a slice enable syntax element that can specify that NN in-loop filtering is not done for the picture or slice.

[0316] A seventy-second embodiment comprises the method of any of the thirty-nineth through seventy-first embodiments, wherein the encoder may set the syntax element to disable NN inloop filtering for any of the following cases: disable filtering for pictures with an odd POC value; disable filtering for any picture with a POC value P not evenly divisible by 4 (i.e. do not filter if ((P mod 4) != 0); disable filtering for pictures belonging to one or more specific layer or temporal layer; disable filtering for picture belonging to the N highest hierarchical levels in a hierarchical bi-directional structure of picture referencing; and / or disable filtering for non-reference pictures

[0317] A seventy-third embodiment comprises an apparatus for decoding a picture comprising a neural network filtering process from a coded video bitstream, comprising: processing circuitry configured to perform any of the steps of any of first to thirty-eighth embodiments; and power supply circuitry configured to supply power to the processing circuitry.

[0318] A seventy-fourth embodiment comprises an apparatus for encoding a picture to a coded video bitstream, comprising: processing circuitry configured to perform any of the steps of any of the thirty-ninth to seventy-second embodiments; power supply circuitry configured to supply power to the processing circuitry.

[0319] A seventy -fifth embodiment comprises the apparatus of the seventy -third or seventy -fourth embodiment, wherein the apparatus comprises at least one of: a user equipment; a network node; a base station.

[0320] A seventy-sixth embodiment comprises a method (2000) for decoding one or more pictures from a coded video bitstream, the method comprising determining (2002) a maximum allowed complexity from one or more syntax elements in the coded video bitstream, wherein the maximum allowed complexity corresponds to a neural network filtering process; decoding (2004) the one or more pictures based on the maximum allowed complexity.

[0321] A seventy-seventh embodiment comprises the method of the seventy-sixth embodiment, wherein the maximum allowed complexity is determined for at least one of the following: one or more pictures, one or more portions of a picture, one or more sets of pictures, or one or more coded video sequences, CVS.

[0322] A seventy-eighth embodiment comprises method of any of the seventy-sixth through seventy-seventh embodiments, wherein the maximum allowed complexity is defined as one or more of: the maximum allowed MAC for NN filtering; the maximum allowed samples that use NN filtering; the maximum allowed percentage of samples that use NN filtering; the maximum allowed NN blocks or percentage of NN blocks that use NN filtering; the maximum allowed CTUs that use NN filtering; or a percentage of CTUs that use NN filtering.

[0323] A seventy-nineth embodiment comprises the method of any of the seventy-sixth through seventy-eighth embodiments, wherein the maximum allowed complexity is a constraint on the coded video bitstream.

[0324] An eightieth embodiment comprises the method of any of the any of the seventy-sixth through seventy-nineth embodiments, wherein decoding (2004) the one or more pictures based on the maximum allowed complexity, comprises selectively filtering the one or more pictures using a neural network filtering process.

[0325] An eighty-first embodiment comprises the method of any of the any of the seventy-sixth through eightieth embodiments, wherein decoding (2004) the one or more pictures based on the maximum allowed complexity comprises determining whether a current portion of a picture of the one or more pictures is filtered using a neural network filtering process in response to a comparison of a complexity value and the maximum allowed complexity.

[0326] An eighty-second embodiment comprises the method of the eighty-first embodiment, wherein determining whether a current portion of the picture is filtered using a neural network filtering process in response to a comparison of a complexity value and the maximum allowed complexity comprises decoding a first syntax element from the coded video bitstream.

[0327] An eighty-third embodiment comprises method of the eighty-first and eighty-second embodiments, wherein determining whether a current portion of the picture is filtered using a neural network filtering process is based on at least one of: he scanning order of the portions in the picture; the current portion of the picture consists of a block which is marked by a block importance mapping, BIM, tool; the boundary strength, BS, information of at least one block in the current portion of the picture; a block type of at least one block in the current portion of the picture; a quantization parameter, QP, value of at least one block in the current portion of the picture; a picture type of the picture containing the current portion of the picture; a temporal ID of the picture containing the current portion of the picture; and / or a layer ID of the picture containing the current portion of the picture.

[0328] An eighty-fourth embodiment comprises the method of any of the seventy-third through eighty -third embodiments, wherein the maximum allowed complexity is decoded from a parameter set comprising at least one of: a PPS; a SPS; a VPS; a header; a picture header; a slice header; a decoding capability information, DCI, NAL unit.

[0329] An eighty-fifth embodiment comprises the method of any of eighty-first through eightyfourth embodiments, wherein selectively filtering the one or more pictures is based on a picture order count, POC.

[0330] An eighty-sixth embodiment comprises the method of any of the seventh-nineth through eighty-fifth embodiments, wherein selectively filtering the one or more pictures is based on a hierarchical layer of the one or more pictures.

[0331] An eighty-seventh embodiment comprises the method of the eighty-fifth embodiment, wherein the hierarchical layer comprises at least one of: a temporal sublayer; a layer; a scalable layer; or a multiview layer.

[0332] An eighty-eighth embodiment comprises a method (2100) for encoding one or more pictures to a coded video bitstream, the method (2100) comprising: obtaining (2102) a maximum allowed complexity for a neural network filter; determining (2104), based on the maximum allowed complexity, one or more portions of each of the one or more pictures to encode using the neural network filter; filtering (2106) the one or more portions using the neural network filter, forming one or more filtered portions; and encoding (2108) each picture of the one or more pictures according to the one or more filtered portions and one or more remaining portions of the picture.

[0333] An eighty-nineth embodiment comprises the method of the eighty-eighth embodiment, further comprising encoding an indication of the maximum allowed complexity for the neural network filter to the coded video bitstream.

[0334] A ninetieth embodiment comprises the method of the eighty-eighth or eighty-nineth embodiments, wherein determining (2104) one or more portions of each picture of the one or more pictures to encode using the NN filter comprises calculating one or more performance gains of filtering using the neural network filter for at least one portion of each picture. A ninety-first embodiment comprises the method of any of the eighty-eighth through ninetieth embodiments, wherein determining (2104) one or more portions of each picture of the one or more pictures to encode using the NN filter is based on at least one of: the scanning order of the portions in the picture; the current portion of the picture consists of a block which is marked by a block importance mapping, BIM, tool; the boundary strength, BS, information of at least one block in the current portion of the picture; a block type of at least one block in the current portion of the picture; a quantization parameter, QP, value of at least one block in the current portion of the picture; a picture type of the picture containing the current portion of the picture; a temporal ID of the picture containing the current portion of the picture; and / or a layer ID of the picture containing the current portion of the picture.

[0335] A ninety-second embodiment comprises the method of any of the eighty-eighth through ninety-first embodiments, wherein the maximum allowed complexity is a constraint on the coded video bitstream.

[0336] A ninety-third embodiment comprises the method of any of the eighty-eighth through ninety-second embodiments, wherein the maximum allowed complexity is determined for at least one of the following: a picture, a portion of a picture, a set of pictures, or a coded video sequence, CVS.

[0337] A ninety-fourth embodiment comprises the method of any of the eighty-eighth through ninety-third embodiments, wherein the maximum allowed complexity is defined as one or more of: the maximum allowed MAC / pixel for NN filtering; the maximum allowed samples that use NN filtering; the maximum allowed percentage of samples that use NN filtering; the maximum allowed NN blocks or percentage of NN blocks that use NN filtering; the maximum allowed CTUs or percentage of CTUs that use NN filtering.

[0338] A ninety-fifth embodiment comprises the method of any of the eighty-eighth through ninety-fourth embodiments, wherein determining (2104) one or more portions of each picture of the one or more pictures to encode using the NN filter is further based on a picture order count, POC.

[0339] A ninety-sixth embodiment comprises the method of any of the eighty-eighth through ninety-fifth embodiments, wherein determining (2104) one or more portions of each picture of the one or more pictures to encode is based on a hierarchical layer of the one or more picture.

[0340] A ninety -seventh embodiment comprises the method of ninety-sixth embodiment, wherein the hierarchical layer comprises at least one of: a temporal sublayer; a layer; a scalable layer; or a multiview layer. A ninety-eighth embodiment comprises an apparatus (4300) for decoding a picture comprising a neural network filtering process from a coded video bitstream, comprising processing circuitry (4302) configured to perform any of the steps of any of the seventy-sixth through eightyseventh embodiments; and power supply circuitry (4308) configured to supply power to the processing circuitry.

[0341] A ninety-nineth embodiment comprises an apparatus (4300) for encoding a picture to a coded video bitstream, comprising processing circuitry (4302) configured to perform any of the steps of any of the eighty -eighth through ninety -seventh embodiments; and power supply circuitry (4308) configured to supply power to the processing circuitry.

[0342] A hundredth embodiment comprises the apparatus (4300) of the ninety-eighth or ninety- nineth embodiments, wherein the apparatus (4300) further comprises at least one of: a user equipment (3200); a network node (3300); a base station.

[0343] A one-hundred-and-first embodiment comprises an apparatus (4300) for decoding a picture comprising a neural network filtering process from a coded video bitstream, comprising processing circuitry (4302); and a memory (4310) storing instructions whereby the processing circuitry is operable to perform the steps of: determining a maximum allowed complexity from one or more syntax elements in the coded video bitstream, wherein the maximum allowed complexity corresponds to a neural network filtering process; decoding the one or more pictures based on the maximum allowed complexity.

[0344] A one-hundred-and-second embodiment comprises an apparatus (4300) for encoding a picture comprising a neural network filtering process from a coded video bitstream, comprising processing circuitry (4302); and a memory (4310) storing instructions whereby the processing circuitry is operable to perform the steps of: obtaining a maximum allowed complexity for a neural network filter; determining, based on the maximum allowed complexity, one or more portions of each of the one or more pictures to encode using the neural network filter; filtering the one or more portions using the neural network filter, forming one or more filtered portions; and encoding each picture of the one or more pictures according to the one or more filtered portions and one or more remaining portions of the picture.

Claims

73CLAIMS1. A method (2000) for decoding one or more pictures from a coded video bitstream, the method (2000) comprising: determining (2002) a maximum allowed complexity from one or more syntax elements in the coded video bitstream, wherein the maximum allowed complexity corresponds to a neural network filtering process; decoding (2004) the one or more pictures based on the maximum allowed complexity.

2. The method of claim 1, wherein the maximum allowed complexity is determined for at least one of the following: one or more pictures, one or more portions of a picture, one or more sets of pictures, or one or more coded video sequences, CVS.

3. The method of any of the previous claims, wherein the maximum allowed complexity is defined as one or more of: the maximum allowed MAC for NN filtering; the maximum allowed samples that use NN filtering; the maximum allowed percentage of samples that use NN filtering; the maximum allowed NN blocks or percentage of NN blocks that use NN filtering; the maximum allowed CTUs that use NN filtering; or a percentage of CTUs that use NN filtering.

4. The method of any of claims 1 -3, wherein the maximum allowed complexity is a constraint on the coded video bitstream.

5. The method of any of the previous claims, wherein decoding (2004) the one or more pictures based on the maximum allowed complexity, comprises selectively filtering the one or more pictures using a neural network filtering process.

6. The method of any of the previous claims, wherein decoding (2004) the one or more pictures based on the maximum allowed complexity comprises determining whether a current portion of a picture of the one or more pictures is filtered using a neural network filtering process in response to a comparison of a complexity value and the maximum allowed complexity.

747. The method of claim 6, wherein determining whether a current portion of the picture is filtered using a neural network filtering process in response to a comparison of a complexity value and the maximum allowed complexity comprises decoding a first syntax element from the coded video bitstream.

8. The method of claim 6 or 7 wherein determining whether a current portion of the picture is filtered using a neural network filtering process is based on at least one of: he scanning order of the portions in the picture; the current portion of the picture consists of a block which is marked by a block importance mapping, BIM, tool; the boundary strength, BS, information of at least one block in the current portion of the picture; a block type of at least one block in the current portion of the picture; a quantization parameter, QP, value of at least one block in the current portion of the picture; a picture type of the picture containing the current portion of the picture; a temporal ID of the picture containing the current portion of the picture; and / or a layer ID of the picture containing the current portion of the picture.

9. The method of any of the previous claims, wherein the maximum allowed complexity is decoded from a parameter set comprising at least one of: a PPS; a SPS; a VPS; a header; a picture header; a slice header; a decoding capability information, DCI, NAL unit.

10. The method of any of claims 6-9, wherein selectively filtering the one or more pictures is based on a picture order count, POC.

11. The method of any of claims 4-10, wherein selectively filtering the one or more pictures is based on a hierarchical layer of the one or more pictures.

12. The method of claim 11 , wherein the hierarchical layer comprises at least one of: a temporal sublayer; a layer; a scalable layer; or a multiview layer.

13. A method (2100) for encoding one or more pictures to a coded video bitstream, the method (2100) comprising: obtaining (2102) a maximum allowed complexity for a neural network filter; determining (2104), based on the maximum allowed complexity, one or more portions of each of the one or more pictures to encode using the neural network filter;75 filtering (2106) the one or more portions using the neural network filter, forming one or more filtered portions; and encoding (2108) each picture of the one or more pictures according to the one or more filtered portions and one or more remaining portions of the picture.

14. The method of claim 13, further comprising encoding an indication of the maximum allowed complexity for the neural network filter to the coded video bitstream.

15. The method of claim 13 or 14, wherein determining (2104) one or more portions of each picture of the one or more pictures to encode using the NN filter comprises calculating one or more performance gains of filtering using the neural network filter for at least one portion of each picture.

16. The method of any of claims 13 through 15 wherein determining (2104) one or more portions of each picture of the one or more pictures to encode using the NN filter is based on at least one of: the scanning order of the portions in the picture; the current portion of the picture consists of a block which is marked by a block importance mapping, BIM, tool; the boundary strength, BS, information of at least one block in the current portion of the picture; a block type of at least one block in the current portion of the picture; a quantization parameter, QP, value of at least one block in the current portion of the picture; a picture type of the picture containing the current portion of the picture; a temporal ID of the picture containing the current portion of the picture; and / or a layer ID of the picture containing the current portion of the picture.

17. The method of any of claims 13-16, wherein the maximum allowed complexity is a constraint on the coded video bitstream.

18. The method of any of claims 13-17, wherein the maximum allowed complexity is determined for at least one of the following: a picture, a portion of a picture, a set of pictures, or a coded video sequence, CVS.

19. The method of any of claims 13-18, wherein the maximum allowed complexity is defined as one or more of: the maximum allowed MAC / pixel for NN filtering; the maximum allowed samples that use NN filtering; the maximum allowed percentage of samples that use NN filtering;76 the maximum allowed NN blocks or percentage of NN blocks that use NN filtering; the maximum allowed CTUs or percentage of CTUs that use NN filtering.

20. The method of any of claims 13-19, wherein determining (2104) one or more portions of each picture of the one or more pictures to encode using the NN filter is further based on a picture order count, POC.

21. The method of any of claims 13-20, wherein determining (2104) one or more portions of each picture of the one or more pictures to encode is based on a hierarchical layer of the one or more picture.

22. The method of claim 21 , wherein the hierarchical layer comprises at least one of: a temporal sublayer; a layer; a scalable layer; or a multiview layer.

23. An apparatus (4300) for decoding a picture comprising a neural network filtering process from a coded video bitstream, comprising: processing circuitry (4302) configured to perform any of the steps of any of claims 1-12; and power supply circuitry (4308) configured to supply power to the processing circuitry.

24. An apparatus (4300) for encoding a picture to a coded video bitstream, comprising: processing circuitry (4302) configured to perform any of the steps of any of claims 13-22; and power supply circuitry (4308) configured to supply power to the processing circuitry.

25. The apparatus (4300) of claims 23 or 24, wherein the apparatus (4300) further comprises at least one of: a user equipment (3200); a network node (3300); a base station.

26. An apparatus (4300) for decoding a picture comprising a neural network filtering process from a coded video bitstream, comprising: processing circuitry (4302); and a memory (4310) storing instructions whereby the processing circuitry is operable to perform the steps of:determining a maximum allowed complexity from one or more syntax elements in the coded video bitstream, wherein the maximum allowed complexity corresponds to a neural network filtering process; decoding the one or more pictures based on the maximum allowed complexity.

27. An apparatus (4300) for encoding a picture comprising a neural network filtering process from a coded video bitstream, comprising: processing circuitry (4302); and a memory (4310) storing instructions whereby the processing circuitry is operable to perform the steps of: obtaining a maximum allowed complexity for a neural network filter; determining, based on the maximum allowed complexity, one or more portions of each of the one or more pictures to encode using the neural network filter; filtering the one or more portions using the neural network filter, forming one or more filtered portions; and encoding each picture of the one or more pictures according to the one or more filtered portions and one or more remaining portions of the picture.

Images