Scale indicator input to multi-scale probability model

Abstract

The embodiments concern a method comprising decoding, from or along a bitstream, information related to elements of a latent representation of at least one image for a resolution level; determining whether to use a scale indicator that indicates information related to the resolution level as input to a distribution predictor to predict parameters of a probability distribution for the elements of the latent representation of the at least one image for the resolution level; and using the distribution predictor to predict the parameters of the probability distribution for the elements of the latent representation of the at least one image for the resolution level; wherein the scale indicator that indicates information related to the resolution level is used as input to the distribution predictor to predict the parameters of the probability distribution for the elements of the latent representation of the at least one image for the resolution level, in response to determining to use the scale indicator The embodiments also concern technical equipment for implementing the method.

Description

SCALE INDICATOR INPUT TO MULTI-SCALE PROBABILITY MODELTECHNICAL FIELD

[0001] The examples and non-limiting embodiments relate generally to scale indicator input to a multi-scale probability model.BACKGROUND

[0002] It is known to process and render video in a multimedia system.BRIEF DESCRIPTION OF THE DRAWINGS

[0003] The foregoing embodiments and other features are explained in the following description, taken in connection with the accompanying drawings, wherein:

[0004] FIG. 1 illustrates examples of how NNs can function as components of a codec’s pipeline with components having parameters that are typically not learned from data.

[0005] FIG. 2 shows an example video coding pipeline where most components are replaced with NNs.

[0006] FIG. 3 shows an example where the analysis network is an encoder NN, and the synthesis network is the decoder NN.

[0007] FIG. 4 shows an example neural network based end-to-end learned video coding system.

[0008] FIG. 5 shows an example architecture of a multi-scale progressive probability model.

[0009] FIG. 6 shows an example architecture of a prediction model in a multi-scale progressive probability model.

[0010] FIG 7 shows an example implementation of a distribution predictor.

[0011] FIG. 8 shows an example implementation of a scale indicator as an additional input to the distribution predictor.

[0012] FIG. 9 shows an encoder according to an embodiment.

[0013] FIG. 10 shows a decoder according to an embodiment.

[0014] FIG. 11 is a block diagram illustrating a system in accordance with an example.

[0015] FIG. 12 is an example apparatus configured to implement the examples described herein.

[0016] FIG. 13 shows a representation of an example of non-volatile memory media used to store instructions that implement the examples described herein

[0017] FIG. 14 is an example method based on the examples described herein.

[0018] FIG. 15 is an example method based on the examples described herein.DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0019] Fundamentals of neural networks

[0020] A neural network (NN) is a computation graph consisting of several layers of computation. Each layer consists of one or more units, where each unit performs an elementary computation. A unit is connected to one or more other units, and the connection may have associated with a weight. The weight may be used for scaling the signal passing through the associated connection. Weights are learnable parameters, i.e., values which can be learned from training data. There may be other learnable parameters, such as those of batch-normalization layers.

[0021] Two of the most widely used architectures for neural networks are feedforward and recurrent architectures. Feed-forward neural networks are such that thereis no feedback loop: each layer takes input from one or more of the layers before and provides its output as the input for one or more of the subsequent layers. Also, units inside a certain layer take input from units in one or more of preceding layers, and provide output to one or more of following layers.

[0022] Initial layers (those close to the input data) extract semantically low-level features such as edges and textures in images, and intermediate and final layers extract more high-level features. After the feature extraction layers there may be one or more layers performing a certain task, such as classification, semantic segmentation, object detection, denoising, style transfer, super-resolution, etc. In recurrent neural nets, there is a feedback loop, so that the network becomes stateful, i.e., it is able to memorize information or a state.

[0023] Neural networks are being utilized in an ever-increasing number of applications for many different types of device, such as mobile phones. Examples include image and video analysis and processing, social media data analysis, device usage data analysis, etc.

[0024] The most important property of neural nets (and other machine learning tools) is that they are able to learn properties from input data, either in supervised way or in unsupervised way. Such learning is a result of a training algorithm, or of a meta-level neural network providing the training signal.

[0025] In general, the training algorithm consists of changing some properties of the neural network so that its output is as close as possible to a desired output. For example, in the case of classification of objects in images, the output of the neural network can be used to derive a class or category index which indicates the class or category that the object in the input image belongs to. Training usually happens by minimizing or decreasing the output’s error, also referred to as the loss. Examples of losses are mean squared error, cross-entropy, etc. In recent deep learning techniques, training is an iterative process, where at each iteration the algorithm modifies the weights of the neural net to make a gradual improvement of the network’s output, i.e., to gradually decrease the loss.

[0026] The terms “model”, “neural network”, “neural net” and “network” are used interchangeably herein, and also the weights of neural networks are sometimes referred to as learnable parameters or simply as parameters.

[0027] Training a neural network is an optimization process, but the final goal is different from the typical goal of optimization. In optimization, the only goal is to minimize a function. In machine learning, the goal of the optimization or training process is to make the model learn the properties of the data distribution from a limited training dataset. In other words, the goal is to learn to use a limited training dataset in order to learn to generalize to previously unseen data, i.e., data which was not used for training the model. This is usually referred to as generalization. In practice, data is usually split into at least two sets, the training set and the validation set. The training set is used for training the network, i.e., to modify its learnable parameters in order to minimize the loss. The validation set is used for checking the performance of the network on data which was not used to minimize the loss, as an indication of the final performance of the model. In particular, the errors on the training set and on the validation set are monitored during the training process to understand the following things:

[0028] - If the network is learning at all - in this case, the training set error should decrease, otherwise the model is in the regime of underfitting.

[0029] - If the network is learning to generalize - in this case, also the validation set error needs to decrease and to be not too much higher than the training set error. If the training set error is low, but the validation set error is much higher than the training set error, or it does not decrease, or it even increases, the model is in the regime of overfitting. This means that the model has just memorized the training set’s properties and performs well only on that set, but performs poorly on a set not used for tuning its parameters.

[0030] Lately, neural networks have been used for compressing and de-compressing data such as images, i.e., in an image codec. The most widely used architecture for realizing one component of an image codec is the auto-encoder, which is a neuralnetwork consisting of two parts: a neural encoder and a neural decoder (these are referred to simply as encoder and decoder herein, even though algorithms which are learned from data instead of being tuned by hand may be referred to). The encoder takes as input an image and produces a code which requires less bits than the input image. This code may be obtained by applying a binarization or quantization process to the output of the encoder. The decoder takes in this code and reconstructs the image which was input to the encoder.

[0031] Such encoder and decoder are usually trained to minimize a combination of bitrate and distortion, where the distortion may be based on one or more of the following metrics: Mean Squared Error (MSE), Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), or similar. These distortion metrics are meant to be correlated to the human visual perception quality, so that minimizing or maximizing one or more of these distortion metrics results into improving the visual quality of the decoded image as perceived by humans.

[0032] Fundamentals of video / image coding

[0033] Video codec consists of an encoder that transforms the input video into a compressed representation suited for storage / transmission and a decoder that can decompress the compressed video representation back into a viewable form. Typically encoder discards some information in the original video sequence in order to represent the video in a more compact form (that is, at lower bitrate).

[0034] Typical hybrid video codecs, for example ITU-T H.263 and H.264, encode the video information in two phases. Firstly pixel values in a certain picture area (or “block”) are predicted for example by motion compensation means (finding and indicating an area in one of the previously coded video frames that corresponds closely to the block being coded) or by spatial means (using the pixel values around the block to be coded in a specified manner). Secondly the prediction error, i.e. the difference between the predicted block of pixels and the original block of pixels, is coded. This is typically done by transforming the difference in pixel values using a specified transform (e.g. Discrete Cosine Transform (DCT) or a variant of it), quantizing the coefficientsand entropy coding the quantized coefficients. By varying the fidelity of the quantization process, encoder can control the balance between the accuracy of the pixel representation (picture quality) and size of the resulting coded video representation (file size or transmission bitrate).

[0035] Inter prediction, which may also be referred to as temporal prediction, motion compensation, or motion-compensated prediction, exploits temporal redundancy. In inter prediction the sources of prediction are previously decoded pictures.

[0036] Intra prediction utilizes the fact that adjacent pixels within the same picture are likely to be correlated. Intra prediction can be performed in spatial or transform domain, i.e., either sample values or transform coefficients can be predicted. Intra prediction is typically exploited in intra coding, where no inter prediction is applied.

[0037] One outcome of the coding procedure is a set of coding parameters, such as motion vectors and quantized transform coefficients. Many parameters can be entropy- coded more efficiently if they are predicted first from spatially or temporally neighboring parameters. For example, a motion vector may be predicted from spatially adjacent motion vectors and only the difference relative to the motion vector predictor may be coded. Prediction of coding parameters and intra prediction may be collectively referred to as in-picture prediction.

[0038] The decoder reconstructs the output video by applying prediction means similar to the encoder to form a predicted representation of the pixel blocks (using the motion or spatial information created by the encoder and stored in the compressed representation) and prediction error decoding (inverse operation of the prediction error coding recovering the quantized prediction error signal in spatial pixel domain). After applying prediction and prediction error decoding means the decoder sums up the prediction and prediction error signals (pixel values) to form the output video frame. The decoder (and encoder) can also apply additional filtering means to improve the quality of the output video before passing it for display and / or storing it as prediction reference for the forthcoming frames in the video sequence.

[0039] In typical video codecs the motion information is indicated with motion vectors associated with each motion compensated image block. Each of these motion vectors represents the displacement of the image block in the picture to be coded (in the encoder side) or decoded (in the decoder side) and the prediction source block in one of the previously coded or decoded pictures. In order to represent motion vectors efficiently those are typically coded differentially with respect to block specific predicted motion vectors. In typical video codecs the predicted motion vectors are created in a predefined way, for example calculating the median of the encoded or decoded motion vectors of the adjacent blocks. Another way to create motion vector predictions is to generate a list of candidate predictions from adjacent blocks and / or colocated blocks in temporal reference pictures and signaling the chosen candidate as the motion vector predictor. In addition to predicting the motion vector values, the reference index of previously coded / decoded picture can be predicted. The reference index is typically predicted from adjacent blocks and / or or co-located blocks in temporal reference picture. Moreover, typical high efficiency video codecs employ an additional motion information coding / decoding mechanism, often called merging / merge mode, where all the motion field information, which includes motion vector and corresponding reference picture index for each available reference picture list, is predicted and used without any modification / correction. Similarly, predicting the motion field information is carried out using the motion field information of adjacent blocks and / or co-located blocks in temporal reference pictures and the used motion field information is signaled among a list of motion field candidate list filled with motion field information of available adjacent / co-located blocks.

[0040] In typical video codecs the prediction residual after motion compensation is first transformed with a transform kernel (like DCT) and then coded. The reason for this is that often there still exists some correlation among the residual and transform can in many cases help reduce this correlation and provide more efficient coding.

[0041] Typical video encoders utilize Lagrangian cost functions to find optimal coding modes, e.g. the desired Macroblock mode and associated motion vectors. This kind of cost function uses a weighting factor to tie together the (exact or estimated)image distortion due to lossy coding methods and the (exact or estimated) amount of information that is required to represent the pixel values in an image area:C = D + XR

[0042] In the above equation, C is the Lagrangian cost to be minimized, D is the image distortion (e.g. Mean Squared Error) with the mode and motion vectors considered, and R the number of bits needed to represent the required data to reconstruct the image block in the decoder (including the amount of data to represent the candidate motion vectors).

[0043] Video coding specifications may enable the use of supplemental enhancement information (SEI) messages or alike. Some video coding specifications include SEI NAL units, and some video coding specifications contain both prefix SEI NAL units and suffix SEI NAL units, where the former type can start a picture unit or alike and the latter type can end a picture unit or alike. An SEI NAL unit contains one or more SEI messages, which are not required for the decoding of output pictures but may assist in related processes, such as picture output timing, post-processing of decoded pictures, rendering, error detection, error concealment, and resource reservation. Several SEI messages are specified in H.264 / AVC, H.265 / HEVC, H.266 / VVC, and H.274 / VSEI standards, and the user data SEI messages enable organizations and companies to specify SEI messages for their own use. The standards may contain the syntax and semantics for the specified SEI messages but a process for handling the messages in the recipient might not be defined. Consequently, encoders may be required to follow the standard specifying a SEI message when they create SEI message(s), and decoders might not be required to process SEI messages for output order conformance. One of the reasons to include the syntax and semantics of SEI messages in standards is to allow different system specifications to interpret the supplemental information identically and hence interoperate. It is intended that system specifications can require the use of particular SEI messages both in the encoding end and in the decoding end, and additionally the process for handling particular SEI messages in the recipient can be specified.

[0044] A design principle has been followed for SEI message specifications: the SEI messages are generally not extended in future amendments or versions of the standard.

[0045] Background on filters in video codecs

[0046] Image and video codecs generally use a set of filters to enhance the visual quality of the predicted visual content and can be applied either in-loop or out-of-loop, or both. In the case of in-loop filters, the filter applied on one block in the currently- encoded frame affects the encoding of another block in the same frame and / or in another frame which is predicted from the current frame. An in-loop filter can affect the bitrate and / or the visual quality. In fact, an enhanced block causes a smaller residual (difference between original block and predicted-and-filtered block), thus requiring less bits to be encoded. An out-of-the loop filter is applied on a frame after it has been reconstructed, the filtered visual content won't be as a source for prediction, and thus it may only impact the visual quality of the frames that are output by the decoder.

[0047] Background information on neural network based image / video coding

[0048] Recently, neural networks (NNs) have been used in the context of image and video compression, by following mainly two approaches.

[0049] In one approach, NNs are used to replace one or more of the components of a codec such as VVC / H.266 whose components and their parameters are typically not learned from data. Examples of components that may be implemented as neural networks are: additional in-loop filter, for example by having the NN as an additional in-loop filter with respect to loop filters with components having parameters that are typically not learned from data, single in-loop filter, for example by having the NN replacing all in-loop filters with components having parameters that are typically not learned from data, intra-frame prediction, inter-frame prediction, transform and / or inverse transform, probability model for the arithmetic codec, etc.

[0050] FIG. 1 illustrates examples of how NNs can function as components of a codec’s pipeline with components having parameters that are typically not learned from data. FIG. 1 shows a codec 100. As shown in FIG. 1, luma intra prediction (1) can bea deep intra prediction net, chroma intra prediction (2) can be a deep cross component prediction net, intra prediction (3) can be a deep intra prediction net, entropy coding (4) can be a deep probability estimation net, T / Q (5) can be a deep transform net, in-loop filter (6) can be a deep loop filter, post filter (7) can be a deep post filter, resolution adaptation (8) can be deep up- / down-sampling coding, and encoder control (9) can be a deep encoder option.

[0051] In another approach, commonly referred to as “end-to-end learned compression”, NNs are used as the main components of the image / video codecs. In this second approach, there are two main options:

[0052] Option 1 : re-use the video coding pipeline with components having parameters that are typically not learned from data, but replace most or all the components with NNs, as shown in FIG. 2.

[0053] Referring to FIG. 2, it illustrates an example of modified video coding pipeline 200 based on neural networks. An example of neural network may include, but is not limited, a compressed representation of a neural network. FIG. 1 is shown to include following components:

[0054] - A neural transform block or circuit 202: this block or circuit transforms the output of a summation / subtraction operation 203 to a new representation of that data, which may have lower entropy and thus be more compressible. Input to the summation / subtraction operation 203 is input data 201.

[0055] - A quantization block or circuit 204: this block or circuit quantizes an input data 101 to a smaller set of possible values.

[0056] - An inverse transform and inverse quantization blocks or circuits 206. These blocks or circuits perform the inverse or approximately inverse operation of the transform and the quantization, respectively.

[0057] - An encoder parameter control block or circuit 208. This block or circuit may control and optimize some or all the parameters of the encoding process, such asparameters of one or more of the encoding blocks or circuits.

[0058] - An entropy coding block or circuit 210. This block or circuit may perform lossless coding, for example, based on entropy. One popular entropy coding technique is arithmetic coding.

[0059] - A neural intra-codec block or circuit 212. This block or circuit may be an image compression and decompression block or circuit, which may be used to encode and decode an intra frame. An encoder 214 may be an encoder block or circuit, such as the neural encoder part of an auto-encoder neural network. A decoder 216 may be a decoder block or circuit, such as the neural decoder part of an auto-encoder neural network. An intra-coding block or circuit 218 may be a block or circuit performing some intermediate steps between encoder and decoder, such as quantization, entropy encoding, entropy decoding, and / or inverse quantization.

[0060] - A deep loop filter block or circuit 220. This block or circuit performs filtering of reconstructed data, in order to enhance it.

[0061] - A decode picture buffer block or circuit 222. This block or circuit is a memory buffer, keeping the decoded frame, for example, reconstructed frames 224 and enhanced reference frames 226 to be used for inter prediction.

[0062] - An inter-prediction block or circuit 228. This block or circuit performs interframe prediction, for example, predicts from frames, for example, frames 232, which are temporally nearby. An ME / MC 230 performs motion estimation and / or motion compensation, which are two key operations to be performed when performing interframe prediction. ME / MC stands for motion estimation / motion compensation. Within inter prediction 228, one or more neural networks may process the output of the ME / MC 230 to generate inter predicted frames 232.

[0063] In this example (Option 1), the forward and inverse transforms were replaced with two neural networks. Also, the loop filter is a neural network.

[0064] Option 2: re-design the whole pipeline, as follows: encoder NN: performs anon-linear transform, quantization and lossless encoding of the encoder NN’s output, lossless decoding and dequantization, decoder NN: performs a non-linear inverse transform.

[0065] The encoder and decoder NNs may be parts of a neural auto-encoder architecture.

[0066] Refer to FIG. 3, where the Analysis Network 302 is the Encoder NN, and the Synthesis Network 304 is the Decoder NN.

[0067] In this Option 2, the input data 301 is analyzed by the Encoder NN (Analysis Network) 302, which outputs a new representation of that input data. The new representation may be more compressible. This new representation may then be quantized (by Quantization 306) to a discrete number of values. The quantized data is then lossless encoded, for example by an arithmetic encoder 308, thus obtaining a bitstream 310. On the decoding side, the bitstream is first lossless decoded, for example by using an arithmetic decoder 312. The lossless decoded data is dequantized and then input to the Decoder NN (Synthesis Network) 304. The output is the reconstructed or decoded data 314.

[0068] In case of lossy compression, the lossy steps may comprise the Encoder NN 302 and / or the quantization 306.

[0069] In order to train this system, a training objective function (also called “training loss”) is typically utilized, which usually comprises one or more terms, or loss terms, or simply losses. In one example, the training loss comprises a reconstruction loss term and a rate loss term. The reconstruction loss encourages the system to decode data that is similar to the input data, according to some similarity metric. Examples of reconstruction losses are:

[0070] - Mean squared error (MSE)

[0071] - Multi-scale structural similarity (MS-SSIM)

[0072] - Losses derived from the use of a pretrained neural network. For example,error(fl, f2), where fl and f2 are the features extracted by a pretrained neural network for the input data and the decoded data, respectively, and error() is an error or distance function, such as LI norm or L2 norm.

[0073] - Losses derived from the use of a neural network that is trained simultaneously with the end-to-end learned codec. For example, adversarial loss can be used, which is the loss provided by a discriminator neural network that is trained adversarially with respect to the codec, following the settings in the context of Generative Adversarial Networks (GANs) and their variants.

[0074] The rate loss encourages the system to compress the output of the encoding stage, such as the output of the arithmetic encoder. Here compress means reduce the number of bits output by the encoding stage.

[0075] When an entropy-based lossless encoder is used, such as an arithmetic encoder, the rate loss typically encourages the output of the Encoder NN to have low entropy. Example of rate losses are the following:

[0076] - A differentiable estimate of the entropy.

[0077] - A sparsification loss, i.e., a loss that encourages the output of the Encoder NN or the output of the quantization to have many zeros. Examples are L0 norm, LI norm, LI norm divided by L2 norm.

[0078] - A cross-entropy loss applied to the output of a probability model, where the probability model may be a NN used to estimate the probability of the next symbol to be encoded by an arithmetic encoder.

[0079] One or more of reconstruction losses may be used, and one or more of the rate losses may be used, as a weighted sum. Typically, the different loss terms are weighted using different weights, and these weights determine how the final system performs in terms of rate-distortion loss. For example, if more weight is given to the reconstruction losses with respect to the rate losses, the system may learn to compress less but to reconstruct with higher accuracy (as measured by a metric that correlates with thereconstruction losses). These weights are usually considered to be hyper-parameters of the training session, and may be set manually by the person designing the training session, or automatically for example by grid search or by using additional neural networks.

[0080] It is to be understood that even in end-to-end learned approaches, there may be components which are not learned from data, such as the arithmetic codec.

[0081] Further information on neural network-based end-to-end learned video coding

[0082] As shown in FIG. 4, a typical neural network-based end-to-end learned video coding system contains encoder 404, quantizer 406, probability model 410, entropy codec (for example arithmetic encoder / decoder that may include arithmetic encoder 408 and arithmetic decoder 414), dequantizer 416, and decoder 418. The encoder 404 and decoder 418 are typically two neural networks, or mainly comprise neural network components. The probability model 410 may also comprise mainly neural network components. Quantizer 406, dequantizer 416 and entropy codec (408, 414) are typically not based on neural network components, but they may also comprise neural network components, potentially.

[0083] On encoder side, the encoder component 404 takes a video as 402 and converts the video from its original signal space into a latent representation that may comprise a more compressible representation of the input. In the case of an input image, the latent representation may be a 3 -dimensional tensor, where two dimensions represent the vertical and horizontal spatial dimensions, and the third dimension represent the “channels” which contain information at that specific location. If the input image is a 128x128x3 RGB image (with horizontal size of 128 pixels, vertical size of 128 pixels, and 3 channels for the Red, Green, Blue color components), and if the encoder downsamples the input tensor by 2 and expands the channel dimension to 32 channels, then the latent representation is a tensor of dimensions (or “shape”) 64x64x32 (i.e., with horizontal size of 64 elements, vertical size of 64 elements, and 32 channels). The order of the different dimensions may differ depending on the convention which isused; in some cases, for the input image, the channel dimension may be the first dimension, so for the above example, the shape of the input tensor may be represented as 3x128x128, instead of 128x128x3. In the case of an input video (instead of just an input image), another dimension in the input tensor may be used to represent temporal information. The quantizer component 406 quantizes the latent representation into discrete values given a predefined set of quantization levels. Probability model 410 and arithmetic codec component (408, 414) work together to perform lossless compression for the quantized latent representation and generate bitstreams to be sent to the decoder side. Given a symbol to be encoded into the bitstream, the probability model estimates the probability distribution of all possible values for that symbol based on a context that is constructed from available information at the current encoding / decoding state, such as the data that has already been encoded / decoded. Then, the arithmetic encoder 408 encodes the input symbols to bitstream 412 using the estimated probability distributions.

[0084] On the decoder side, opposite operations are performed. The arithmetic decoder 414 and the probability model 410 first decode symbols from the bitstream 412 to recover the quantized latent representation. Then the dequantizer 416 reconstructs the latent representation in continuous values and pass it to decoder 418 to recover the input video / image by generating recovered input image 420. The probability model 410 in this system is shared between the encoding and decoding systems. In practice, this means that a copy of the probability model 410 is used at encoder side, and another exact copy is used at decoder side.

[0085] In this system, the encoder 404, probability model 410, and decoder 418 are normally based on deep neural networks. The system is trained in an end-to-end manner by minimizing the following rate-distortion loss function:L = D + AR,

[0086] In the above equation, D is the distortion loss term, R is the rate loss term, and A is the weight that controls the balance between the two losses. The distortion loss term may be the mean square error (MSE), structure similarity (SSIM) or other metricsthat evaluate the quality of the reconstructed video. Multiple distortion losses may be used and integrated into D, such as a weighted sum of MSE and SSIM. The rate loss term is normally the estimated entropy of the quantized latent representation, which indicates the number of bits necessary to represent the encoded symbols, for example, bits-per-pixel (bpp).

[0087] For lossless video / image compression, the system contains only the probability model 410 and arithmetic encoder 408 and arithmetic decoder 414. The system loss function contains only the rate loss, since the distortion loss is always zero (i.e., no loss of information).

[0088] Background information on the multi-scale progressive probability model

[0089] A probability model may be used in an end-to-end learned codec to estimate the probability distribution of the elements in a latent tensor, which is the output of an encoder. The estimated probability distribution may be used by an arithmetic encoder to encode the latent tensor into a bitstream at the encoding stage, or by an arithmetic decoder to decode the latent tensor from the bitstream at the decoding stage. For lossless image and video compression, the probability model estimates the probability distribution of the elements in the input image or video for the arithmetic encoder to encode the input image or video at the encoding stage; at the decoding stage, the probability model estimates the probability distribution of the elements for the arithmetic decoder to decode the output image or video. In this disclosure, the term latent tensor or latent representation may also refer to the input image or video in a lossless image or video compression system.

[0090] A multi-scale progressive probability model may partition the elements in a latent tensor into multiple groups. The elements in one group may be processed in parallel and the groups may be processed sequentially (e.g. next immediately after a previous iteration and where items of future iterations are subsequent to items of previous iterations). FIG. 5 shows the architecture of a multi-scale progressive probability model 500 at the encoding stage. Input latent tensor x(0)(502) is firstdownsampled into a certain number of low-resolution representations, e.g. x (504), x^ (506). The downsampling operation may use the nearest neighborhood, bilinear, or bicubic algorithm. If the downsampling algorithm is not the nearest neighborhood method, extra information may be transferred from the encoder to the decoder to recover round-off error due to the downsampling operation. The probability distribution of the elements in the representation at the lowest resolution, i.e. x(2J' (506) in FIG. 5, may be modeled as identically and independently distributed with a Gaussian distribution model, a uniform distribution model, or a mixture of probability distribution models. The probability of elements in the latent tensors in resolution levels other than the lowest one (including x (504) and x (502)) may be modeled by a conditional distribution model (whose parameters are estimated by a prediction model), where the conditioning information (also referred to as context) may comprise the representation at lower resolution levels. In FIG. 5, z (including z (508) and z^ (510)) is an auxiliary output from the prediction model (such as prediction model 520) at resolution level i where i = 1,2 , that may be used by the prediction model (such as prediction model 530) at the next resolution level (i — 1) as an extra input, z^ (510) may be a zero tensor, p^ (including p^ (514) and p® (512)) is the estimated parameters of the (conditional) probability distribution model for elements at resolution level i.

[0091] As shown in FIG. 5 the distribution predictor 602 is part of prediction model 520 and prediction model 530, where the distribution predictor 602 within prediction model 520 is used to determine p^ (514), and the distribution predictor 602 within prediction model 530 is used to determine p® (512).

[0092] At the decoding stage, the latent tensor at the lowest resolution, i.e. x^ (506) in FIG. 5 may be first decoded from the bitstream using the predefined probability distribution model. A multi-scale progressive probability model may use the elements in the latent tensor at resolution level i, e.g., x^las the context to estimate the parameters of the distribution model for the elements in the latent tensor at a higher resolution level, e.g.The estimated probability distribution may be used by thearithmetic decoder to decode the elements in the bitstream to obtain the latent tensor at higher resolution levelThe procedure may repeat until all elements in the latent tensor at the highest resolution level,(502) are decoded.

[0093] The prediction models at different resolution levels may share the weights or a subset of the weights. In one example, the prediction models at different resolutions are the same or substantially the same.

[0094] To further improve the accuracy of the probability distribution estimation, the elements in the latent tensor at a resolution level may be further partitioned into several groups. The groups may be processed sequentially. The elements in a group are modeled by independent conditional distribution models using the elements that have already been processed as the context. I.e., the elements of the latent tensor at a resolution level are processed in steps, where the elements in a group associated to a step are processed in parallel. The architecture of the prediction model is shown in FIG. 6.

[0095] Letbe the number of groups into which the elements in the latent representation at resolution level i are partitioned. FIG. 6 shows the prediction model at resolution level i and step j, where jThe distribution predictor 602 predicts the parameters of the probability distribution (604) for the elements in the latent representation at resolution level i and step(606) is the auxiliary input for the distribution predictor 602. For the first step, zb,i) =z(t+1), where z^+i) jsthe auxiliary output from the resolution level i + 1, i.e., z^t+1^ =z(t+1<w(l+1))(608) is a tensor that contains the true values for the elements that have already been processed and the predicted values of the elements that have not been processed at step j. x(l’Vjis derived by upsampling(610) is a binary-valued mask tensor with the same shape of x^ (608) indicating the positions of the elements in x1’^ (608) that have the true values,(604) is the estimated parameters of the probability distribution for the elements in group j at resolution level i. At the encoding stage, after(604) is calculated, the tensor updater (612) component may update the elements in group j withthe corresponding true values in x (613) to generate x(l’-i+Vj(614), and the mask updater component (616) may update the mask tensor m^'7^ (610) accordingly to generate m^,7 +1^ (618). At the last step, i.e., j = N^llet z^ = Z^1,NW

[0096] At the decoding stage, the calculated(604) is used to decode the corresponding elements in the bitstream. After a group of elements is decoded, the corresponding values are updated in x1’^ (608) to generate x^J+i) (614) and mask tensor m^'7^ (610) is updated accordingly to generate m(7'7 +1) (618). The prediction model repeats this operation in N steps until all elements at the resolution level i are processed. As further shown in FIG. 6, the distribution predictor 602 is used to generate z^ 620, or an auxiliary output for group j + 1 immediately after group j.

[0097] In a multi-scale probability model, the prediction models, shown in FIG. 5, at different resolution levels may share the weights or a subset of the weights. In one example, the prediction models at different resolutions are the same or substantially the same. This significantly reduce the size of the model. However, the input data at different resolutions may exhibit different distribution and degrade the performance of the probability model. Described herein are methods that improves the efficiency of a multi-scale probability model with negligible increase of the number of parameters and computation complexity.

[0098] In this disclosure, a scale indicator may be used as an additional input to the distribution predictor in a multi-scale probability model. The scale indicator may be a one-hot vector and processed by a linear projection operator. The output of the linear projection operator may be applied to the one or more feature maps in the distribution predictor.

[0099] The encoder may signal to the decoder whether the scale indicator shall be used as an additional input to the distribution predictor. In another embodiment, the encoder may signal to the decoder the one or more scales that the scale indictor shall be used as an additional input of the distribution predictor.

[0100] In another embodiment, the resolution or parameter derived from theresolution of the input image of the codec may be used as additional inputs to the distribution predictor.

[0101] In another embodiment, a value computed based at least on a current scale and a previous scale, such as a difference value between the current scale and the previous scale, may be input to the distribution predictor.

[0102] In another embodiment, the information derived from the frame position of the input frame of the codec may be used as additional inputs to the distribution predictor in video compression, especially when the reconstructed intra frames are used as extra context.

[0103] FIG. 7 shows an example implementation of a distribution predictor 602 for an i-th scale. Input, (608) are first concatenated with concatenation operation 702 and processed by an input conv layer 704. The input conv layer 704 may include a convolution operation, adding bias and a non-linear activation operation. The output of the input conv layer 704 is processed by a backbone component 706. The backbone 706 may be a convolution-based neural network or a transformer based neural network. The output of the backbone component 706 is processed by an output convl component 708 to generate the estimated distribution parameters604 and an output conv2 component 710 to generate the auxiliary input data zbu+i) £20 for the next step.

[0104] FIG. 8 shows an example implementation of scale indicator 802 as an additional input to the distribution predictor 602.

[0105] In one embodiment, a scale indicator 802 is given as an additional input to the distribution predictor 602 of a multi-scale probability model. The scale indicator 802 may be a one-hot vector, where the element at the position associated with the current scale is one and other elements are zeros. The one-hot vector may be processed by a linear projection component 804. The output of the linear projection may be a vector with the same dimension of the channel dimension of the feature maps to which it applies. The output of the linear projection 804 may be applied in channel-wise to theoutput of the addition of the bias operator 806, for example, being added to the feature map output by the bias addition operator 808. This example implementation is shown in FIG. 8.

[0106] Several other embodiments of the distribution predictor 602 and multi-scale probability model (including multi-scale probability model 500) are described as follows:

[0107] In another embodiment, the output of the linear operation may be applied to the feature map output by the activation operation 810.

[0108] In another embodiment, the output of the linear operation may be used as channel-wise scale factors, for example, to enhance or attenuate channels of the feature maps that it is applied.

[0109] In another embodiment, the linear operation may be replaced by a look-up table, where each row of the table contains the parameters to be applied to the channels of the feature maps.

[0110] In another embodiment, the scale indicator 802 may be a planar tensor with the same spatial dimension of input608 where the element value in the tensor indicates the current scale and concatenated with other inputs of the distribution predictor.[OHl] In another embodiment, the output of the linear projection 804 may be applied to one or more convolution layers in the backbone 706.

[0112] In another embodiment, the output of the linear projection may be applied to multiple places in the distribution predictor, for example, applied to both initial inputs of the distribution predictor and one or more convolution layers in the backbone.

[0113] In another embodiment, the output of the linear projection may be used as an additional token for a transformer-based backbone. In another embodiment, the output of the linear projection may be stored as a hidden state. The hidden state may be updatedin one or more scales with the scale indicator, or the output of the linear projection, or the information derived from them. The hidden state or the information derived from it may be combined with the input or output of the linear projection in the current scale. In a related embodiment, the combination operations may include gating or other nonlinear units.

[0114] In another embodiment, the hidden state may be also updated by one or more of the inputs of the distribution predictor.

[0115] In one example, the output oi+1of the linear projection given scale indicate at scale i + 1 can be stored as hidden state hi+1= oi+1. At scale i, the hidden state may be updated as ht= a(hi+) O °i, where O denotes element-wise multiplication and <J( ) may be a sigmoid function. Then htmay be applied to the feature maps. In another example, the hidden state may be combined with the decoded image x^ as a channel-wise multiplicator ht=• Oj), where F(-)represents an identity function, a linear, or a non-linear operation. In another embodiment, the output of the linear projection may be applied to the one or more estimated parameters of the distribution functions pbJ) porexample, for a Gaussian distribution function, a distribution function is defined by a mean parameter and a scale parameter. The output of the linear projection may be a scale factor that is applied to the scale parameter of the estimated distribution function.

[0116] In another embodiment, the output of the linear projection or information derived from it may be used to determine the order in which the channels are processed. For example, a scaling output or information derived from it may indicate channels that should be processed individually and channels that may be processed in parallel i.e. with the same context.

[0117] In another embodiment, the output of the linear projection or information derived from it may be used to determine the order in which to process the sub- blocks / groups.

[0118] In another embodiment, the output of the linear projection or informationderived from it may be used to determine the order in which to process the pixels in groups.

[0119] In another embodiment, the output of the linear projection or information derived from it may be used to determine the group partitioning pattern. The partitioning patterns may be predefined or may be derived by a predefined algorithm.

[0120] In another embodiment, the encoder may signal to the decoder information indicating whether the scale indicator shall be used as an additional input to the distribution predictor.

[0121] In another embodiment, the encoder may signal to the decoder one or more values that may be used to modify or adjust a predefined scale indicator.

[0122] In another embodiment, the encoder may signal to the decoder information about one or more scales for which the scale indicator shall be used as an additional input to the distribution predictor.

[0123] In another embodiment, the encoder may signal to the decoder the position in the distribution predictor where the scale indicator shall be applied, for example, to indicate if the scale indicator shall be applied to the input conv layer and / or output convolution components.

[0124] In another embodiment, the encoder may signal to the decoder a combination of information which is listed in the previous embodiments.

[0125] In another embodiment, the resolution of the input image or information derived from the resolution of the input image may be used as additional input parameters of the distribution predictor. Information derived from the input resolution may be the product (e.g., result of multiplication operation) of the width and height of the input image.

[0126] In another embodiment, a value computed based at least on a current scale and a previous scale, such as a difference value between the current scale and the previous scale, may be input to the distribution predictor, where the current scale may be a i-thscale and the previous scale may be a (i+l)-th scale, and where the value may be comprised in the scale indicator.

[0127] In another embodiment, information indicating a next (or following) scale with respect to a current scale, such as a (i-l)-th scale when the current scale is a i-th scale, or information derived thereof, may be input to the distribution predictor. For example, such information may be used by the distribution predictor for determining the auxiliary output which is then used as an auxiliary input for the next scale.

[0128] In another embodiment, the scale indicator, or information derived from the scale indicator, may be used for modifying one or more components or portions of the distribution predictor, such as for modifying one or more layers of the distribution predictor. In one example, based on the scale indicator, a transpose convolutional layer with stride greater than 1 may be inserted into the architecture of the distribution predictor at a predefined position of the distribution predictor. In another example, based on the scale indicator, a transpose convolutional layer with stride greater than 1 may replace a convolutional layer with stride equal to 0. In yet another example, a transpose convolutional layer with stride greater than 1, where the value of the stride is set according to the scale indicator, is used in or added to the architecture of the distribution predictor.

[0129] In another embodiment, the information derived from the frame position of the input frame of the codec may be used as additional inputs to the distribution predictor in video compression, especially when the reconstructed intra frames are used as extra context. The information derived from the frame position may be the absolute or relative frame index, or in another embodiment, it may be the relative distance to the corresponding reconstructed intra frame.

[0130] FIG. 9 shows an encoder 900 according to an embodiment. FIG. 9 illustrates an image to be encoded (In), a predicted representation of an image block (P'n), a prediction error signal (Dn), a reconstructed prediction error signal (D'n), a preliminary reconstructed image (I'n), a final reconstructed image (R'n), a transform (T) and inverse transform (T-1), a quantization (Q) and inverse quantizationentropy encoding(E), a reference frame memory (RFM), inter prediction (Pinter), intra prediction (Pintra) mode selection (MS) and filtering (F).

[0131] As shown in FIG. 9, within the encoder 900, the scale indicator 802 is input to the multi-scale probability model 500. A module 910 within the encoder 900 determines elements of the latent representation of the input data from the probability distribution that is an output of the multi-scale probability model 500.

[0132] FIG. 10 shows a decoder 1000 according to an embodiment. FIG. 10 illustrates a predicted representation of an image block (P'n), a reconstructed prediction error signal (D'n), a preliminary reconstructed image (I'n), a final reconstructed image (R'n), an inverse transform (T-1), an inverse quantizationan entropy decoding (E1), a reference frame memory (RFM), a prediction (either inter or intra) (P), and filtering (F).

[0133] As shown in FIG. 10, within the decoder 1000, the scale indicator 802 is optionally used as input to the multi-scale probability model 500. A module 1010 within the decoder 1000 determines elements of the latent representation of the input data from the probability distribution that is an output of the multi-scale probability model 500.

[0134] A video encoder transforms the input video into a compressed representation suited for storage / transmission and a video decoder decompresses the compressed video representation back into a viewable form. Typically, an encoder discards some information in the original video sequence in order to represent the video in a more compact form (that is, at lower bitrate).

[0135] A video encoder may encode the video information in two phases. Firstly, pixel values in a certain picture area (or “block”) are predicted for example by motion compensation means (finding and indicating an area in one of the previously coded video frames that corresponds closely to the block being coded) or by spatial means (using the pixel values around the block to be coded in a specified manner). Secondly the prediction error, e.g., the difference between the predicted block of pixels and theoriginal block of pixels, is coded. This is typically done by transforming the difference in pixel values using a specified transform (e.g., Discrete Cosine Transform (DCT) or a variant of it), quantizing the coefficients and entropy coding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the accuracy of the pixel representation (picture quality) and size of the resulting coded video representation (file size or transmission bitrate).

[0136] Inter prediction, which may also be referred to as temporal prediction, motion compensation, or motion-compensated prediction, exploits temporal redundancy. In inter prediction the sources of prediction are previously decoded pictures (a.k.a. reference pictures).

[0137] Intra prediction utilizes the fact that adjacent pixels within the same picture are likely to be correlated. Intra prediction can be performed in spatial or transform domain, e.g., either sample values or transform coefficients can be predicted. Intra prediction is typically exploited in intra coding, where no inter prediction is applied.

[0138] An intra picture may be defined as a coded picture that is decoded using intra prediction only, or in other words, does not make use of inter prediction in decoding. An intra picture may be interchangeably called an intra frame.

[0139] An inter picture may be defined as a coded picture whose decoding may include intra prediction and inter prediction. An inter picture may be interchangeably called an inter frame.

[0140] FIG. 11 is a block diagram illustrating a system 1100 in accordance with several examples. In an example, the encoder 1130 is used to encode an image or video from the scene 1115 (the scene 1115 may be captured using one or more cameras), and the encoder 1130 is implemented in a transmitting apparatus 1180. The encoder 1130 produces a bitstream 1110 comprising signaling that is received by the receiving apparatus 1182, which implements a decoder 1140. The encoder 1130 sends the bitstream 1110 that comprises the herein described signaling. The decoder 1140 forms the image or video for the scene 1115-1, and the receiving apparatus 1182 wouldpresent this to the user, e.g., via a smartphone, television, or projector among many other options.

[0141] In some examples, the transmitting apparatus 1180 and the receiving apparatus 1182 are at least partially within a common apparatus, and for example are located within a common housing 1150. In other examples the transmitting apparatus 1180 and the receiving apparatus 1182 are at least partially not within a common apparatus and have at least partially different housings. Therefore in some examples, the encoder 1130 and the decoder 1140 are at least partially within a common apparatus, and for example are located within a common housing 1150. For example the common apparatus comprising the encoder 1130 and decoder 1140 implements a codec. In other examples the encoder 1130 and the decoder 1140 are at least partially not within a common apparatus and have at least partially different housings, but when together still implement a codec.

[0142] In some examples, 3D media from the capture (e.g., volumetric capture) at a viewpoint 1112 of the scene 1115, which includes a person 1113) is converted via projection to a series of 2D representations with occupancy, geometry, attributes and / or displacements. Additional atlas information is also included in the bitstream to enable inverse reconstruction. For decoding, the received bitstream 1110 is separated into its components with atlas information; occupancy, geometry, displacement, and attribute 2D representations. A 3D reconstruction is performed to reconstruct the scene 1115-1 created looking at the viewpoint 1112-1 with a “reconstructed” person 1113-1. The “- 1” are used to indicate that these are reconstructions of the original.

[0143] As indicated at 1120, the decoder 1140 performs an action or actions based on the received scale indicator signaling.

[0144] Encoding 1190 performs the examples described herein related to using a scale indicator input as input to a multi-scale probability model on the encoding side. Decoding 1192 performs the decoding-side related examples described herein related to a scale indicator being used as input to a multi-scale probability model.

[0145] FIG. 12 is an example apparatus 1200, which may be implemented in hardware, configured to implement the examples described herein. The apparatus 1200 comprises at least one processor 1202 (e.g., an FPGA and / or CPU and / or GPU), one or more memories 1204 including computer program code 1205, the computer program code 1205 having instructions to carry out the methods described herein, wherein the at least one memory 1204 and the computer program code 1205 are configured to, with the at least one processor 1202, cause the apparatus 1200 to implement circuitry, a process, component, module, or function (implemented with control module 1206) to implement the examples described herein.

[0146] Apparatus 1200 may be a smartphone, personal digital device or assistant, smart television, laptop, pad, tablet, head-mounted display (HMD), or other user device or terminal device. The memory 1204 may be a non-transitory memory, a transitory memory, a volatile memory (e.g. RAM), or a non-volatile memory (e.g., ROM).

[0147] As shown in FIG. 12, codec 1230 optionally includes a scale indicator 804 that is optionally used as input to the multi-scale probability model 500.

[0148] The apparatus 1200 includes a display and / or VO interface 1208, which includes user interface (UI) circuitry and elements, that may be used to display features or a status of the methods described herein (e.g., as one of the methods is being performed or at a subsequent time), or to receive input from a user such as with using a keypad, camera, touchscreen, touch area, microphone, biometric recognition, one or more sensors, etc. The apparatus 1200 includes one or more communication e.g. network (N / W) interfaces (I / F(s)) 1210. The communication I / F(s) 1210 may be wired and / or wireless and communicate over the Intemet / other network(s) via any communication technique including via one or more links 1224. The communication I / F(s) 1210 may comprise one or more transmitters or one or more receivers.

[0149] The transceiver 1216 comprises one or more transmitters 1218 and one or more receivers 1220. The transceiver 1216 and / or communication I / F(s) 1210 may comprise standard well-known components such as an amplifier, filter, frequencyconverter, (de)modulator, and encoder / decoder circuitries and one or more antennas,such as antennas 1214 used for communication over wireless link 1226.

[0150] The control module 1206 of the apparatus 1200 comprises one of or both parts 1206-1 and / or 1206-2, which may be implemented in a number of ways. The control module 1206 may be implemented in hardware as control module 1206-1, such as being implemented as part of the one or more processors 1202. The control module 1206-1 may be implemented also as an integrated circuit or through other hardware such as a programmable gate array. In another example, the control module 1206 may be implemented as control module 1206-2, which is implemented as computer program code (having corresponding instructions) 1205 and is executed by the one or more processors 1202. For instance, the one or more memories 1204 store instructions that, when executed by the one or more processors 1202, cause the apparatus 1200 to perform one or more of the operations as described herein. Furthermore, the one or more processors 1202, one or more memories 1204, and example algorithms (e.g., as flowcharts and / or signaling diagrams), encoded as instructions, programs, or code, are means for causing performance of the operations described herein.

[0151] The apparatus 1200 to implement the functionality of control 1206 may correspond to any of the apparatuses depicted herein. Alternatively, apparatus 1200 and its elements may not correspond to any of the other apparatuses depicted herein, as apparatus 1200 may be part of a self-organizing / optimizing network (SON) node or other node, such as a node in a cloud.

[0152] The apparatus 1200 may also be distributed throughout the network including within and between apparatus 1200 and any network element (such as a base station and / or terminal device and / or user equipment).

[0153] Interface 1212 enables data communication and signaling between the various items of apparatus 1200, as shown in FIG. 12. For example, the interface 1212 may be one or more buses such as address, data, or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment, and the like. Computer program code (e.g. instructions) 1205, including control 1206 may comprise object-oriented software configured to pass data or messages between objects within computer program code 1205. Computer program code (e.g. instructions) 1205, including control 1206 may comprise procedural, functional, or scripting code. The apparatus 1200 need not comprise each of the features mentioned, or may comprise other features as well. The various components of apparatus 1200 may at least partially reside in a common housing 1228, or a subset of the various components of apparatus 1200 may at least partially be located in different housings, which different housings may include housing 1228.

[0154] FIG. 13 shows a schematic representation of non-volatile memory media 1300a (e.g. computer / compact disc (CD) or digital versatile disc (DVD)) and 1300b (e.g. universal serial bus (USB) memory stick) and 1300c (e.g. cloud storage for downloading instructions and / or parameters 1302 or receiving emailed instructions and / or parameters 1302) storing instructions and / or parameters 1302 which when executed by a processor allows the processor to perform one or more of the operations of the methods described herein. Instructions and / or parameters 1302 may represent or correspond to a non-transitory computer readable medium.

[0155] FIG. 14 is an example method 1400 based on the examples described herein. At 1410, the method includes using a distribution predictor to predict parameters of a probability distribution for elements of a latent representation of at least one image for a resolution level. At 1420, the method includes wherein an input to the distribution predictor comprises a scale indicator that indicates information related to the resolution level. At 1430, the method includes coding the elements of the latent representation of the at least one image for the resolution level using the predicted parameters of the probability distribution into or along a bitstream. Method 1400 may be performed with codec 100, video coding pipeline 200, arithmetic coding 308, arithmetic encoder 408, probability model 410, multi-scale probability model 500, distribution predictor 602, encoder 900, transmitting apparatus 1180 with encoder 1130, or apparatus 1200.

[0156] FIG. 15 is an example method 1500 based on the examples described herein.At 1510, the method includes decoding, from or along a bitstream, information relatedto elements of a latent representation of at least one image for a resolution level. At 1520, the method includes determining whether to use a scale indicator that indicates information related to the resolution level as input to a distribution predictor to predict parameters of a probability distribution for the elements of the latent representation of the at least one image for the resolution level. At 1530, the method includes using the distribution predictor to predict the parameters of the probability distribution for the elements of the latent representation of the at least one image for the resolution level. At 1540, the method includes wherein the scale indicator that indicates information related to the resolution level is used as input to the distribution predictor to predict the parameters of the probability distribution for the elements of the latent representation of the at least one image for the resolution level, in response to determining to use the scale indicator. Method 1500 may be performed with codec 100, video coding pipeline 200, arithmetic decoding 312, probability model 410, arithmetic decoder 414, multiscale probability model 500, distribution predictor 602, decoder 1000, receiving apparatus 1182 with decoder 1140, or apparatus 1200.

[0157] The following examples are provided and described herein.

[0158] Example 1. An apparatus including: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: use a distribution predictor to predict parameters of a probability distribution for elements of a latent representation of at least one image for a resolution level; wherein an input to the distribution predictor comprises a scale indicator that indicates information related to the resolution level; and code the elements of the latent representation of the at least one image for the resolution level using the predicted parameters of the probability distribution into or along a bitstream.

[0159] Example 2. The apparatus of example 1, wherein the apparatus is caused to: predict, using a first prediction model, an auxiliary output at the resolution level; wherein the auxiliary output at the resolution level is used as input to a second prediction model used to predict parameters of a probability distribution for elements of the latent representation of at least one image for another resolution level that ishigher than the resolution level.

[0160] Example 3. The apparatus of example 2, wherein an auxiliary input at a resolution level lower than the resolution level is used as input to the first prediction model to predict the auxiliary output at the resolution level.

[0161] Example 4. The apparatus of any of examples 1 to 3, wherein the apparatus is further caused to: process, using a linear projection component, the scale indicator to obtain a linear projection component output.

[0162] Example 5. The apparatus of example 4, wherein the scale indicator is a one- hot vector, wherein an element at a position associated with a current scale is one, and an element not at the position associated with the current scale is zero.

[0163] Example 6. The apparatus of any of examples 4 to 5, wherein the linear projection component output comprises a vector having a dimension that is the same as a channel dimension of at least one feature map to which the vector applies.

[0164] Example 7. The apparatus of any of examples 4 to 6, wherein the apparatus is further caused to: apply the linear projection component output obtained from processing the scale indicator using the linear projection component to one or more feature maps in the distribution predictor.

[0165] Example 8. The apparatus of any of examples 4 to 7, wherein the apparatus is further caused to: apply the output of the linear projection component in channel -wise to an output of a bias operator.

[0166] Example 9. The apparatus of any of examples 1 to 8, wherein the apparatus is further caused to: signal, to a decoder, an indication of whether to use the scale indicator, or skip and not use the scale indicator.

[0167] Example 10. The apparatus of any of examples 1 to 9, wherein the apparatus is further caused to: signal, to a decoder, one or more scales processed by the distribution predictor.

[0168] Example 11. The apparatus of any of examples 1 to 10, wherein an input to the distribution predictor comprises: a resolution of an input image of a codec comprising the distribution predictor, or a parameter derived from a resolution of an input image of a codec comprising the distribution predictor.

[0169] Example 12. The apparatus of any of examples 1 to 11, wherein the apparatus is further caused to: compute a value based on a current scale and a previous scale; wherein the value computed based on the current scale and the previous scale is used as input to the distribution predictor.

[0170] Example 13. The apparatus of any of examples 1 to 12, wherein the value computed based on the current scale and the previous scale comprises a difference between the current scale and the previous scale.

[0171] Example 14. The apparatus of any of examples 1 to 13, wherein the apparatus is further caused to: derive information from a frame position of an input frame of a codec comprising the distribution predictor; wherein the information derived from the frame position of the input frame of the codec is used as input to the distribution predictor for video compression.

[0172] Example 15. The apparatus of example 14, wherein reconstructed intra frames are used as extra context for the video compression.

[0173] Example 16. The apparatus of any of examples 1 to 15, wherein inputs to the distribution predictor to predict the parameters of the probability distribution for the elements of the latent representation of the at least one image for the resolution level comprise: an input tensor that contains elements of the latent representation of the at least one image for the resolution level that have been processed up to but not including a step corresponding to a latent representation partition, and predicted values of the elements of the latent representation of the at least one image for the resolution level that have not been processed at the step corresponding to the latent representation partition, and an auxiliary input corresponding to the latent representation partition of the latent representation of the at least one image for the resolution level, and a masktensor corresponding to the latent representation partition of the latent representation of the at least one image for the resolution level, wherein the mask tensor indicates positions of the elements of the latent representation partition of the latent representation of the at least one image for the resolution level that have been processed up to but not including the step corresponding to the latent representation partition.

[0174] Example 17. The apparatus of any of examples 1 to 16, wherein an auxiliary input associated with a latent representation partition of the latent representation of the at least one image for the resolution level is used as input to the distribution predictor to predict the parameters of the probability distribution for elements of the latent representation partition of the latent representation of the at least one image for the resolution level.

[0175] Example 18. The apparatus of example 17, wherein the apparatus is further caused to: use the distribution predictor to predict a next auxiliary input associated with a next latent representation partition of a latent representation of the at least one image for the resolution level; wherein the next auxiliary input associated with the next latent representation partition of the latent representation of the at least one image for the resolution level is next after the auxiliary input associated with the latent representation partition of the latent representation of the at least one image for the resolution level; wherein the next latent representation partition of the latent representation of the at least one image for the resolution level is next after the latent representation partition of the latent representation of the at least one image for the resolution level.

[0176] Example 19. The apparatus of any of examples 1 to 18, wherein the apparatus is further caused to: use the distribution predictor to predict a next auxiliary input associated with a next latent representation having a resolution that is higher than the latent representation of the at least one image for the resolution level; wherein an auxiliary input associated with the latent representation of the at least one image for a resolution level lower than the resolution level is used as input to the distribution predictor to predict the parameters of the probability distribution for the elements of the latent representation of the at least one image for the resolution level; wherein the nextauxiliary input associated with the next latent representation having the resolution that is higher than the latent representation of the at least one image for the resolution level is next after the auxiliary input associated with the latent representation of the at least one image for a resolution level; wherein the next latent representation having the resolution that is higher than the latent representation of the at least one image for the resolution level is next after the latent representation of the at least one image for a resolution level lower than the resolution level.

[0177] Example 20. The apparatus of any of examples 1 to 19, wherein the apparatus is further caused to: update an input tensor that contains elements of the latent representation of the at least one image for the resolution level that have been processed up to but not including a step corresponding to a latent representation partition and predicted values of the elements of the latent representation of the at least one image for the resolution level that have not been processed at the step corresponding to the latent representation partition, to generate an updated input tensor that contains elements of the latent representation of the at least one image for the resolution level that have been processed up to and including the step corresponding to the latent representation partition and predicted values of the elements of the latent representation of the at least one image for the resolution level that have not been processed at a step corresponding to a next latent representation partition.

[0178] Example 21. The apparatus of any of examples 1 to 20, wherein the apparatus is further caused to: update a mask tensor corresponding to a latent representation partition of the latent representation of the at least one image for the resolution level to generate an updated mask tensor corresponding to a next latent representation partition of the latent representation of the at least one image for the resolution level; wherein the mask tensor indicates positions of the elements of the latent representation of the at least one image for the resolution level that have been processed up to but not including a step corresponding to the latent representation partition; wherein the updated mask tensor indicates positions of the elements of the latent representation of the at least one image for the resolution level that have been processed up to but not including a step corresponding to the next latent representation partition; wherein the next latentrepresentation partition of the latent representation of the at least one image for the resolution level is next after the latent representation partition of the latent representation of the at least one image for the resolution level.

[0179] Example 22. The apparatus of example 21, wherein the predicted parameters of the probability distribution for the elements of the latent representation of the at least one image for the resolution level are obtained based on a latent tensor associated with a previous resolution level that is lower than the resolution level.

[0180] Example 23. The apparatus of any of examples 1 to 22, wherein the scale indicator, or information derived from the scale indicator, is used for modifying one or more components, portions, or layers of the distribution predictor.

[0181] Example 24. The apparatus of any of examples 1 to 23, wherein the scale indicator, or information derived from the scale indicator, is used for modifying one or more layers of the distribution predictor.

[0182] Example 25. The apparatus of any of examples 1 to 24, wherein the apparatus is further caused to: insert, based on the scale indicator, a transpose convolutional layer with stride greater than 1 into an architecture of the distribution predictor at a predefined position of the distribution predictor.

[0183] Example 26. An apparatus including: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: decode, from or along a bitstream, information related to elements of a latent representation of at least one image for a resolution level; determine whether to use a scale indicator that indicates information related to the resolution level as input to a distribution predictor to predict parameters of a probability distribution for the elements of the latent representation of the at least one image for the resolution level; and use the distribution predictor to predict the parameters of the probability distribution for the elements of the latent representation of the at least one image for the resolution level; wherein the scale indicator that indicates information related to the resolution level is used as input to the distribution predictor to predict the parametersof the probability distribution for the elements of the latent representation of the at least one image for the resolution level, in response to determining to use the scale indicator.

[0184] Example 27. The apparatus of example 26, wherein the apparatus is further caused to: process, using a linear projection component, the scale indicator to obtain a linear projection component output.

[0185] Example 28. The apparatus of example 27, wherein the scale indicator is a one-hot vector, wherein an element at a position associated with a current scale is one, and an element not at the position associated with the current scale is zero.

[0186] Example 29. The apparatus of any of examples 27 to 28, wherein the linear projection component output comprises a vector having a dimension that is the same as a channel dimension of at least one feature map to which the vector applies.

[0187] Example 30. The apparatus of any of examples 27 to 29, wherein the apparatus is further caused to: apply the linear projection component output obtained from processing the scale indicator using the linear projection component to one or more feature maps in the distribution predictor.

[0188] Example 31. The apparatus of any of examples 27 to 30, wherein the apparatus is further caused to: apply the output of linear projection component in channel-wise to an output of a bias operator.

[0189] Example 32. The apparatus of any of examples 26 to 31 , wherein the apparatus is further caused to: receive, from an encoder, signaling comprising an indication of whether to use the scale indicator as input to a distribution predictor, or to skip the scale indicator and not use the scale indicator as input to the distribution predictor; use the scale indicator as input to the distribution predictor to predict the parameters of the probability distribution for the elements of the latent representation of the at least one image for the resolution level, in response to the signaling received from the encoder indicating to use the scale indicator as input to the distribution predictor; and skip and do not use the scale indicator as input to the distribution predictor to predict the parameters of the probability distribution for the elements of the latent representationof the at least one image for the resolution level, in response to the signaling received from the encoder indicating to skip the scale indicator and not use the scale indicator as input to the distribution predictor.

[0190] Example 33. The apparatus of any of examples 26 to 32, wherein the apparatus is further caused to: receive, from an encoder, signaling of one or more scales processed by the distribution predictor.

[0191] Example 34. The apparatus of any of examples 26 to 33, wherein an input to the distribution predictor comprises: a resolution of an input image of a codec comprising the distribution predictor, or a parameter derived from a resolution of an input image of a codec comprising the distribution predictor.

[0192] Example 35. The apparatus of any of examples 26 to 34, wherein the apparatus is further caused to: compute a value based on a current scale and a previous scale; wherein the value computed based on the current scale and the previous scale is used as input to the distribution predictor.

[0193] Example 36. The apparatus of any of examples 26 to 35, wherein the value computed based on the current scale and the previous scale comprises a difference between the current scale and the previous scale.

[0194] Example 37. The apparatus of any of examples 26 to 36, wherein the apparatus is further caused to: derive information from a frame position of an input frame of a codec comprising the distribution predictor; wherein the information derived from the frame position of the input frame of the codec is used as input to the distribution predictor for video compression.

[0195] Example 38. The apparatus of example 37, wherein reconstructed intra frames are used as extra context for the video compression.

[0196] Example 39. The apparatus of any of examples 26 to 38, wherein the apparatus is further caused to: update an input tensor that contains elements of the latent representation of the at least one image for the resolution level that have been processedup to but not including a step corresponding to a latent representation partition and predicted values of the elements of the latent representation of the at least one image for the resolution level that have not been processed at the step corresponding to the latent representation partition, to generate an updated input tensor that contains elements of the latent representation of the at least one image for the resolution level that have been processed up to and including the step corresponding to the latent representation partition, and predicted values of the elements of the latent representation of the at least one image for the resolution level that have not been processed at a step corresponding to a next latent representation partition.

[0197] Example 40. The apparatus of any of examples 26 to 39, wherein the apparatus is further caused to: update a mask tensor corresponding to a latent representation partition of the latent representation of the at least one image for the resolution level to generate an updated mask tensor corresponding to a next latent representation partition of the latent representation of the at least one image for the resolution level; wherein the mask tensor indicates positions of the elements of the latent representation of the at least one image for the resolution level that have been processed up to but not including a step corresponding to the latent representation partition; wherein the updated mask tensor indicates positions of the elements of the latent representation of the at least one image for the resolution level that have been processed up to but not including a step corresponding to the next latent representation partition; wherein the next latent representation partition of the latent representation of the at least one image for the resolution level is next after the latent representation partition of the latent representation of the at least one image for the resolution level.

[0198] Example 41. The apparatus of any of examples 26 to 40, wherein the predicted parameters of the probability distribution for the elements of the latent representation of the at least one image for the resolution level are obtained based on a latent tensor associated with a previous resolution level that is lower than the resolution level.

[0199] Example 42. The apparatus of any of examples 26 to 41, wherein the scale indicator, or information derived from the scale indicator, is used for modifying one ormore components, portions, or layers of the distribution predictor.

[0200] Example 43. The apparatus of any of examples 26 to 42, wherein the scale indicator, or information derived from the scale indicator, is used for modifying one or more layers of the distribution predictor.

[0201] Example 44. The apparatus of any of examples 26 to 43, wherein the apparatus is further caused to: insert, based on the scale indicator, a transpose convolutional layer with stride greater than 1 into an architecture of the distribution predictor at a predefined position of the distribution predictor.

[0202] Example 45. A method including: using a distribution predictor to predict parameters of a probability distribution for elements of a latent representation of at least one image for a resolution level; wherein an input to the distribution predictor comprises a scale indicator that indicates information related to the resolution level; and coding the elements of the latent representation of the at least one image for the resolution level using the predicted parameters of the probability distribution into or along a bitstream.

[0203] Example 46. A method including: decoding, from or along a bitstream, information related to elements of a latent representation of at least one image for a resolution level; determining whether to use a scale indicator that indicates information related to the resolution level as input to a distribution predictor to predict parameters of a probability distribution for the elements of the latent representation of the at least one image for the resolution level; and using the distribution predictor to predict the parameters of the probability distribution for the elements of the latent representation of the at least one image for the resolution level; wherein the scale indicator that indicates information related to the resolution level is used as input to the distribution predictor to predict the parameters of the probability distribution for the elements of the latent representation of the at least one image for the resolution level, in response to determining to use the scale indicator.

[0204] Example 47. An apparatus including: means for using a distribution predictorto predict parameters of a probability distribution for elements of a latent representation of at least one image for a resolution level; wherein an input to the distribution predictor comprises a scale indicator that indicates information related to the resolution level; and means for coding the elements of the latent representation of the at least one image for the resolution level using the predicted parameters of the probability distribution into or along a bitstream.

[0205] Example 48. An apparatus including: means for decoding, from or along a bitstream, information related to elements of a latent representation of at least one image for a resolution level; means for determining whether to use a scale indicator that indicates information related to the resolution level as input to a distribution predictor to predict parameters of a probability distribution for the elements of the latent representation of the at least one image for the resolution level; and means for using the distribution predictor to predict the parameters of the probability distribution for the elements of the latent representation of the at least one image for the resolution level; wherein the scale indicator that indicates information related to the resolution level is used as input to the distribution predictor to predict the parameters of the probability distribution for the elements of the latent representation of the at least one image for the resolution level, in response to determining to use the scale indicator.

[0206] Example 49. A computer readable medium including instructions stored thereon for performing at least the following: using a distribution predictor to predict parameters of a probability distribution for elements of a latent representation of at least one image for a resolution level; wherein an input to the distribution predictor comprises a scale indicator that indicates information related to the resolution level; and coding the elements of the latent representation of the at least one image for the resolution level using the predicted parameters of the probability distribution into or along a bitstream.

[0207] Example 50. A computer readable medium including instructions stored thereon for performing at least the following: decoding, from or along a bitstream, information related to elements of a latent representation of at least one image for aresolution level; determining whether to use a scale indicator that indicates information related to the resolution level as input to a distribution predictor to predict parameters of a probability distribution for the elements of the latent representation of the at least one image for the resolution level; and using the distribution predictor to predict the parameters of the probability distribution for the elements of the latent representation of the at least one image for the resolution level; wherein the scale indicator that indicates information related to the resolution level is used as input to the distribution predictor to predict the parameters of the probability distribution for the elements of the latent representation of the at least one image for the resolution level, in response to determining to use the scale indicator.

[0208] References to a ‘computer’, ‘processor’, etc. should be understood to encompass not only computers having different architectures such as single / multi- processor architectures and sequential / parallel architectures but also specialized circuits such as field-programmable gate arrays (FPGAs), application specific circuits (ASICs), signal processing devices and other processing circuitry. References to computer program, instructions, code etc. should be understood to encompass software for a programmable processor or firmware such as, for example, the programmable content of a hardware device such as instructions for a processor, or configuration settings for a fixed-function device, gate array or programmable logic device, etc.

[0209] The term “non-transitory,” as used herein, is a limitation of the medium itself (i.e., tangible, not a signal) as opposed to a limitation on data storage persistency (e.g., RAM vs. ROM).

[0210] As used herein, the term ‘circuitry’, ‘circuit’ and variants may refer to any of the following: (a) hardware circuit implementations, such as implementations in analog and / or digital circuitry, and (b) combinations of circuits and software (and / or firmware), such as (as applicable): (i) a combination of processor(s) or (ii) portions of processor(s) / software including digital signal processor(s), software, and one or more memories that work together to cause an apparatus to perform various functions, and (c) circuits, such as a microprocessor(s) or a portion of a microprocessor(s), that requiresoftware or firmware for operation, even when the software or firmware is not physically present. As a further example, as used herein, the term ‘circuitry’ would also cover an implementation of merely a processor (or multiple processors) or a portion of a processor and its (or their) accompanying software and / or firmware. The term ‘circuitry’ would also cover, for example and when applicable to the particular element, a baseband integrated circuit or applications processor integrated circuit for a mobile phone or a similar integrated circuit in a server, a cellular network device, or another network device. Circuitry or circuit may also be used to mean a function or a process used to execute a method.

[0211] It should be understood that the foregoing description is only illustrative. Various alternatives and modifications may be devised by those skilled in the art. For example, features recited in the various dependent claims could be combined with each other in any suitable combination(s). In addition, features from different embodiments described above could be selectively combined into a new embodiment. Accordingly, the description is intended to embrace all such alternatives, modifications and variances which fall within the scope of the appended claims.

[0212] The following acronyms and abbreviations that may be found in the specification and / or the drawing figures are defined as follows (the abbreviations may be appended with each other or with other characters using e.g. a hyphen, dash (-), or number (or abbreviations having a character may be the same with a character removed), and may be case insensitive):2D two-dimensional3D three-dimensionalASIC application specific integrated circuitAVC advanced video coding bpp bits per pixel cat concatenation conv convolution or convolutionalCPU central processing unitDCT Discrete Cosine TransformFPGA field programmable gate arrayGAN generative adversarial networkGPU graphics processing unitH.2xx family of video coding standards (e.g. H.263, H.264, H.265,H.266, H.274)HEVC high efficiency video codingHMD head-mounted displayI / F interfaceI / O input / outputITU-T International Telecommunication Union TelecommunicationStandardization SectorLO norm number of non-zero elements in a vectorLI norm sum of the absolute value of the entries in the vectorL2 norm distance of a point from the origin in the Euclidean planeMC motion compensationME motion estimationMSE mean squared errorMS-SSIM multi-scale structural similarityNAL network abstraction layerNet networkNN neural networkN / W networkOpt optionPred predictionPSNR peak signal-to-noise ratioQ quantizationRAM random access memoryRGB red green blueRFM reference frame memoryROM read only memorySEI supplemental enhancement informationSON self-organizing / optimizing networkSSIM structural similarity index measureT transformUI user interfaceUSB universal serial busVVC versatile video coding

Claims

CLAIMS:

1. An apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: use a distribution predictor to predict parameters of a probability distribution for elements of a latent representation of at least one image for a resolution level; wherein an input to the distribution predictor comprises a scale indicator that indicates information related to the resolution level; and code the elements of the latent representation of the at least one image for the resolution level using the predicted parameters of the probability distribution into or along a bitstream.

2. The apparatus of claim 1, wherein the apparatus is caused to: predict, using a first prediction model, an auxiliary output at the resolution level; wherein the auxiliary output at the resolution level is used as input to a second prediction model used to predict parameters of a probability distribution for elements of the latent representation of at least one image for another resolution level that is higher than the resolution level.

3. The apparatus of claim 2, wherein an auxiliary input at a resolution level lower than the resolution level is used as input to the first prediction model to predict the auxiliary output at the resolution level.

4. The apparatus of any of claims 1 to 3, wherein the apparatus is further caused to: process, using a linear projection component, the scale indicator to obtain a linear projection component output.

5. The apparatus of claim 4, wherein the scale indicator is a one-hot vector, wherein an element at a position associated with a current scale is one, and an element not at the position associated with the current scale is zero.

6. The apparatus of any of claims 4 to 5, wherein the linear projection component output comprises a vector having a dimension that is the same as a channel dimension of at least one feature map to which the vector applies.

7. The apparatus of any of claims 4 to 6, wherein the apparatus is further caused to: apply the linear projection component output obtained from processing the scale indicator using the linear projection component to one or more feature maps in the distribution predictor.

8. The apparatus of any of claims 4 to 7, wherein the apparatus is further caused to: apply the output of the linear projection component in channel -wise to an output of a bias operator.

9. The apparatus of any of claims 1 to 8, wherein the apparatus is further caused to: signal, to a decoder, an indication of whether to use the scale indicator, or skip and not use the scale indicator.

10. The apparatus of any of claims 1 to 9, wherein the apparatus is further caused to: signal, to a decoder, one or more scales processed by the distribution predictor.

11. The apparatus of any of claims 1 to 10, wherein an input to the distribution predictor comprises: a resolution of an input image of a codec comprising the distribution predictor, or a parameter derived from a resolution of an input image of a codec comprising the distribution predictor.4712. The apparatus of any of claims 1 to 11, wherein the apparatus is further caused to: compute a value based on a current scale and a previous scale; wherein the value computed based on the current scale and the previous scale is used as input to the distribution predictor.

13. The apparatus of any of claims 1 to 12, wherein the value computed based on the current scale and the previous scale comprises a difference between the current scale and the previous scale.

14. The apparatus of any of claims 1 to 13, wherein the apparatus is further caused to: derive information from a frame position of an input frame of a codec comprising the distribution predictor; wherein the information derived from the frame position of the input frame of the codec is used as input to the distribution predictor for video compression.

15. The apparatus of claim 14, wherein reconstructed intra frames are used as extra context for the video compression.

16. The apparatus of any of claims 1 to 15, wherein inputs to the distribution predictor to predict the parameters of the probability distribution for the elements of the latent representation of the at least one image for the resolution level comprise: an input tensor that contains elements of the latent representation of the at least one image for the resolution level that have been processed up to but not including a step corresponding to a latent representation partition, and predicted values of the elements of the latent representation of the at least one image for the resolution level that have not been processed at the step corresponding to the latent representation partition, and an auxiliary input corresponding to the latent representation partition of the latent representation of the at least one image for the resolution level, and48a mask tensor corresponding to the latent representation partition of the latent representation of the at least one image for the resolution level, wherein the mask tensor indicates positions of the elements of the latent representation partition of the latent representation of the at least one image for the resolution level that have been processed up to but not including the step corresponding to the latent representation partition.

17. The apparatus of any of claims 1 to 16, wherein an auxiliary input associated with a latent representation partition of the latent representation of the at least one image for the resolution level is used as input to the distribution predictor to predict the parameters of the probability distribution for elements of the latent representation partition of the latent representation of the at least one image for the resolution level.

18. The apparatus of claim 17, wherein the apparatus is further caused to: use the distribution predictor to predict a next auxiliary input associated with a next latent representation partition of a latent representation of the at least one image for the resolution level; wherein the next auxiliary input associated with the next latent representation partition of the latent representation of the at least one image for the resolution level is next after the auxiliary input associated with the latent representation partition of the latent representation of the at least one image for the resolution level; wherein the next latent representation partition of the latent representation of the at least one image for the resolution level is next after the latent representation partition of the latent representation of the at least one image for the resolution level.

19. The apparatus of any of claims 1 to 18, wherein the apparatus is further caused to: use the distribution predictor to predict a next auxiliary input associated with a next latent representation having a resolution that is higher than the latent representation of the at least one image for the resolution level; wherein an auxiliary input associated with the latent representation of the at least one image for a resolution level lower than the resolution level is used as input tothe distribution predictor to predict the parameters of the probability distribution for the elements of the latent representation of the at least one image for the resolution level; wherein the next auxiliary input associated with the next latent representation having the resolution that is higher than the latent representation of the at least one image for the resolution level is next after the auxiliary input associated with the latent representation of the at least one image for a resolution level; wherein the next latent representation having the resolution that is higher than the latent representation of the at least one image for the resolution level is next after the latent representation of the at least one image for a resolution level lower than the resolution level.

20. The apparatus of any of claims 1 to 19, wherein the apparatus is further caused to: update an input tensor that contains elements of the latent representation of the at least one image for the resolution level that have been processed up to but not including a step corresponding to a latent representation partition and predicted values of the elements of the latent representation of the at least one image for the resolution level that have not been processed at the step corresponding to the latent representation partition, to generate an updated input tensor that contains elements of the latent representation of the at least one image for the resolution level that have been processed up to and including the step corresponding to the latent representation partition and predicted values of the elements of the latent representation of the at least one image for the resolution level that have not been processed at a step corresponding to a next latent representation partition.

21. The apparatus of any of claims 1 to 20, wherein the apparatus is further caused to: update a mask tensor corresponding to a latent representation partition of the latent representation of the at least one image for the resolution level to generate an updated mask tensor corresponding to a next latent representation partition of the latent representation of the at least one image for the resolution level;wherein the mask tensor indicates positions of the elements of the latent representation of the at least one image for the resolution level that have been processed up to but not including a step corresponding to the latent representation partition; wherein the updated mask tensor indicates positions of the elements of the latent representation of the at least one image for the resolution level that have been processed up to but not including a step corresponding to the next latent representation partition; wherein the next latent representation partition of the latent representation of the at least one image for the resolution level is next after the latent representation partition of the latent representation of the at least one image for the resolution level.

22. The apparatus of claim 21, wherein the predicted parameters of the probability distribution for the elements of the latent representation of the at least one image for the resolution level are obtained based on a latent tensor associated with a previous resolution level that is lower than the resolution level.

23. The apparatus of any of claims 1 to 22, wherein the scale indicator, or information derived from the scale indicator, is used for modifying one or more components, portions, or layers of the distribution predictor.

24. The apparatus of any of claims 1 to 23, wherein the scale indicator, or information derived from the scale indicator, is used for modifying one or more layers of the distribution predictor.

25. The apparatus of any of claims 1 to 24, wherein the apparatus is further caused to: insert, based on the scale indicator, a transpose convolutional layer with stride greater than 1 into an architecture of the distribution predictor at a predefined position of the distribution predictor.

26. An apparatus comprising: at least one processor; andat least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to: decode, from or along a bitstream, information related to elements of a latent representation of at least one image for a resolution level; determine whether to use a scale indicator that indicates information related to the resolution level as input to a distribution predictor to predict parameters of a probability distribution for the elements of the latent representation of the at least one image for the resolution level; and use the distribution predictor to predict the parameters of the probability distribution for the elements of the latent representation of the at least one image for the resolution level; wherein the scale indicator that indicates information related to the resolution level is used as input to the distribution predictor to predict the parameters of the probability distribution for the elements of the latent representation of the at least one image for the resolution level, in response to determining to use the scale indicator.

27. The apparatus of claim 26, wherein the apparatus is further caused to: process, using a linear projection component, the scale indicator to obtain a linear projection component output.

28. The apparatus of claim 27, wherein the scale indicator is a one-hot vector, wherein an element at a position associated with a current scale is one, and an element not at the position associated with the current scale is zero.

29. The apparatus of any of claims 27 to 28, wherein the linear projection component output comprises a vector having a dimension that is the same as a channel dimension of at least one feature map to which the vector applies.

30. The apparatus of any of claims 27 to 29, wherein the apparatus is further caused to:52apply the linear projection component output obtained from processing the scale indicator using the linear projection component to one or more feature maps in the distribution predictor.

31. The apparatus of any of claims 27 to 30, wherein the apparatus is further caused to: apply the output of the linear projection component in channel -wise to an output of a bias operator.

32. The apparatus of any of claims 26 to 31, wherein the apparatus is further caused to: receive, from an encoder, signaling comprising an indication of whether to use the scale indicator as input to a distribution predictor, or to skip the scale indicator and not use the scale indicator as input to the distribution predictor; use the scale indicator as input to the distribution predictor to predict the parameters of the probability distribution for the elements of the latent representation of the at least one image for the resolution level, in response to the signaling received from the encoder indicating to use the scale indicator as input to the distribution predictor; and skip and do not use the scale indicator as input to the distribution predictor to predict the parameters of the probability distribution for the elements of the latent representation of the at least one image for the resolution level, in response to the signaling received from the encoder indicating to skip the scale indicator and not use the scale indicator as input to the distribution predictor.

33. The apparatus of any of claims 26 to 32, wherein the apparatus is further caused to: receive, from an encoder, signaling of one or more scales processed by the distribution predictor.

34. The apparatus of any of claims 26 to 33, wherein an input to the distribution predictor comprises: a resolution of an input image of a codec comprising the distribution predictor,53or a parameter derived from a resolution of an input image of a codec comprising the distribution predictor.

35. The apparatus of any of claims 26 to 34, wherein the apparatus is further caused to: compute a value based on a current scale and a previous scale; wherein the value computed based on the current scale and the previous scale is used as input to the distribution predictor.

36. The apparatus of any of claims 26 to 35, wherein the value computed based on the current scale and the previous scale comprises a difference between the current scale and the previous scale.

37. The apparatus of any of claims 26 to 36, wherein the apparatus is further caused to: derive information from a frame position of an input frame of a codec comprising the distribution predictor; wherein the information derived from the frame position of the input frame of the codec is used as input to the distribution predictor for video compression.

38. The apparatus of claim 37, wherein reconstructed intra frames are used as extra context for the video compression.

39. The apparatus of any of claims 26 to 38, wherein the apparatus is further caused to: update an input tensor that contains elements of the latent representation of the at least one image for the resolution level that have been processed up to but not including a step corresponding to a latent representation partition and predicted values of the elements of the latent representation of the at least one image for the resolution level that have not been processed at the step corresponding to the latent representation partition, to generate an updated input tensor that contains elements of the latent representation of the at least one image for the resolution level that have been processed54up to and including the step corresponding to the latent representation partition, and predicted values of the elements of the latent representation of the at least one image for the resolution level that have not been processed at a step corresponding to a next latent representation partition.

40. The apparatus of any of claims 26 to 39, wherein the apparatus is further caused to: update a mask tensor corresponding to a latent representation partition of the latent representation of the at least one image for the resolution level to generate an updated mask tensor corresponding to a next latent representation partition of the latent representation of the at least one image for the resolution level; wherein the mask tensor indicates positions of the elements of the latent representation of the at least one image for the resolution level that have been processed up to but not including a step corresponding to the latent representation partition; wherein the updated mask tensor indicates positions of the elements of the latent representation of the at least one image for the resolution level that have been processed up to but not including a step corresponding to the next latent representation partition; wherein the next latent representation partition of the latent representation of the at least one image for the resolution level is next after the latent representation partition of the latent representation of the at least one image for the resolution level.

41. The apparatus of any of claims 26 to 40, wherein the predicted parameters of the probability distribution for the elements of the latent representation of the at least one image for the resolution level are obtained based on a latent tensor associated with a previous resolution level that is lower than the resolution level.

42. The apparatus of any of claims 26 to 41, wherein the scale indicator, or information derived from the scale indicator, is used for modifying one or more components, portions, or layers of the distribution predictor.

43. The apparatus of any of claims 26 to 42, wherein the scale indicator, or information derived from the scale indicator, is used for modifying one or more layers of the55distribution predictor.

44. The apparatus of any of claims 26 to 43, wherein the apparatus is further caused to: insert, based on the scale indicator, a transpose convolutional layer with stride greater than 1 into an architecture of the distribution predictor at a predefined position of the distribution predictor.

45. A method comprising: using a distribution predictor to predict parameters of a probability distribution for elements of a latent representation of at least one image for a resolution level; wherein an input to the distribution predictor comprises a scale indicator that indicates information related to the resolution level; and coding the elements of the latent representation of the at least one image for the resolution level using the predicted parameters of the probability distribution into or along a bitstream.

46. A method comprising: decoding, from or along a bitstream, information related to elements of a latent representation of at least one image for a resolution level; determining whether to use a scale indicator that indicates information related to the resolution level as input to a distribution predictor to predict parameters of a probability distribution for the elements of the latent representation of the at least one image for the resolution level; and using the distribution predictor to predict the parameters of the probability distribution for the elements of the latent representation of the at least one image for the resolution level; wherein the scale indicator that indicates information related to the resolution level is used as input to the distribution predictor to predict the parameters of the probabilitydistribution for the elements of the latent representation of the at least one image for the resolution level, in response to determining to use the scale indicator.57

Images