Block-based codec supporting transform coding using machine learning predictors and method of training a machine learning predictor for use in block-based coding
By integrating machine learning predictors to transform coding and optimizing transform coefficient handling, the method improves compression efficiency in block-based video coding, addressing suboptimal rate-distortion issues in traditional methods.
Patent Information
- Application Number
- PCT/EP2026/062715
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-09
- Filing Date
- 2026-05-04
- Publication Date
- 2026-11-12
AI Technical Summary
Traditional block-based video compression methods rely heavily on rate-distortion optimized quantization, which does not effectively incorporate machine learning predictors, leading to suboptimal coding efficiency in terms of compression rate and distortion.
Implementing machine learning predictors to transform coding processes, allowing for improved compression efficiency by directly mapping transform stages from the transform domain to the sample domain, and utilizing reductive and additive offset predictors to adjust transform coefficients, along with a training method that enhances the dequantization function and bitrate cost model for better parameter tuning.
Enhances coding efficiency by improving compression rate and distortion performance through statistical dependency exploitation and optimized transform coefficient handling, resulting in a better tradeoff between implementation effort and rate-distortion relation.
Smart Images

Figure IMGF000098_0001 
Figure IMGF000099_0001 
Figure IMGF000099_0002
Abstract
Description
[0001] Block-based Codec supporting Transform Coding using Machine Learning Predictors and Method of training a Machine Learning Predictor for use in Block-based Coding
[0002] DESCRIPTION
[0003] TECHNICAL FIELD
[0004] Embodiments of the present invention relate to apparatuses for block based encoding or decoding of pictures supporting transform coding using machine learning predictors such as video en / decoders en / decoding pictures of video sequences and methods for block based encoding or decoding of pictures supporting transform coding using machine learning predictors; and further, methods of training a machine learning predictor used in block based encoding or decoding of pictures.
[0005] BACKGROUND
[0006] Traditional transform coding in block-based video compression heavily relies on rate- distortion optimized quantization. In the case of using orthogonal transforms and uniform reconstruction quantizers, the relationship between the quantization step size, the distortion and the expected bitrate is well-understood [Goy01], The quantization and entropy coding of transform coefficients in VVC incorporates many of these aspects as described in [Sch+21c], Recall that the following methods are used for this:
[0007] 1. With a fixed scan order, the position of the final non-zero coefficient in a trans- form block is signalled, thereby omitting the signalling of the preceding zero coefficients.
[0008] 2. Coded block flags are used for signalling if a coefficient subgroup consists entirely of zero coefficient levels or not.
[0009] 3. The binarization of the transform coefficients consists of the sig, g1, par, gt3 flags, the sign flag and truncated Rice and Exponential-Golomb codes for coding coefficient levels larger than three.
[0010] 4. Templates for selecting the context models of the transform coefficient coding are used which are based on the coefficient levels at co-located positions inside the transform block.
[0011] filing version, rm5. Trellis-coded quantization with four states and a parity-based state transition rule is used to exploit the benefits of vector quantization over uniform scalar quantization.
[0012] The aspects in 1 and 2 make use of the energy compaction properties of the family of DST and DCT transforms in image and video compression. Furthermore, the binarization in 3 reflects the assumption that the transformed residual approximately resembles a Laplacian distribution. Finally, the benefits of conditional entropy coding methods in 4 and vector quantization in 5 are exploited with minor computational burden on the throughput of the arithmetic coding engine.
[0013] In practice, the encoder uses the synthesis transform’s inverse to compute the transform coefficients and then performs rate-distortion optimized quantization as described. However, in end-to-end learned transform coding, the encoder is necessarily a part of the training stage and its impact on the expected bitrate and distortion in combination with the decoder must be taken into account. In learned image compression as described in [Bal+21; MS20; Bal+18; MBT18], this is accomplished by optimizing a probability model of the latent variable which is uniformly quantized and transmitted. Additionally, the impact of different quantization approximations is described in works of [Guo+21; TA23; Bor+24], In this disclosure, it has been demonstrated that nonetheless, the coding efficiency of these learned codecs is further improved when some of the aforementioned aspects in 1-5 are implemented on the encoder side after the training stage. Yet, when learning-based approaches are used for obtaining transforms in video compression, these entropy coding methods should be included in the training stage. In particular, a smooth approximation of the quantization needs to take into account the impact of the underlying entropy coding design.
[0014] SUMMARY
[0015] It is the object of the present invention to provide concepts for block-based codecs using transform-based residual coding and / or intra-prediction which are improved in that they lead to improved coding efficiency such as compression rate and compression distortion.
[0016] This is achieved by the subject matter of the independent claims.
[0017] In accordance with a first aspect of the present invention, a block-based codec which uses transform-based residual coding is improved in terms of coding efficiency such as compression rate and / or compression performance. This is achieved in the following manner.
[0018] filing version, rmThe decoder predicts a predicted sample filling of the predetermined block and decodes a synthesis transform block of a prediction residual for the predetermined block from the data stream and subjects the synthesis transform block to a machine learning transform predictor for reconstructing the predetermined block. The machine learning transform predictor transforms, or performs a transform transition of, the synthesis transform block from a transform domain to a sample domain to obtain a sample domain prediction residual. By this measure, the machine learning transform predictor improves the coding efficiency by directly mapping or mapping a proper subset of one or more transform stages (e.g. transformations, or transformation stages) out of a sequence of transform stages (e.g. transformations, or transformation stages) of the synthesis transform block from the transform domain to the sample domain, thereby obtaining the sample domain prediction residual. In other words, the machine learning transform predictor may carry out one or more learned transforms implementing a single transform or any of intermediate transforms belonging to a sequence of transforms (e.g. relating to the transform transition) of the synthesis transform block in a manner providing improved (e.g. increased) coding efficiency such as improved compression rate or improved compression performance (e.g. improved bitrate-distortion). That is, the machine learning transform predictor (e.g. implementable as a neural network) may exploit possible statistical dependencies among samples of the synthesis transform block, representing a transform domain prediction residual, to transform it, or perform the transform transition thereof, for obtaining the sample domain prediction residual by way of improved compression rate and enhanced compression performance. The decoder corrects the predicted sample filling using the sample domain prediction residual obtained by subjecting the synthesis transform block to the machine learning transform predictor. Therefore, the decoder achieves an improved reconstruction of the predetermined block in a manner harmonizing with improved compression efficiency.
[0019] The encoder predicts a predicted sample filling of the predetermined block and encodes an analysis transform block of a prediction residual for the predetermined block into the data stream, wherein the analysis transform block is for reconstructing the predetermined block. For example, the encoder may perform the reconstruction in its prediction loop and / or for R / D optimization purposes. The encoder subjects a synthesis transform block to a machine learning transform predictor. The machine learning transform predictor transforms, or performs a transform transition of, the synthesis transform block from a synthesis transform domain to a sample domain to obtain a sample domain prediction residual. By this measure, the machine learning transform predictor improves the coding efficiency by directly mapping or mapping a proper subset of one or more transform stages (e.g. transformations, or transformation stages) out of a sequence of transform stages (e.g. transformations, or transformation stages) of the
[0020] filing version, rmsynthesis transform block from the transform domain to the sample domain, thereby obtaining the sample domain prediction residual. In other words, the machine learning transform predictor may carry out one or more learned transforms implementing a single transform or any of intermediate transforms belonging to a sequence of transforms (e.g. relating to the transform transition) of the synthesis transform block in a manner providing improved (e.g. increased) coding efficiency such as improved compression rate or improved compression performance (e.g. improved bitrate-distortion). That is, the machine learning transform predictor (e.g. implantable as a neural network) may exploit possible statistical dependencies among samples of the synthesis transform block (e.g. in the prediction loop or as part of a R / D optimization stage), representing a transform domain prediction residual, to transform it, or perform the transform transition thereof, for obtaining the sample domain prediction residual by way of improved compression rate and enhanced compression performance. The encoder corrects the predicted sample filling using the sample domain prediction residual obtained by subjecting the synthesis transform block to the machine learning transform predictor. Therefore, the encoder achieves an improved reconstruction of the predetermined block in a manner harmonizing with improved compression efficiency.
[0021] Embodiments according to the first aspect of the invention provide a decoder for block based decoding of a predetermined block of a picture. The decoder is configured to: predict a predicted sample filling of the predetermined block. The decoder is configured to: decode a synthesis transform block of a prediction residual for the predetermined block from the data stream. The decoder is configured to: reconstruct the predetermined block by subjecting the synthesis transform block to a machine learning transform predictor. The machine learning transform predictor is configured to transform, or perform a transform transition of, the synthesis transform block from a transform domain to a sample domain, to obtain a sample domain prediction residual. The decoder is configured to: reconstruct the predetermined block by correcting the predicted sample filling using the sample domain prediction residual.
[0022] Further embodiments according to the first aspect of the invention provide a decoder, wherein the synthesis transform block may have been obtained an encoder using a further machine learning transform predictor. The further machine learning transform predictor may be configured to transform, or perform a transform transition of, the prediction residual from the sample domain to the transform domain.
[0023] Further embodiments according to the first aspect of the invention provide a decoder which may be additionally configured to: predict the predicted sample filling of the predetermined block based on a machine learning prediction predictor by inputting into the machine learning
[0024] filing version, rmprediction predictor one or more previously decoded samples neighbouring the predetermined block as inputs, and obtaining the predicted sample filling based on an output of the machine learning prediction predictor.
[0025] Further embodiments according to the first aspect provide an encoder for block based encoding of a predetermined block of a picture. The encoder is configured to: predict a predicted sample filling of the predetermined block. The encoder is configured to: encode an analysis transform block of a prediction residual for the predetermined block into the data stream. The analysis transform block is for reconstructing the predetermined block by subjecting a synthesis transform block to a machine learning transform predictor. The machine learning transform predictor is configured to transform or perform a transform transition of the synthesis transform block from a synthesis transform domain to a sample domain to obtain a sample domain prediction residual, correcting the predicted sample filling using the sample domain prediction residual. The analysis transform block is for reconstructing the predetermined block by correcting the predicted sample filling using the sample domain prediction residual. For example, the encoder might actually perform the reconstruction of the predetermined block, namely in the prediction loop and / or for R / D optimization purposes.
[0026] Further embodiments according to the first aspect of the invention provide an encoder which may be additionally configured to: subject the prediction residual for the predetermined block to a further machine learning transform predictor. The further machine learning transform predictor may be configured to transform, or perform a transform transition of, the prediction residual from the sample domain to a analysis transform domain, to obtain the analysis transform block representing a transform domain prediction residual.
[0027] Further embodiments according to the first aspect of the invention provide an encoder which may be additionally configured to: predict the predicted sample filling of the predetermined block based on a machine learning prediction predictor by inputting into the machine learning prediction predictor one or more previously encoded samples neighbouring the predetermined block as inputs, and obtaining the predicted sample filling based on an output of the machine learning prediction predictor.
[0028] In accordance with a second aspect of the invention, an encoder which uses transform-based residual coding is improved in terms of coding efficiency such as compression rate and / or compression performance, when encoding an analysis transform block of a prediction residual by encoding a modified transform block into the data stream, wherein the modified transform block comprises corrected transform coefficients being lower in magnitude than that of the
[0029] filing version, rmanalysis transform block and the corrected transform coefficients of the modified transform block are computed / obtained based on an absolute value of a reductive offset block, which is computed using a machine learning reductive offset predictor, and the signs of the transform coefficients of the analysis transform block. By this measure, the modified transform block being lower in magnitude, since it is obtained / computed by incorporating the absolute value of the reductive offset block and the signs of the transform coefficients of the analysis transform block, results in an improved compression rate (e.g. a positive impact on the bitrate cost, i.e. a reduced bitrate cost). In other words, the transform coefficients representing the analysis transform block are, being subjected to the machine learning reductive offset predictor, reduced in magnitude (e.g. depreciated or lessened in value of its transform coefficients), thus resulting in an improved tradeoff between implementation effort and a good rate-distortion relation. Further, as part of a R / D optimizer and / or a prediction loop of the encoder, the encoder is improved in terms of coding efficiency such as compression rate and / or compression performance, when obtaining (e.g. computing) an improved transform block based on a sum of the modified transform block and an additive offset block, the latter being obtained (e.g. computed) based on the modified transform block using a machine learning improvement predictor, and subjecting the improved transform block to a synthesis transform block, and correcting the predicted sampled filling using the prediction residual obtained by the synthesis transform block. By this measure, reconstructed transform coefficients, obtained from the modified transform block, are improved additively (i.e. the transform coefficients of the improved transform block being greater in magnitude than that of the modified transform block) by the additive offset block obtained (e.g. computed) using the machine learning improvement predictor, resulting in an improved compression rate (e.g. a positive impact on the bitrate cost, i.e. a reduced bitrate cost). In other words, the transform coefficients representing the modified transform block, being subjected to the machine learning improvement predictor, are improved in magnitude (e.g. increased or enhanced in value of its transform coefficients), prior to being subjected to the synthesis transform block. The synthesis transform block provides the prediction residual which corrects the predicted sample filling as part of the encoder’s prediction loop and / or of a R / D optimizer, thereby resulting in an improved tradeoff between implementation effort and a good rate-distortion relation.
[0030] Embodiments according to the second aspect of the invention provide an encoder for block based encoding of a predetermined block of a picture. The encoder is configured to: predict a predicted sample filling of the predetermined block. The encoder is configured to: encode an analysis transform block of a prediction residual for the predetermined block into the data stream by encoding transform coefficients of the analysis transform block of the prediction residual by subjecting the analysis transform block, accompanied with the predicted sample
[0031] filing version, rmfilling, to a machine learning reductive offset predictor so as to obtain a reductive offset block. The encoder is configured to: encode an analysis transform block of a prediction residual for the predetermined block into the data stream by encoding transform coefficients of the analysis transform block of the prediction residual by correcting each transform coefficient of the analysis transform block based on a corresponding absolute value of the reductive offset block and the sign of the transform coefficient to obtain a corresponding corrected transform coefficient of a modified transform block lower in magnitude. The encoder is configured to: encode an analysis transform block of a prediction residual for the predetermined block into the data stream by encoding transform coefficients of the analysis transform block of the prediction residual by encoding the modified transform block into the data stream. The encoder is configured to, as part of a R / D optimizer and / or a prediction loop of the encoder, subjecting the modified transform block, accompanied with the predicted sample filling, to a machine learning improvement predictor so as to obtain an additive offset block. The encoder is configured to, as part of the R / D optimizer and / or the prediction loop of the encoder, forming a sum of the modified transform block and the additive offset block to obtain an improved transform block. The encoder is configured to, as part of the R / D optimizer and / or the prediction loop of the encoder, subjecting the improved transform block to a synthesis transform block. The encoder is configured to, as part of the R / D optimizer and / or the prediction loop of the encoder, correcting the predicted sample filling using the prediction residual.
[0032] In accordance with a third aspect of the invention, a method of training a machine learning predictor to be used for block based coding is improved to allow for improved coding efficiency such as compression rate and / or compression performance, when subjecting the initial set of parameters of the machine learning predictor and obtaining an updated set of parameters based on a dequantization function and / or a bitrate cost model, wherein the dequantization function provides an enhanced or improved approximation of a dequantization process involving transform coefficients of a transform block for a block based codec, the approximation being enhanced or improved in terms of, at least, its differentiability (e.g. smoothness of its derivative(s)) and associated coding cost; and wherein the bitrate cost model provides an enhanced or improved approximation of a bitrate cost of coding transform coefficients of a transform block for a block based codec, the approximation being enhanced or improved in terms of, at least, its differentiability (e.g. smoothness of its derivative(s)) and associated coding cost. The dequantization function retains original values of the transform coefficients if a corresponding quantized value is non-zero and disregards those transform coefficients if a corresponding quantized value is non-zero, thus resulting in the training stage of the machine learning predictor in which transform coding is harmonized in a manner with improved compression performance such as achieving an improved bitrate-distortion relation, and filing version, rmtherewith facilitating the training of the machine learning predictor. The bitrate cost model obtains / computes a bitrate cost associated with each transform coefficient of the transform block using a density function, (e.g. as a parameterized probability density function), whose attributes comprise a scan index of the transform coefficient, a last scan index of the transform block indicative of a last non-zero quantized transform coefficient and a quantization parameter of the predetermined block, and thus the bitrate cost model takes into account energy compaction, distribution of the transform coefficients and quantization of the transform block. By this measure, the bitrate cost model estimates the bitrate cost of the transform block in a manner which exploits properties of the transform block to achieve improved compression performance, such as an improved rate distortion relation, and low implementation effort, and therewith facilitating the training of the machine learning predictor.
[0033] Embodiments according to the third aspect of the invention provide a method of training a machine learning predictor configured to be used for block based coding of a predetermined block of a picture. The method comprises: providing an initial set of parameters of the machine learning predictor. The method comprises: subjecting the initial set of parameters to a training stage for obtaining an updated set of parameters based on: a dequantization function and / or a bitrate cost model. The dequantization function is configured to, for each transform coefficient of a set of transform coefficients of a transform block of a prediction residual for the predetermined block, each transform coefficient having an original value, set a dequantized value of the transform coefficient to the original value of the transform coefficient if a quantized value of the transform coefficient is non-zero, and set a dequantized value of the transform coefficient to zero if a quantized value of the transform coefficient is zero, wherein the quantized value of the transform coefficient is derived by a quantizer based on a quantization parameter of the predetermined block. The bitrate cost model is configured to, for each transform coefficient of a set of transform coefficients of a transform block of a prediction residual for the predetermined block, obtain a bitrate cost associated with the transform block (i.e. bitrate cost associated with coding the transform block into / from the data stream) using a density function. The density function is configured to output a probability value associated with a quantization index of the transform coefficient. Attributes of the density function comprise a scan index of the transform coefficient, a last scan index associated with the transform block, which is indicative of a last non-zero-quantized transform coefficient, and a quantization parameter of the predetermined block.
[0034] Further embodiments according to the third aspect of the invention provide a method for training a machine learning predictor. The machine learning predictor as a machine learning transform predictor may be configured to take, as input, a transform block of a prediction residual for the predetermined block, and / or a predicted sample filling of the predetermined
[0035] filing version, rmblock, and provide, as output, a sample domain prediction residual; and / or the machine learning predictor as a machine learning transform predictor may be configured to take, as input, a sample domain prediction residual for the predetermined block, and / or a predicted sample filling of the predetermined block, and provide, as output, a transform block representing a transform domain prediction residual; and / or the machine learning predictor as a machine learning prediction predictor may be configured to take, as input, values of one or more previously coded samples neighbouring the predetermined block, and provide, as output, a predicted sample filling of the predetermined block; and / or the machine learning predictor as a machine learning reductive offset predictor is configured to take, as input, an analysis transform block of a prediction residual for the predetermined block and a predicted sample filling of the predetermined block, and provide, as output, a reductive offset block, wherein the reductive offset block is configured to correct each transform coefficient of the analysis transform block based on corresponding absolute value of the reductive offset block and the sign of the transform coefficient for obtaining a corresponding corrected transform coefficient of a modified transform block lower in magnitude; and / or the machine learning predictor as a machine learning improvement predictor may be configured to take, as input, an analysis transform block of a prediction residual for the predetermined block and a predicted sample filling of the predetermined block, and provide, as output, an additive offset block, wherein the additive offset block is configured to correct each transform coefficient of the transform block based on a corresponding value of the additive offset block for obtaining a corresponding improved transform coefficient of an improved transform block greater in magnitude.
[0036] Further embodiments according to the third aspect of the invention provide a method for training a machine learning predictor. The method may further comprise: as an epoch of the training stage, computing a training error according to a training objective. The training error may depend on the output of the machine learning predictor, and / or the dequantization function, and / or the bitrate cost model. The method may comprise: as the epoch of the training stage, updating the initial set of parameters so as to obtain a subsequent set of parameters related to the corresponding training error being lower in value while leaving parameters of the bitrate cost model unchanged. The method may comprise outputting the subsequent set of parameters as inputs for a subsequent epoch of the training stage.
[0037] Further embodiments according to the invention are defined by the subject matter of the dependent claims of the present application.
[0038] BRIEF DESCRIPTION OF THE FIGURES
[0039] filing version, rmEmbodiments of the present disclosure are described in more detail below with respect to the figures, in which:
[0040] Fig. 1 illustrates an example of a transform-based encoder;
[0041] Fig. 2 illustrates an example of a transform-based decoder;
[0042] Fig. 3 illustrates an example of block partitioning;
[0043] Fig. 4 illustrates an implementation of a decoder according to embodiments of the first aspect;
[0044] Fig. 5 illustrates an implementation of an encoder according to embodiments of the first aspect;
[0045] Fig. 6 illustrates neural networks associated with machine learning predictors according to embodiments of the first aspect;
[0046] Fig. 7 illustrates an implementation of an encoder according to embodiments of the second aspect;
[0047] Fig. 8 illustrates neural networks associated with machine learning predictors according to embodiments of the second aspect;
[0048] Fig. 9 illustrates an implementation of an encoder according to embodiments of the first aspect in combination with the second aspect;
[0049] Fig. 10 illustrates an implementation of a method of training a machine learning predictor according to embodiments of the third aspect;
[0050] Fig. 11 illustrates an implementation of a method of training a machine learning predictor according to embodiments of the third aspect;
[0051] Fig. 12 illustrates an implementation of a method of training a machine learning predictor according to embodiments of the third aspect;
[0052] filing version, rmFig. 13 illustrates an implementation of a method of training a machine learning predictor according to embodiments of the third aspect;
[0053] Fig. 14 illustrates a quantitative comparison of bitrate cost with and without using an implementation of a method of training a machine learning predictor according to embodiments of the third aspect;
[0054] Fig. 15 illustrates an architecture of a neural network as an implementation of the machine learning transform predictor in an encoder according to embodiments of the first aspect; and
[0055] Fig. 16 illustrates an architecture of a neural network as an implementation of the machine learning prediction predictor in an encoder according to embodiments of the first aspect.
[0056] DETAILED DESCRIPTION OF THE FIGURES
[0057] In the above sections, specific embodiments have been summarily described. In the following, further embodiments are described which are based on the above thoughts and ideas, but are broadened such as by provision of details. Before describing the further embodiments and details thereof, however, a general codec framework is described, into which all embodiments for decoder and encoder described herein could be built into, or with which all these embodiments might be combined.
[0058] Note that in the drawings the same or similar elements or elements that have the same or similar functionality have the same reference signs assigned or are identified with the same name. In the following description, a plurality of details is set forth to provide a thorough explanation of embodiments of the disclosure. However, it will be apparent to one skilled in the art that other embodiments may be implemented without these specific details. In addition, features of the different embodiments described herein may be combined with each other, unless specifically noted otherwise.
[0059] The description presents in the following two sorts, or categories, of embodiments, namely some for which experiments have been conducted which are then also presented and some which are derived from the presentation of the former ones. In particular, the former embodiments have been derived by modifying a particular block-based codec using transform-
[0060] filing version, rmbased residual codec and are more specific. The other embodiments form a kind of abstraction or broadening of the more specific embodiments.
[0061] The following description preliminarily switches to a description of the broader or abstracted embodiments. Thereinafter, the description is resumed with the presentation of the more specific embodiments. However, it should be mentioned that any detail(s) described with respect to one embodiment may individually or in combination be used to form a new embodiment when transferred onto a different embodiment. The description starts with a presentation of a description of an encoder and a decoder of a block-based predictive codec for coding pictures of a video, for instance as a video sequence, in order to form an example for a coding framework into which embodiments of the present invention may be built in. The respective encoder and decoder are described with respect to Fig. 1, Fig. 2, and Fig. 3. Thereinafter the description of the broadened embodiments of the concept of the present invention is presented. While being, as said, combinable with Fig. 1 - 3, the embodiments described before and afterwards may, however, also be used to form encoders and decoders not operating according to the coding framework underlying the encoder and decoder of Fig.
[0062] 1, and Fig. 2.
[0063] Fig. 1 shows an apparatus for predictively coding a picture 12 into a data stream 40 exemplarily using transform-based residual coding. The apparatus, or encoder, is indicated using reference sign 10. Fig. 2 shows a corresponding decoder 50, i.e. an apparatus 50 configured to predictively decode the picture 12’ from the data stream 40 also using transform-based residual decoding, wherein the apostrophe has been used to indicate that the picture 12’ as reconstructed by the decoder 50 deviates from picture 12 originally encoded by apparatus 10 in terms of coding loss introduced by a quantization of the prediction residual signal.
[0064] The encoder 10 is configured to subject the prediction residual signal to spatial-to-spectral transformation and to encode the prediction residual signal, thus obtained, into the data stream 40. Likewise, the decoder 50 is configured to decode the prediction residual signal from the data stream 40 and subject the prediction residual signal thus obtained to spectral-to-spatial transformation.
[0065] Internally, the encoder 10 may comprise a prediction residual signal former 16 which generates a prediction residual 17 so as to measure a deviation of a prediction signal 19 from the original signal, i.e. from the picture 12. The prediction residual signal former 16 may, for instance, be a subtractor which subtracts the prediction signal from the original signal, i.e. from the picture 12. The encoder 10 then further comprises a transformer 20 which subjects the prediction
[0066] filing version, rmresidual signal 17 to a sample domain to transform domain transform / transformation, for example a spatial-to-spectral transformation, to obtain a transform domain prediction residual 17’, for example a spectral-domain prediction residual signal 17’, which is then subject to quantization by a quantizer 26, also comprised by the encoder 10. The thus quantized prediction residual signal 17” is coded into bitstream 40. To this end, encoder 10 may optionally comprise an entropy coder 36 which entropy codes the prediction residual signal as transformed and quantized into data stream 40. The prediction signal 19 is generated by a prediction stage 37 of encoder 10 on the basis of the prediction residual signal 17” encoded into, and decodable from, data stream 40. To this end, the prediction stage 37 may internally, as is shown in Fig. 1, comprise a dequantizer 28 which dequantizes prediction residual signal 17” so as to gain transform domain prediction residual signal 17’”, or for example spectral-domain prediction residual signal 17’”, which corresponds to signal 17’ except for quantization loss, followed by an inverse transformer 32 which subjects the latter prediction residual signal 17’” to an inverse transformation, i.e. a transform domain to sample domain transform / transformation, or for example, a spectral-to-spatial transformation, to obtain prediction residual signal 17””, which corresponds to the original prediction residual signal 17 except for quantization loss. A combiner 64 of the prediction stage 37 then recombines, such as by addition, the prediction signal 19 and the prediction residual signal 17”” so as to obtain a reconstructed signal 21, i.e. a reconstruction of the original signal 12. Reconstructed signal 21 may correspond to signal 12’. A prediction module 14 of prediction stage 37 then generates the prediction signal 19 on the basis of signal 21 by using, for instance, spatial prediction, i.e. intra-picture prediction, and / or temporal prediction, i.e. inter-picture prediction.
[0067] Likewise, Fig. 2 shows a decoder 50, which may be internally composed of components corresponding to, and interconnected in a manner corresponding to, prediction stage 37. In particular, entropy decoder 60 of decoder 50 may entropy decode the quantized transform domain prediction residual signal 17”, or for example spectral-domain prediction residual signal 17”, from the data stream, whereupon dequantizer 28, inverse transformer 32, combiner 64 and prediction module 14, interconnected and cooperating in the manner described above with respect to the modules of prediction stage 37, recover the reconstructed signal on the basis of prediction residual signal 17” so that, as shown in Fig. 2, the output of combiner 64 results in the reconstructed signal, namely picture 12’.
[0068] Although not specifically described above, it is readily clear that the encoder 10 may set some coding parameters including, for instance, prediction modes, motion parameters and the like, according to some optimization scheme such as, for instance, in a manner optimizing some rate and distortion related criterion, i.e. coding cost. For example, encoder 10 and decoder 50
[0069] filing version, rmand the corresponding modules 14 respectively, may support different prediction modes such as intra-coding modes and inter-coding modes. The granularity at which encoder and decoder switch between these prediction mode types may correspond to a subdivision of picture 12 and 12’, respectively, into coding segments or coding blocks. In units of these coding segments, for instance, the picture may be subdivided into blocks being intra-coded and blocks being inter-coded. Intra-coded blocks are predicted on the basis of a spatial, already coded / decoded neighborhood of the respective block as is outlined in more detail below.
[0070] Several intra-coding modes may exist and be selected for a respective intra-coded segment including directional or angular intra-coding modes according to which the respective segment is filled by extrapolating the sample values of the neighborhood along a certain direction which is specific for the respective directional intra-coding mode, into the respective intra-coded segment. The intra-coding modes may, for instance, also comprise one or more further modes such as a DC coding mode, according to which the prediction for the respective intra-coded block assigns a DC value to all samples within the respective intra-coded segment, and / or a planar intra-coding mode according to which the prediction of the respective block is approximated or determined to be a spatial distribution of sample values described by a two-dimensional linear function over the sample positions of the respective intra-coded block with driving tilt and offset of the plane defined by the two-dimensional linear function on the basis of the neighboring samples. Compared thereto, inter-coded blocks may be predicted, for instance, temporally.
[0071] For inter-coded blocks, motion vectors may be signaled within the data stream, the motion vectors indicating the spatial displacement of the portion of a previously coded picture of the video to which picture 12 belongs, at which the previously coded / decoded picture is sampled in order to obtain the prediction signal for the respective inter-coded block. This means, in addition to the residual signal coding comprised by data stream 40, such as the entropy-coded transform coefficient levels representing the quantized transform domain prediction residual signal 17”’, or for example the quantized spectral-domain prediction residual signal 17”, data stream 40 may have encoded thereinto coding mode parameters for assigning the coding modes to the various blocks, prediction parameters for some of the blocks, such as motion parameters for inter-coded segments, and optional further parameters such as parameters for controlling and signaling the subdivision of picture 12 and 12’, respectively, into the segments. The decoder 50 uses these parameters to subdivide the picture in the same manner as the encoder did, to assign the same prediction modes to the segments, and to perform the same prediction to result in the same prediction signal.
[0072] filing version, rmFig. 3 illustrates the relationship between the reconstructed signal, i.e. the reconstructed picture 12’, on the one hand, and the combination of the prediction residual signal 17”” as signaled in the data stream 40, and the prediction signal 19, on the other hand. As already denoted above, the combination may be an addition.
[0073] The prediction signal 19 is illustrated in Fig. 3 as a subdivision of the picture area into intracoded blocks which are illustratively indicated using hatching, and inter-coded blocks which are illustratively indicated not-hatched. The subdivision may be any subdivision, such as a regular subdivision of the picture area into rows and columns of square blocks or non-square blocks, or a multi-tree subdivision of picture 12 from a tree root block into a plurality of leaf blocks of varying size, such as a quadtree subdivision or the like, wherein a mixture thereof is illustrated in Fig. 3 in which the picture area is first subdivided into rows and columns of tree root blocks which are then further subdivided in accordance with a recursive multi-tree subdivisioning into one or more leaf blocks.
[0074] Again, data stream 40 may have an intra-coding mode coded thereinto for intra-coded blocks 80, which assigns one of several supported intra-coding modes to the respective intra-coded block 80. For inter-coded blocks 82, the data stream 40 may have one or more motion parameters coded thereinto. Generally speaking, inter-coded blocks 82 are not restricted to being temporally coded. Alternatively, inter-coded blocks 82 may be any block predicted from previously coded portions beyond the current picture 12 itself, such as previously coded pictures of a video to which picture 12 belongs, or picture of another view or an hierarchically lower layer in the case of encoder and decoder being scalable encoders and decoders, respectively.
[0075] The prediction residual signal 17”” in Fig. 3 is also illustrated as a subdivision of the picture area into blocks 84. These blocks might be called transform blocks in order to distinguish same from the coding blocks 80 and 82. In effect, Fig. 3 illustrates that encoder 10 and decoder 50 may use two different subdivisions of picture 12 and picture 12’, respectively, into blocks, namely one subdivisioning into coding blocks 80 and 82, respectively, and another subdivision into transform blocks 84. Both subdivisions might be the same, i.e. each coding block 80 and 82, may concurrently form a transform block 84, but Fig. 3 illustrates the case where, for instance, a subdivision into transform blocks 84 forms an extension of the subdivision into coding blocks 80, 82 so that any border between two blocks of blocks 80 and 82 overlays a border between two blocks 84, or alternatively speaking each block 80, 82 either coincides with one of the transform blocks 84 or coincides with a cluster of transform blocks 84. However, the subdivisions may also be determined or selected independent from each other so that filing version, rmtransform blocks 84 could alternatively cross block borders between blocks 80, 82. As far as the subdivision into transform blocks 84 is concerned, similar statements are thus true as those brought forward with respect to the subdivision into blocks 80, 82, i.e. the blocks 84 may be the result of a regular subdivision of picture area into blocks (with or without arrangement into rows and columns), the result of a recursive multi-tree subdivisioning of the picture area, or a combination thereof or any other sort of blockation. Just as an aside, it is noted that blocks 80, 82 and 84 are not restricted to being of quadratic, rectangular or any other shape.
[0076] Fig. 3 further illustrates that the combination of the prediction signal 19 and the prediction residual signal 17”” directly results in the reconstructed signal 12’. However, it should be noted that more than one prediction signal 19 may be combined with the prediction residual signal 17”” to result into picture 12’ in accordance with alternative embodiments.
[0077] In Fig. 3, the transform blocks 84 shall have the following significance. Analysis transform block (or e.g. transformer) 20 and synthesis transform block (e.g. inverse transformer) 32 perform their transformations in units of these transform blocks 84. For instance, many codecs use some sort of DST or DCT for all transform blocks 84. Some codecs allow for skipping the transformation so that, for some of the transform blocks 84, the prediction residual signal is coded in the signal domain, or for example sample domain or spatial domain, directly. However, in accordance with embodiments described below, encoder 10 and decoder 50 are configured in such a manner that they support several transforms. For example, the transformations supported by encoder 10 and decoder 50 could comprise:
[0078] • DCT-II (or DCT-III), where DCT stands for Discrete Cosine Transform
[0079] • DST-IV, where DST stands for Discrete Sine Transform
[0080] • DCT-IV
[0081] • DST-VII
[0082] • Identity Transformation (IT)
[0083] • Low Frequency Non-Separable Transform (LFNST)
[0084] • Non-separable Primary Transform (NSPT)
[0085] Naturally, while the analysis transform block (e.g. or transformer) 20 would support all of the forward transform versions of these transforms, the decoder 50 or the synthesis transform block (e.g. or inverse transformer) 32 would support the corresponding backward or inverse versions thereof:
[0086] • Inverse DCT-II (or inverse DCT-III)
[0087] • Inverse DST-IV
[0088] filing version, rm• Inverse DCT-IV
[0089] • Inverse DST-VII
[0090] • Identity Transformation (IT)
[0091] • Inverse Low Frequency Non-Separable Transform (LFNST)
[0092] • Inverse Non-separable Primary Transform (NSPT)
[0093] The subsequent description provides more details on which transforms could be supported by encoder 10 and decoder 50. In any case, it should be noted that the set of supported transforms may comprise distinct scenarios, for instance a scenario in which merely one transform such as one a transform relating, or bridging, from transform domain to sample domain, or for example spectral-to-spatial, or a transform relating, or bridging, from sample domain to transform domain, or for example spatial-to-spectral transform, and for instance, scenarios in which more than one transform is employed as part of, or in, a sequence of transforms relating, or bridging, from transform domain to sample domain, or for example spectral-to-spatial domain or from sample domain to transform domain, or for example spatial-to-spectral domain.
[0094] As already outlined above, Fig. 1, Fig. 2 and Fig. 3 have been presented as an example where the inventive concept described further below may be implemented in order to form specific examples for encoders and decoders according to the present application. Insofar, the encoder and decoder of Fig. 1, and Fig. 2, respectively, may represent possible implementations of the encoders and decoders described herein below. Fig. 1, and Fig. 2 are, however, only examples. An encoder according to embodiments of the present application may, however, perform encoding of a picture 12 using the concept outlined in more detail below and being different from the encoder of Fig. 1 such as, for instance, in that same is no video encoder, but a still picture encoder, in that same does not support inter-prediction, or in that the sub-division into blocks 80 is performed in a manner different than exemplified in Fig. 3. Likewise, decoders according to embodiments of the present application may perform block-based decoding of picture 12’ from data stream 40 using the coding concept further outlined below, but may differ, for instance, from the decoder 50 of Fig. 2 in that same is no video decoder, but a still picture decoder, in that same does not support intra-prediction, or in that same sub-divides picture 12’ into blocks in a manner different than described with respect to Fig. 3, for instance.
[0095] As illustrated in Fig. 2, decoder 50 may further comprise a filtering module 68, which filters the reconstructed signal 12’, the prediction 14 being performed based on the filtered reconstructed signal 12’. Similarly, encoder 10 of Fig. 1 may comprise a filtering module 68, which may perform the same filtering as filtering module 68 of decoder 50, in the prediction stage 37 to filter the reconstructed signal 21. As the filtering is performed in the prediction loop provided
[0096] filing version, rmby prediction stage 37 (e.g., in combination with operator 16, transformer 20, and quantizer 26), the filtering by filtering module 68 may be referred to as in-loop filtering. Accordingly, embodiments of the invention may optionally be implemented as described with respect to Fig.
[0097] 1, 2, and 3.
[0098] In the following, embodiments of the invention are described, which may optionally be implemented as described with respect to Fig. 1, Fig. 2, and / or Fig. 3, wherein the features described above may be combined with the embodiments described below individually or in combination with each other. For instance, in-loop filtering 68 might be missing. Further, interprediction might not be used, or, vice versa, intra-prediction.
[0099] Fig. 4 shows an embodiment of a decoder 50 while Fig. 5 shows an embodiment of an encoder 10 wherein the decoder 50 and the encoder 10 may be adapted to constitute a block based codec for block based coding of a predetermined block 13 of a picture 12. At first, details of the decoder 50 for block based decoding of the predetermined block 13 of the picture 12 will be described followed by details of the encoder 10 for block based encoding of the predetermined block 13 of the picture 12.
[0100] As previously indicated, the decoder may be a video decoder or a picture decoder which, in addition to the details set out with respect to Fig. 4, comprises one or more of the functionalities described above with respect to Fig. 1 to 3.
[0101] Especially, the decoder 50 of. Fig. 4 supports, for instance, a certain coding mode or coding tool which is illustrated in Fig. 4 for the predetermined block 13 of the picture 12. In line with what has been already been stated earlier, this picture 12 may either be a part of a video sequence or be a part, or possibly a whole, of an image such as a standalone still image.
[0102] As already mentioned, the decoder 50 performs the decoding in a block-wise or block-based manner. That is, the decoder 50 subdivides the picture 12 into one or more blocks, in units of which the decoder 50 decodes the picture 12 from the data stream 40. Generally, the subdivision may end up into one or more blocks of constant size such as an array of blocks arranged in rows and columns or into one or more blocks of different block sizes such as derived by use of a hierarchical multi-tree subdivisioning applied to the whole picture area of picture 12, or derived from a pre-partitioning of picture 12 into an array of tree blocks which are then subject to multi-tree subdivisioning. The hierarchical subdivision information may be signalled in the data stream 40. It is noted that these examples shall not be considered as excluding other possible approaches of subdivisioning the picture 12 into one or more blocks.
[0103] filing version, rmThus, the predetermined block 13 which the decoder 50 decodes may be formed from the picture 12 using such or similar approaches.
[0104] Note that the predetermined block 13 may by a quadratic block or a rectangular or non-square block, while any other shape may be used as well.
[0105] The decoder 50 predicts 14 a predicted sample filling of the predetermined block 13 using an intra-prediction mode which the decoder 50 derives from the data stream 40, such as decoding an index of the intra-prediction mode from the data stream 40. This prediction has been denoted by way of a prediction module 14 in Figure 4.
[0106] Alternatively, the predicted sample filling may be predicted using inter-prediction.
[0107] In accordance with embodiments, the predicted sample filling of the predetermined block 13 might be predicted by selecting a selected intra-prediction mode out of different intra-prediction modes (e.g. a plurality of intra-prediction modes) signalled in the data stream 40, which the decoder 50 supports. In accordance with embodiments, the different intra-prediction modes (e.g. a plurality of intra-prediction modes) may be known, or existing, intra-prediction modes made available to the decoder 50 in the data stream 40.
[0108] Examples of the different intra-prediction modes supported by the decoder 50 include, in accordance with embodiments, one or more of a DC mode, a planar mode, a plurality of angular modes, one or more matrix-based intra prediction (MIP) modes, according to each of which the predicted sample filling is obtained by a multiplication between a matrix associated with the respective matrix-based intra prediction mode and a vector obtained from sample values in a neighborhood of the predetermined block, and one or more intra-subpartition modes, according to each of which the predicted sample filling is obtained by subdividing the predetermined block into subpartitions and intra-predicting the subpartitions sequentially with correcting a subpartition using the prediction residual before intra-predicting a next subpartition. Any combination of the just-mentioned modes shall form the supported set of intra-prediction modes of the decoder 50.
[0109] Note that, in any of the just-described intra-prediction modes, one or more previously decoded samples neighboring the predetermined block 13 based on which the predicted sample filling is predicted need not to be directly adjacent to the predetermined block 13. Rather, the neighborhood, or equivalently described a region in the vicinity of the predetermined block 13, within which the decoded samples are located based on which the predicted sample filling is
[0110] filing version, rmpredicted may also cover or merely cover the one or more previously decoded samples which are not directly adjacent to the block boundary. The previously decoded samples based on which the intra-prediction is performed may be located using a certain criterion such as, for example, one locating same in terms of a specific maximum distance from the block boundary (e.g. boundary of the predetermined block 13) or the like. That is, for instance, the neighborhood may denote a template of previously decoded samples each having a distance from the block boundary of block 13 which is equal to or smaller than a specific distance and this specific distance may be defined by default or may be transmitted in the data stream 40.
[0111] The one or more intra-subpartition modes may subdivide the predetermined block 13 into subpartitions along a first direction (e.g. a horizontal direction) and / or along a second direction (e.g. a vertical direction). In case of subdividing only along one direction, the subpartitions might be as wide as the block 13 in a direction perpendicular to the direction of subdividing the block 13 into the partitions. The sub-partitions might then be intra-predicted and corrected using the block’s prediction residual sequentially, thereby improving the intra predictor for subsequently intra predicted sub-partitions.
[0112] Again, the illustrations provided in Fig. 4 are merely an example. Decoder 50 might support further intra-prediction modes or fewer or a different collection. Further, it is feasible that the decoder 50 may select more than one intra-prediction mode for predicting the predicted sample filling so as to perform the prediction using multiple hypotheses. For the selection of the intra prediction mode, the decoder 50 may decode an index from the data stream 40 for block 13 which index is indicative of the selected intra prediction mode.
[0113] The predicted sample filling is generated using the selected intra-prediction mode based on the values of already decoded samples (e.g. the one or more previously decoded sample, or equivalently also referred to as reference samples; e.g. these decoded samples or reference samples may be available to the decoder 50 via the decoded picture buffer) in the neighborhood of the predetermined block 13. As described, these samples in area may directly abut the circumference, or boundary, of block 13, but this may additionally or alternatively comprise other samples. The already decoded samples in the area may be adjacent, or located next, to the predetermined block along a left-hand boundary of the predetermined block 13 and a top boundary of the predetermined block 13.
[0114] Again, concepts of decoder 50 with respect to an intra-predicted block have been described so far, these concepts would be readily transferable onto inter-predicted blocks, so as to be configured to apply these concepts onto such inter-predicted block alternatively or additionally.
[0115] filing version, rmFurther, the decoder 50 may use intra-prediction in combination with inter-prediction to predict the predicted sample filling 18. For example, the decoder 50 may be configured to combine a predictor resulting from intra-prediction with one obtained by inter-prediction. Details of such a predictor such as a ML prediction predictor for intra-prediction will be described later in this disclosure. For inter-prediction, the decoder 50 may be configured to decode information on motion vector(s) for block 13 from the data stream so as to perform motion-compensated prediction of block 13.
[0116] The decoder 50 decodes a synthesis transform block 32 of a prediction residual for the predetermined block 13 from the data stream 40. The synthesis transform block 32 is denoted by T-1as an example in Fig. 4. The synthesis transform block 32 may relate to a synthesis transform (or transformation or mapping) (e.g. or an inverse transform) of the prediction residual of the predetermined block 13. Decoding the synthesis transform block 32 means that transform coefficients of the synthesis transform block 32, which are transmitted in the data stream and available to the decoder are decoded from the data stream 40.
[0117] The decoder 50 reconstructs the predetermined block 13 by subjecting the synthesis transform block 32 to a machine learning transform predictor 34, which has been denoted by β in Fig. 4 as an example and further in the disclosure. For example, parameters of the machine learning transform predictor 34 may be denoted by ψβelsewhere in the disclosure, regardless of its specific implementation. It should be noted that block 34 in Fig. 4 denotes that the corresponding transform (e.g. transformation or mapping) of the synthesis transform block 32 might be implemented by the ML transform predictor 34 rather than influencing or existing in addition to the transforms (or, for example, transformations or mappings) of the synthesis transform block 32 to which the ML transform predictor 34 is attached to.
[0118] The ML transform predictor 34 is configured to transform 35 (e.g. to carry out a transform or transformation or a mapping of) of the synthesis transform block 32 from a transform domain to a sample domain to obtain a sample domain prediction residual.
[0119] Alternatively, the ML transform predictor 34 is configured to perform a transform transition 39 of the synthesis transform block 32 from the transform domain to the sample domain to obtain the sample domain prediction residual. The transform transition 39 meaning a proper subset of one or more transformation stages (e.g. or transformations or mappings) out of a sequence (e.g. or a chain) of transform stages (e.g. or transformations or mappings) leading from the transform domain to the sample domain (e.g. culminating in the sample domain). It is indicated here that the two blocks, i.e. the transform 35 and the transform transition 39, have been
[0120] filing version, rmillustrated using dashed lines, without preferring one over the other, for the purpose of representing the alternatives which the ML transform predictor 34 subjects the synthesis transform block 32 to.
[0121] The transform domain here relates to the domain of the transform coefficients signalled in the data stream 40 representing (e.g. or constituting, or forming) the synthesis transform block 32. The sample domain here relates to the domain of the samples of the predetermined block 13 reconstructable by the decoder 50.
[0122] The decoder 50 then reconstructs the predetermined block 13 by correcting the predicted sample filling of the predetermined block 13 using the sample domain prediction residual. In particular, as seen in Fig. 4, the sample domain prediction residual obtained by, or as a result of, the synthesis transform block 32, which is accompanied by the ML transform predictor 34 implementing it, is combined with the predicted sample filling obtained by, or as a result of, the prediction module 14 (which is optionally accompanied by a machine learning prediction predictor 36, details of which will be described later in this disclosure) by a combiner or a combiner unit 64, thereby correcting the predicted sample filling.
[0123] For instance, in accordance with embodiments, the synthesis transform block 32 may have been obtained by an encoder 10 using a further machine learning transform predictor 38 (e.g. a), which is configured to transform, or perform a transform transition of, the prediction residual from the sample domain to the transform domain. In other words, this further ML predictor 38 may be a predictor, or ML predictor, different from the corresponding decoder side ML predictor 36. Details of such an encoder 10 using the further machine learning transform predictor 38 (e.g. a) will be described alongside Fig. 5 further in the present disclosure.
[0124] In accordance with embodiments, the ML transform predictor 34 may be non-linear with respect to the synthesis transform block 32. This means, for example, that the ML transform predictor 34 may be implemented using concepts which exploit a non-linear dependence on the synthesis transform block 32. In particular, the ML transform predictor 34 may be implemented using concepts which exploit a non-linear dependence on the transform coefficients (e.g. a set of transform coefficients) of the synthesis transform block 32. An example of such concepts is provided in the form of neural networks further in the disclosure, which may optionally be trainable using training data.
[0125] Further, in accordance with embodiments, the ML transform predictor 34 may be configured to additionally take as input the one or more previously decoded samples neighbouring the
[0126] filing version, rmpredetermined block 13. In other words, the inputs (or for example, attributes, or equivalently, arguments) of the ML transform predictor 34 may comprise both the one or more previously decoded sample neighbouring the predetermined block 13 and the transform coefficients of the synthesis transform block 32.
[0127] According to embodiments, the ML transform predictor 34 may be non-linear with respect to the one or more previously decoded sample neighbouring the predetermined block 13. That is, the ML transform predictor 34 may be configured to combine the one or more previously decoder samples neighbouring the predetermined block 13 and the synthesis transform block in a non-linear manner. This means, for example, that the ML transform predictor 34 may be implemented using concepts which exploit a non-linear dependence on both the already decoded samples in neighborhood and the transform coefficients of the synthesis transform block 32. An example of such concepts is provided in the form of neural networks further in the disclosure, which may optionally be trainable using training data.
[0128] Further, in accordance with embodiments, the ML transform predictor 34 may be configured to additionally take as input the predicted sample filling. In other words, the inputs (or for example, attributes, or equivalently, arguments) of the ML transform predictor 34 may comprise the predicted sample filling, the one or more previously decoded sample neighbouring the predetermined block 13 and the transform coefficients of the synthesis transform block 32.
[0129] According to embodiments, the ML transform predictor 34 may be non-linear with respect to predicted sample filling. That is, the ML transform predictor 34 may be configured to combine the predicted sample filling, the one or more previously decoder samples neighbouring the predetermined block 13 and the synthesis transform block in a non-linear manner. This means, for example, that the ML transform predictor 34 may be implemented using concepts which exploit a non-linear dependence on the predicted sample filling, the already decoded samples in neighborhood and the transform coefficients of the synthesis transform block 32. An example of such concepts is provided in the form of neural networks further in the disclosure, which may optionally be trainable using training data.
[0130] In accordance with embodiments, the synthesis transform block 32 may be configured to sequentially transform (e.g. perform a plurality of transformations or mappings of the synthesis transform block in a chain-wise or successive manner, or in other words, one after another) from the transform domain to an intermediate domain. In addition, or as an alternative, to these embodiments, the synthesis transform block 32 may be configured to sequentially transform
[0131] filing version, rm(e.g. perform a plurality of transformations or mappings of the synthesis transform block in a chain-wise or successive manner, or in other words, one after another) from the intermediate domain to the sample domain. Further, additionally or alternatively, in accordance with embodiments, the synthesis transform domain may be configured to sequentially transform from a first intermediate transform domain to a second intermediate transform domain in the case where more than one intermediate transform domain may be used, or preferred. For example, this may be the case wherein the synthesis transform block 32 is decoded by making use of more than two transforms, such as a scenario where a primary transform, a secondary transform and a tertiary transform are used.
[0132] In accordance with embodiments, combinations of the aforementioned transform domains, that is, the transform domain and / or the intermediate transform domain(s) may be configured to be at least one of (e.g. or any of): a DCT domain, a DST domain, a LFNST domain, a NSPT domain, a linear transform domain and an orthogonal transform domain.
[0133] Further, according to embodiments, transformations (e.g. or transforms, or mappings) relating (or, e.g. bridging) the transform domain to the intermediate transform domain and / or the intermediate transform domain to the sample domain may be configured to be a linear transform or a orthogonal transform or both a linear and orthogonal transform. Further, for example, in embodiments wherein more than two transformations may be used, any of the intermediate transformations may be configured to be a linear transform or a orthogonal transform or both a linear and orthogonal transform.
[0134] Moreover, the size of the transformations associated with the synthesis transform block 32 may be such that it corresponds to the size of the synthesis transform block 32 in terms of samples, i.e. the number of transform coefficients may equal the number of samples in block 12, but this may also be different. Alternatively, the size of the transformations may be such that it corresponds to that of a sub-block of the synthesis transform block 32.
[0135] So far, embodiments have been concentrated on the predetermined block 13 which may have associated therewith one or more of the following characteristics such as:
[0136] a) size of block 13 such as x samples wide and y sample high, and
[0137] b) prediction mode such as a selected intra prediction mode in case of block 13 being an intra predicted block, or an inter prediction mode in which case one or more motion vectors might be signaled in the data stream 40 for block 13, and
[0138] filing version, rmc) the type of the transformation (e.g. or transformation or mapping) of the synthesis transform block 32 in case of, for instance, decoder 50 supporting more than just one transformation type for transforming 35 or performing the transform transition 39 of the synthesis transform block to obtain the sample domain prediction residual, and d) the one or more previously decoded samples neighbouring the predetermined block 13 (e.g. or one or more reference sample, which may be available to the decoder 50 in the decoded picture buffer).
[0139] Therefore, embodiments are presented according to which the above characteristics may be varied resulting in different implementations of the machine learning transform predictor 34. According to embodiments, the decoder 50 may be configured to support different block sizes for the predetermined block 13 and corrects the predicted sample filling using the sample domain prediction residual by varying a number of parameters defining the ML transform predictor 34 and the parameters defining the ML transform predictor 34, depending on a block size of the predetermined block 13. In other words, for example, the decoder 50 may correct the predicted sample filling using the sample domain prediction residual by selecting a block size among the different sizes (e.g. and supported by the decoder) for the predetermined block 13, and thus, varies the parameters, and a number or a count there of, associated with ML transform predictor 34 accordingly.
[0140] According to embodiments, the decoder 50 may be configured to support different intra prediction modes for the predetermined block 13 signalled in the data stream 40 and corrects the predicted sample filling using the sample domain prediction residual by varying a number of parameters defining the ML transform predictor 34 and the parameters defining the ML transform predictor 34, depending on an intra prediction mode of the predetermined block 13. In other words, for example, the decoder 50 may correct the predicted sample filling using the sample domain prediction residual by selecting an intra prediction mode among the different intra prediction modes (e.g. signalled in the data stream 40 and thus, supported by the decoder) for the predetermined block 13, and thus, varies the parameters, and a number or a count there of, associated with ML transform predictor 34 accordingly. Further, the decoder 50 may be configured to support one or more groups of intra-prediction modes into which the different intra prediction modes are grouped for the predetermined block 13, wherein the manner in which the one or more intra-prediction groups are grouped out of the different intraprediction modes may be signalled in the data stream 40 along with the different intra prediction modes. The decoder 50 may then configured to correct the predicted sample filling using the sample domain prediction residual by varying a number of parameters defining the ML transform predictor 34 and the parameters defining the ML transform predictor 34, depending on a group of intra prediction modes of the predetermined block 13. In other words, for example, the decoder 50 may correct the predicted sample filling using the sample domain filing version, rmprediction residual by selecting a group of intra prediction mode among the different groups of intra prediction modes (e.g. signalled in the data stream 40 and thus, supported by the decoder) for the predetermined block 13, and thus, varies the parameters, and a number or a count there of, associated with ML transform predictor 34 accordingly.
[0141] According to embodiments, the decoder 50 may be configured to corrects the predicted sample filling using the sample domain prediction residual by varying a number of parameters defining the ML transform predictor 34 and the parameters defining the ML transform predictor 34, depending on the one or more previously decoded samples neighbouring the predetermined block. In other words, for example, the decoder 50 may correct the predicted sample filling using the sample domain prediction residual by selecting one or more previously decoded sample among the one or more previously decoded samples neighbouring the predetermined block 13, and thus, varies the parameters, and a number or a count there of, associated with ML transform predictor 34 accordingly.
[0142] With respect to the sample domain prediction residual, the decoder 50 operates as follows: the sample domain prediction residual is obtained using the ML transform predictor 34, which is implemented by a corresponding or associated neural network: That is, as shown in Fig. 7, the ML transform predictor 34 is associated or implemented a neural network 51.
[0143] In accordance with embodiments, the machine learning transform predictor 34 may be implemented by: an initial synthesis transform (for example denoted using B0later in the disclosure), a neural network 51 and a combiner unit. The initial synthesis transform may be configured to receive a set of transform coefficients of the synthesis transform block 32 (for example, reconstructed transform coefficients û), and obtain an initial sample domain prediction residual (e.g. for example denoted using B0û later in the disclosure) based on the set of transform coefficients. The neural network 51 may comprise an initial non-linear fully-connected layer and one or more succeeding layers. The neural network 51 may be configured to receive the set of transform coefficients as inputs of the initial layer, receive an output of the initial layer as an input of the one or more succeeding layers.
[0144] The combiner unit may be configured to combine the initial sample domain prediction residual and an output of a final layer among the one or more succeeding layers to obtain the sample domain prediction residual. For example, the combiner unit may simply add, or for example average, the initial sample domain prediction residual and the output of a final layer among the one or more succeeding layers. In examples, the combiner units may even combine the the initial sample domain prediction residual and an output of a final layer among the one or more
[0145] filing version, rmsucceeding layers as per a weighted averaging operation, wherein weights may be related, or given, by additional or existing parameters of the NN 51.
[0146] The neural network 51 may, for instance, also be referred to as the synthesis transform neural network 51 to further distinguish it from other neural networks described in this disclosure.
[0147] The attributes of the neural network 51 or, in different terms, the input(s) of the initial layer of the neural network 51, may comprise:
[0148] 1) the set of transform coefficients of the synthesis transform block 32;
[0149] further, 2) the predicted sample filling of the predetermined block 13;
[0150] further, 3) the one or more previously decoded samples neighbouring the predetermined block 13. In other words, reference samples or already decoded samples available to the decoder in the decoded picture buffer may be additionally taken as input by the neural network. A number (or a count) of the one or more previously decoded samples neighbouring the predetermined block which is input into the neural network may be vary from anywhere from one previously decoded sample to all previously decoded samples of the one or more previously decoded samples neighbouring the predetermined block 13;
[0151] further, 4) a quantization parameter of the predetermined block 13 such as the quantization step size based on which the decoder dequantizes or scales the transform coefficients. The quantization parameter may be configured to adjust the granularity at which the dequantizer, or dequantization block, 28, which dequantizes (e.g. scales) quantized transform coefficients; and
[0152] further, 5) an index of the selected intra-prediction mode selected from a plurality of intraprediction modes (or different intra-prediction modes),
[0153] It is noted that both the options 4) and 5) have been not been explicitly indicated in Fig. 4 by way of being depicted schematically contained within a block / module (such as the block 24 shown in Fig. 5 which will be described later), wherein the block / module could be connected to the dequantizer 28 and the ML transform predictor 34 (e.g. β). Hence, such a modification of Fig. 4, while constituting a distinct embodiment compared to the one exemplarily illustrated in Fig. 4 would indicate possible inputs of the NN 51 implementing the ML transform predictor 34 (e.g. β).
[0154] filing version, rmIn accordance with embodiments, the final layer of the one or more succeeding layers of the neural network 51 is a linear layer. It is to be understood that linearity of the final layer relates to the attributes, or inputs, of the neural network 51 implementing the ML transform predictor 34 such as any of the set of transform coefficients of the synthesis transform block 32, the predicted sample filling of the predetermined block 13, the one or more previously decoded samples neighbouring the predetermined block 13, the quantization parameter of the predetermined block 13, and the the index of the selected intra-prediction mode.
[0155] Further, in accordance with embodiments, the the neural network 51 may be configured to receive the set of transform coefficients of the synthesis transform block 32 concatenated with the predicted sample filling (e.g. meaning joining these together in sequence, or as a single entity) as inputs of the initial layer of the neural network 51. Even further, in accordance with embodiments, the neural network may be configured to receive any, or at least one, of the following: the one or more previously decoded samples neighbouring the predetermined block 13, the quantization parameter of the predetermined block 13, and the index of a selected intra-prediction mode selected from a plurality of intra-prediction modes concatenated (e.g. meaning, joining any of these together in sequence, or as a single entity) with the above described inputs, i.e. the predicted sample filling and the set of transform coefficients, as inputs of the initial layer of the neural network 51. For instance, further to the above options, additional reconstruction information obtainable by the decoder 50 may be taken as inputs of the initial layer of the neural network 51 along with examples of the above mentioned inputs. For instance, the additional reconstruction information obtainable by the decoder 50 may be concatenated along with examples of the above mentioned inputs of the initial layer of the neural network 51.
[0156] According to embodiments, the initial synthesis transform, or transformation or mapping, may be an orthogonal transform. In other words, the ML transform predictor 34 may be initialized by an orthogonal transform to obtain the initial sample domain prediction residual. Additionally, In accordance with embodiments, the initial synthesis transform may be at least one of, or any of, a DCT, a DST, a LFNST, a NSPT, and a linear transform.
[0157] As another optional feature, the synthesis transform NN 51 in the above description comprises a non-linear activation function, as an example a rectified linear unit (ReLu). It is to be noted that other forms which introduce any non-linearities may be used in combination with or without the ReLu.
[0158] filing version, rmFor instance, it holds in general, according to embodiments, the activation functions in the layers of the synthesis transform NN 51 may include non-linear functions such as leaky ReLu, tanh, binary step function, SELU, ELU, sigmoid / logistic and / or parametric ReLu. This means, for example, that a first grouping of the layers of the NN 51 may comprise non-linear activation functions different from a second grouping of the layers of the NN 51. As another optional feature, it also holds in general, that any of the layers of the NN 51 may comprise more than one non-linear activation function. Additionally or alternatively, as an instantiation, some of the layers, or some neurons among the layer, of the NN 51 may even comprise an identity function as the activation function.
[0159] According to embodiments, the decoder 50 may support one or more of a number of different initial synthesis transforms for the ML transform predictor 34, wherein each initial synthesis transform is related to a neural network 51iimplementing the ML transform predictor 34 and may correct the predicted sample filling using the sample domain prediction residual obtained by selecting and using a corresponding neural network 51irelated to an initial synthesis transform among the different initial synthesis transforms of the predetermined block 13. In other words, the decoder 50 may support different neural networks 51iwherein the index i may range from anywhere including one to including a maximal number of different initial synthesis transforms supported by the decoder 50. The decoder may perform a selection among the different neural networks, wherein the different neural networks are classified according to the different initial synthesis transform of the ML transform predictor 34. The different neural networks may be further classified according to a number (e.g. a count of) of parameters defining the NN 51iand values of the parameters of the NN 51iwhich the decoder 50 may be configured to vary.
[0160] According to embodiments, the decoder 50 may support one or more of a number of different block sizes for the predetermined block 13, wherein each block size is related to a neural network 51iimplementing the ML transform predictor 34 and may correct the predicted sample filling using the sample domain prediction residual obtained by selecting and using a corresponding neural network 51irelated to a block size among the different block sizes of the predetermined block 13.
[0161] In other words, the decoder 50 may support different neural networks 51iwherein the index i may range from anywhere including one to including a maximal number of different block sizes of the predetermined block 13 supported by the decoder 50. The decoder may perform a selection among the different neural networks, wherein the different neural networks are classified according to the different block sizes of the predetermined block 13. The different neural networks may be further classified according to a number (e.g. a count of) of parameters
[0162] filing version, rmdefining the NN 51jand values of the parameters of the NN 51jwhich the decoder 50 may be configured to vary.
[0163] Further, in the case of a block size equalling a mutually transposed block size among the different block sizes of the predetermined block 13, the decoder may correct the predicted sample filling using the sample domain prediction residual by transposing the input of the initial layer and the output of the final layer of a corresponding neural network 51 j related to the mutually transposed block size of the predetermined block 13, and using the parameters of the neural network 51 j related to the mutually transposed block size of the predetermined block 13.
[0164] Further, in the case of the block size exceeding a predetermined block size value, the corresponding neural network 51 j takes, as an input, a sub-portion of the synthesis transform block 32. In other words, the input of the NN 51 j in such an instance would be a subset of the set of transform coefficients of the synthesis transform block 32.
[0165] For example, in case of a sub-portion of the synthesis transform block 32 (e.g. a N x N subblock of the synthesis transform block 32) being taken as input to the neural network 51 j, the neural network 51 i may be configured to output, such as an output of its final layer, corresponding to the inputted sub-portion of the synthesis transform block 32. Further, in addition to this, the combiner unit, associated with the neural network 51iimplementing the ML transform predictor 34, may be configured to combine the initial sample domain prediction residual and the output based on the inputted sub-portion of the synthesis transform block to obtain the sample domain prediction residual. In this regard, for example, it may be the NN 51i may be configured to output only based on the inputted sub-portion of the synthesis transform block, which then the combiner unit may combine with the initial sample domain prediction residual to obtain the sample domain prediction residual.
[0166] For example, the sub-portion of the synthesis transform block 32 may comprise a N x N subblock of the synthesis transform block 32, with the N xN sub-block of the synthesis transform block comprising a zero frequency or DC transform coefficient. It may be that the sub-portion of the synthesis transform block 32 is obtained by configuring, e.g. setting, transform coefficients associated with a set of specific frequencies, each of which exceeds a predetermined frequency threshold associated with the transform block, to zero. In this regard, transform coefficients associated with the set of specific frequencies which are set to zero may be referred to as high frequency transform coefficients, while the transform coefficients associated with the sub-portion may be referred to as low frequency transform coefficients. The predetermined frequency threshold which may help demarcate, or distinguish, or categorize, the low-frequency and high-frequency transform coefficients, may be varied across
[0167] filing version, rmthe NN 51idepending on application or use-case. In other words, the sub-portion of the synthesis transform block 32 may correspond to a set of transform coefficients being lowest in terms of frequency, the set of transform coefficient being derived from a super set of transform coefficients subsuming the set of transform coefficients being lowest in terms of frequency. For example, the sub-portion may be 2x2 or 4x4 or 8x8 or 16x16 block comprising the lowest frequency coefficients, such sub-portions may comprise non-square, or rectangular, blocks additionally or alternatively.
[0168] According to embodiments, the decoder 50 may support one or more of a number of different intra-prediction modes for the predetermined block 13 or one or more of a group of intraprediction modes into which the different intra-prediction modes are grouped, each of both of which is related to a neural network 51jimplementing the ML transform predictor 34 and may correct the predicted sample filling using the sample domain prediction residual obtained by selecting and using a corresponding neural network 51jrelated to an intra-prediction mode, or a group of intra-prediction modes, among the different intra-prediction modes, or the groups thereof, of the predetermined block 13. In other words, the decoder 50 may support different neural networks 51jwherein the index may range from anywhere including one to including a maximal number of different intra-prediction modes or a maximal number of different groups of intra-prediction modes of the predetermined block 13 supported by the decoder 50. The decoder may perform a selection among the different neural networks, wherein the different neural networks are classified according to the different intra-prediction modes or groups thereof of the predetermined block 13. The different neural networks may be further classified according to a number (e.g. a count of) of parameters defining the NN 51jand values of the parameters of the NN 51jwhich the decoder 50 may be configured to vary.
[0169] In other words, it may be that, with respect to obtaining the sample domain prediction residual, one or more of the following may be true:
[0170] 1) the decoder 50 manages and comprises a set of NNs 51j(and possibly associated parameters and a number or count thereof) for each block size or each of different groups of block sizes into which all supported block sizes are grouped, and the decoder 50, when obtaining (e.g. computing) the sample domain prediction residual, firstly selects the corresponding - the one fitting to the size of block 12 - set of NNs 51j(and possibly associated parameters and a number or count thereof) before acting and behaving as described above with respect to that set of NNs 51j(and possibly associated parameters and a number or count thereof). That is, for instance, the decoder may select NNs 51jusing a predetermined criterion which is associated with the supported block sizes of the block 13.
[0171] filing version, rma) In accordance with embodiments, the grouping of the blocks may group mutually transposed blocks, i.e. blocks and correspondingly transposed blocks comprising the same size, together. In other words, blocks of shape Wx H and H x W, may be grouped together, where W and H denote dimensions of the block, e.g. with W being the width of one block in samples and H being the height of this block in samples. Without loss of generality, in case of the block shape Wx H, the synthesis transform NN 51jwould process its inputs and outputs without additional operations whereas in case of the transposed block shape H x W, the inputs, namely the set of transform coefficients and the predicted sample filling 18 (i.e. the prediction signal) would be transposed before application onto the NN, thereby allowing the sample domain prediction residual obtained (e.g. computed) by the NN 51jfor the block shape W x H to be used, and subsequently, the output of the NN would be transposed back into the shape H x W thus obtaining (e.g. computing) the sample domain prediction residual for the transposed block shape H x W.
[0172] b) In accordance with embodiments, if the block size selected by the decoder 50 is larger, or greater, than a predetermined block size value, the decoder 50 may take a subportion of the synthesis transform block 32. For instance, this sub-portion of the synthesis transform block 32 may correspond to the predetermined block size value. For example, this sub-portion of the synthesis transform block 32 may mean corresponding to a subset of the set of transform coefficients of, or representing, the synthesis transform block 32.
[0173] ) the decoder 50 uses (and possibly selects), for instance, the same set of NNs (and possibly associated parameters and a number or count thereof) irrespective of the prediction mode of block 13, but it could, naturally, be that the decoder 50 manages and comprises a set of NNs 51 j (and possibly associated parameters and a number or count thereof) for each intra-prediction mode or each of different groups of intra-prediction modes into which all supported intra-prediction mods are grouped, and the decoder 50, when obtaining (e.g. computing) the sample domain prediction residual, may firstly select the corresponding - the one fitting to the intra prediction mode of block 13 - set of NNs (and possibly associated parameters and a number or count thereof) before acting and behaving as described above with respect to that set of NNs (and possibly associated parameters and a number or count thereof) such as by use of the index of the selected intra-prediction mode for block 13 which index might be signaled in the data stream.
[0174] ) the decoder 50 uses (and possibly selects), for instance, the same set of NNs (and possibly associated parameters and a number or count thereof) irrespective of the transform type of the initial synthesis transform, but it could, naturally, be that the decoder 50 manages
[0175] filing version, rmand comprises a set of NNs 51j(and possibly associated scalar product parameters 51j) for each transform type or each of different groups of transform types into which all supported transform types are grouped, and the decoder, when obtaining (e.g. computing) the sample domain prediction residual, may firstly select the corresponding - the one fitting to the transform type of the initial synthesis transform - set of NNs (and possibly associated parameters and a number or count thereof) before acting and behaving as described above with respect to that set of NNs (and possibly associated parameters and a number or count thereof).
[0176] 4) Naturally, it may be that the decoder 50 corrects the predicted sample filling using the sample domain prediction residual only to a subset of the possible block characteristics a - d listed above, such as only to blocks in a certain range of sizes, the initial synthesis transform is of one or more of certain transform types, and blocks of one or more intraprediction modes. Moreover, inter-predicted blocks may or may not be subject to correction of the predicted sample filling using the sample domain prediction residual.
[0177] For example, in regard to the grouping of the mutually transposed blocks, such as square blocks, the decoder 50 may select and transpose a block depending on a predetermined criterion, which relates to an index of intra-prediction mode of the said block. For example, the predetermined criterion may be the block being associated with a specific intra-prediction mode index, which exceeds a predetermined intra-prediction mode threshold. For example, the decoder may transpose square blocks with vertical angular intra prediction modes where the mode associated with the square blocks is greater than 34. By this measure, the vertically prediction blocks, i.e. blocks which have been predicted using the vertical angular prediction mode, may be considered, for the purpose of decdoding using the NN 51 i, as horizontally predicted blocks as a cosequence of the transpose operation. This provides the benefit of a reduced variance of inputs which the NNs 51 i may be subjected to for learning and thus fewer blocks may be used in training the NN while covering different associated intra prediction modes.
[0178] With respect to the above discussion, parameters of the NN 51jmay comprise a number (or a count) of hidden layers (i.e. the one or more succeeding layers), and weights, and a number or a count thereof, and biases, and a number or a count thereof.
[0179] In accordance with embodiments, the number (or count) of transform coefficients in the set of transform coefficients may be equal to a number of samples of the synthesis transform block 32.
[0180] filing version, rmIn accordance with embodiments, the set of transform coefficients may be decoded by the decoder 50 from the data stream 40 based on a scan order. For example, the scan order defined within the synthesis transform block 32 may be a diagonal scan order starting at the highest frequency component, e.g., a bottom right position in the synthesis transform block 32 and at the DC component, which may be positioned at the top left of the the synthesis transform block 32. In other words, the scan order may roughly traverse the set of transform coefficients from higher to lower frequency, so that transform coefficients associated with lower frequency components may be listed further down in the scan order. However, the scan order may not necessarily follow strictly an order set up by the frequency, in particular, as transform coefficients may be associated with vertical and horizontal frequency components, there may be multiple transform coefficients, for which their order according to higher or lower frequency is not well defined.
[0181] Hitherto, in the description of the decoder 50 as exemplarily illustrated in Fig. 4 has been restricted to known or existing intra-prediction modes. In the following, a new intra-prediction mode, i.e. a learned intra prediction mode as implemented by a machine learning prediction predictor 36, indicated by the block 36 in Fig. 4, will be described as comprising further embodiments of the invention. Before proceeding with the description of such intra prediction by way of the ML prediction predictor 36, it is emphasized that the ML prediction predictor 36 is an optional feature and may be removed or added upon preference to formulate distinct embodiments of the invention. It should be noted that block 36 in Fig. 4 denotes that the corresponding prediction (e.g. transformation or mapping) of the prediction module 14 might be implemented by the ML prediction predictor 36 rather than influencing or existing in addition to the prediction (or, for example, transformations or mappings) of the prediction module 14 to which the ML prediction predictor 36 is attached to.
[0182] For instance, the machine learning prediction predictor 36 has been denoted by β in Fig. 4-6, 9 as an example and further in the disclosure. For example, parameters of the machine learning transform predictor 36 may be denoted by ψpelsewhere in the disclosure, regardless of its specific implementation.
[0183] According to embodiments, the decoder 50 may be configured to predict the predetermined block 13 based on a machine learning prediction predictor 36 by inputting into the ML prediction predictor 36 the one or more previously decoded samples neighbouring the predetermined block 13 as inputs, and obtaining the predicted sample filling based on an output of the machine learning prediction predictor. Therefore, while this ML prediction predictor 36 may predict the predicted sample filling, the already described ML transform predictor 34 may correct this predicted sample filling using the sample domain prediction residual, and thus the
[0184] filing version, rmML transform predictor 34 and the ML prediction predictor may be used in conjunction with each other in the decoder 50, as illustrated in Fig. 4.
[0185] In accordance with embodiments, the ML prediction predictor 36 may be non-linear with respect to the one or more previously decoded samples neighbouring the predetermined block 13. This means, for example, that the ML prediction predictor 36 may be implemented using concepts which exploit a non-linear dependence on the one or more previously decoded samples neighbouring the predetermined block 13. An example of such concepts is provided in the form of neural networks further in the disclosure, which may optionally be trainable using training data.
[0186] Embodiments are presented according to which the characteristics of, or relating to, the predetermined block 13 such as a block size thereof and the one or more previously decoded samples neighbouring it may be varied resulting in different implementations of the machine learning transform predictor 34.
[0187] According to embodiments, the decoder 50 may be configured to support different block sizes for the predetermined block 13 and to obtain the predicted sample filling by varying a number of parameters defining the ML prediction predictor 36 and the parameters defining the ML prediction predictor 36, depending on a block size of the predetermined block 13. In other words, for example, the decoder 50 may obtain the predicted sample filling by selecting a block size among the different sizes (e.g. and supported by the decoder) for the predetermined block 13, and thus, varies the parameters, and a number or a count there of, associated with ML prediction predictor 36 accordingly. Equivalently, the decoder 50 may perform a selection among the different ML prediction predictors 36, wherein the different ML prediction predictors are classified according to the different block sizes of the predetermined block 13, and further the different ML prediction predictors 36 may be classified according to a number (e.g. a count of) of parameters defining it and their values which the decoder 50 may be configured to vary. Further, according to embodiments, the decoder 50 may be configured to obtain the predicted sample filling by varying a number of parameters defining the ML prediction predictor 36 and the parameters defining the ML prediction predictor 36, depending on one or more previously decoded samples neighbouring the predetermined block. In other words, for example, the decoder 50 may obtain the predicted sample filling by selecting one or more previously decoded samples among the one or more previously decoded samples neighbouring the predetermined block 13, and thus, varies the parameters, and a number or a count there of, associated with ML prediction predictor 36 accordingly. Equivalently, the decoder 50 may perform a selection among the different ML prediction predictors 36, wherein the different ML prediction predictors are classified according to the one or more previously decoded samples
[0188] filing version, rmneighbouring the predetermined block, and further the different ML prediction predictors 36 may be classified according to a number (e.g. a count of) of parameters defining it and their values which the decoder 50 may be configured to vary.
[0189] With respect to the predicted sample filling, the decoder 50 operates as follows: the predicted sample filling is obtained using the ML prediction predictor 36, which is implemented by a corresponding or associated neural network: That is, as shown in Fig. 7, the ML transform predictor 34 is associated or implemented by a neural network 61.
[0190] In accordance with embodiments, the machine learning prediction predictor 36, denoted as p in the block 36 of Fig. 4, may be implemented by an initial predictor, denoted by pred0in this disclosure (and represented by 63a in Fig. 16 later in the disclosure), the neural network 61 and a predictor combiner unit (represented by 63f in Fig. 16 later in the disclosure). The initial predictor may be configured to receive the values of one or more previously decoded samples neighbouring the predetermined block 13 (e.g. reconstructed reference samples ŝr; represented by 63b in Fig. 16 later in the disclosure) and to predict an initial predicted sample filling, denoted by pred0in this disclosure, of the predetermined block 13 using the values of the one or more previously decoded samples neighbouring the predetermined block 13. The neural network 61 may comprise an initial non-linear fully-connected layer (represented by 63c in Fig. 16 later in the disclosure) and one or more succeeding layers (represented by 63d in Fig. 16 later in the disclosure), and the neural network 61 may configured to receive the values of one or more previously decoded samples neighbouring the predetermined block 13 as inputs of the initial layer, and receive an output of the initial layer as an input of the one or more succeeding layers.
[0191] The predictor combiner unit may be configured to combine the initial predicted sample filling and an output of a final layer (represented by 63e in Fig. 16 later in the disclosure) among the one or more succeeding layers to obtain the predicted sample filling. For example, the predictor combiner unit may simply add, or for example average, the initial predicted sample filling and the output of a final layer among the one or more succeeding layers. In examples, the predictor combiner unit may even combine the the initial predicted sample filling and the output of the final layer among the one or more succeeding layers as per a weighted averaging operation, wherein weights may be related, or given, by additional or existing parameters of the NN 61. The neural network 61 may, for instance, also be referred to as the intra-prediction neural network 61 to further distinguish it from other neural networks described in this disclosure.
[0192] The attributes of the neural network 51 or, in different terms, the input(s) of the initial layer of the neural network 51, may comprise: the one or more previously decoded samples neighbouring the predetermined block 13. In other words, reference samples or already
[0193] filing version, rmdecoded samples available to the decoder in the decoded picture buffer may be additionally taken as input by the prediction neural network 61. A number (or a count) of the one or more previously decoded samples neighbouring the predetermined block which is input into the prediction neural network 61 may be vary from anywhere from one previously decoded sample to all previously decoded samples of the one or more previously decoded samples neighbouring the predetermined block 13.
[0194] In accordance with embodiments, the final layer of the one or more succeeding layers of the neural network 61 is a linear layer. It is to be understood that linearity of the final layer relates to the attributes, or inputs, of the neural network 61 implementing the ML prediction predictor 36 such as the one or more previously decoded samples neighbouring the predetermined block 13.
[0195] Further, in accordance with embodiments, the the intra-prediction neural network 61 may be configured to receive the values of one or more previously decoded samples neighbouring the predetermined block concatenated (e.g. meaning, joining these together in sequence, or as a single entity) with the initial predicted sample filling as the inputs of the initial layer of the neural network 61.
[0196] As another optional feature, the intra-prediction NN 61 in the above description comprises a non-linear activation function, as an example a rectified linear unit (ReLu). It is to be noted that other forms which introduce any non-linearities may be used in combination with or without the ReLu.
[0197] For instance, it holds in general, according to embodiments, the activation functions in the layers of the intra-prediction NN 61 may include non-linear functions such as leaky ReLu, tanh, binary step function, SELU, ELU, sigmoid / logistic and / or parametric ReLu. This means, for example, that a first grouping of the layers of the NN 61 may comprise non-linear activation functions different from a second grouping of the layers of the NN 61. As another optional feature, it also holds in general, that any of the layers of the NN 61 may comprise more than one non-linear activation function. Additionally or alternatively, as an instantiation, some of the layers, or some neurons among the layer, of the NN 61 may even comprise an identity function as the activation function.
[0198] According to embodiments, the decoder 50 may support one or more of a number of different intra-prediction modes for the initial predictor of the machine learning prediction predictor 36 or one or more of a group of intra-prediction modes into which the different intra-prediction
[0199] filing version, rmmodes are grouped for the initial predictor of the machine learning prediction predictor 36, each of both is related to a neural network 61jimplementing the ML prediction predictor 36 and may obtain the predicted sample filling by selecting and using a corresponding neural network 61jrelated to an intra-prediction mode, or a group of intra-prediction modes, among the different intra-prediction modes, or the groups thereof, of the initial predictor. In other words, the decoder 50 may support different neural networks 61jwherein the index may range from anywhere including one to including a maximal number of different intra-prediction modes or a maximal number of different groups of intra-prediction modes of the initial predictor supported by the decoder 50. The decoder 50 may perform a selection among the different neural networks 61j, wherein the different neural networks 61jare classified according to the different intra-prediction modes or groups thereof of the initial predictor. The different neural networks 61 j may be further classified according to a number (e.g. a count of) of parameters defining the NN 61jand values of the parameters of the NN 61jwhich the decoder 50 may be configured to vary.
[0200] According to embodiments, the decoder 50 may support one or more of a number of different block sizes for the predetermined block 13, wherein each block size is related to a neural network 61jimplementing the ML prediction predictor 36 and may obtain the predicted sample filling by selecting and using a corresponding neural network 61 j related to a block size among the different block sizes of the predetermined block 13. In other words, the decoder 50 may support different neural networks 61 j wherein the index / may range from anywhere including one to including a maximal number of different block sizes of the predetermined block 13 supported by the decoder 50. The decoder 50 may perform a selection among the different neural networks 61 j, wherein the different neural networks are classified according to the different block sizes of the predetermined block 13. The different neural networks may be further classified according to a number (e.g. a count of) of parameters defining the NN 51jand values of the parameters of the NN 51jwhich the decoder 50 may be configured to vary.
[0201] Further, in the case of the block size exceeding a predetermined block size value, the corresponding neural network 61jtakes, as an input, a subset of the one or more previously decoded samples neighbouring the predetermined block 13.
[0202] Further, in the case of a block size equalling a mutually transposed block size among the different block sizes of the predetermined block 13, the decoder may obtain the predicted sample filling by transposing the input of the initial layer and the output of the final layer of a corresponding neural network 61jrelated to the mutually transposed block size of the predetermined block 13, and using the parameters of the neural network 61jrelated to the mutually transposed block size of the predetermined block 13.
[0203] filing version, rmIn other words, it may be that, with respect to obtaining the predicted sample filling, one or more of the following may be true:
[0204] 1) the decoder 50 manages and comprises a set of NNs 61j(and possibly associated parameters and a number or count thereof) for each block size or each of different groups of block sizes into which all supported block sizes are grouped, and the decoder 50, when obtaining the predicted sample filling, firstly selects the corresponding - the one fitting to the size of block 13 - set of NNs 61j(and possibly associated parameters and a number or count thereof) before acting and behaving as described above with respect to that set of NNs 61j(and possibly associated parameters and a number or count thereof). That is, for instance, the decoder may select NNs 61jusing a predetermined criterion which is associated with the supported block sizes of the block 13.
[0205] a) In accordance with embodiments, the grouping of the blocks may group mutually transposed blocks, i.e. blocks and correspondingly transposed blocks comprising the same size, together. In other words, blocks of shape Wx H and H x W, may be grouped together, where W and H denote dimensions of the block, e.g. with W being the width of one block in samples and H being the height of this block in samples. Without loss of generality, in case of the block shape Wx H, the synthesis transform NN 61jwould process its inputs and outputs without additional operations whereas in case of the transposed block shape H x W, the inputs, namely the one or more previously decoder samples neighbouring the predetermined block would be transposed before application onto the NN, thereby allowing the predicted sample filling obtained by the NN 61jfor the block shape W x H to be used, and subsequently, the output of the NN would be transposed back into the shape H x W thus obtaining the predicted sample filling for the transposed block shape H x W.
[0206] b) In accordance with embodiments, if the block size selected by the decoder 50 is larger, or greater, than a predetermined block size value, the decoder 50 may take a subset of the one or more previously decoded samples neighbouring the predetermined block. For instance, this subset of the one or more previously decoded samples neighbouring the predetermined block may correspond to a number of samples corresponding to the predetermined block size value.
[0207] 2) Naturally, it may be that the decoder 50 obtain the predicted sample filling only to a subset of the possible block characteristics listed above, such as only to blocks in a certain range of sizes. Moreover, inter-predicted blocks may or may not be subject to obtaining of the predicted sample filling.
[0208] filing version, rmWith respect to the above discussion, parameters of the NN 61jmay comprise a number (or a count) of hidden layers (i.e. the one or more succeeding layers), and weights, and a number or a count thereof, and biases, and a number or a count thereof.
[0209] In accordance with embodiments, different intra-prediction modes for the initial predictor (e.g. predo) includes one or more of a planar mode, a DC mode, a plurality of angular modes, one or more matrix-based intra prediction modes, according to each of which the predicted sample filling is obtained by a multiplication between a matrix associated with the respective matrixbased intra prediction mode and a vector obtained from sample values in a neighbourhood of the predetermined block, and one or more intra-subpartition modes, according to each of which the predicted sample filling is obtained by subdividing the predetermined block into subpartitions and intra-predicting the subpartitions sequentially with correcting a subpartition using the sample domain prediction residual, for example as obtained by the ML transform predictor 34, before intra-predicting a next subpartition.
[0210] So far optional details of the decoder 50 such as the entropy decoder 60, the in-loop filter 68 and the out-of-loop filter 70 as shown in Fig. 4, have not been described, the present application proceeds further with description of such details.
[0211] Decoder 50 of Fig. 4 comprises an entropy decoder, which entropy decodes quantization indices from the data stream 40 (or in other words, reverse the action of a corresponding entropy coding on a corresponding encoder side). In particular, the quantization indices may be represented by signs and absolute values. Further, the dequantizer 28, may dequantize the quantization indices, thereby deriving the transform coefficients of the synthesis transform block 32. For example, the dequantization may include a scaling, and optionally, a rounding operation.
[0212] It is also indicated in Fig. 4 that the dequantized transform coefficients are exemplarily denoted by v̂ in similarity to equations (1.1.4) and (1.4.2) described further in the disclosure, the transform coefficients of the synthesis transform block 32 being subjected to the machine learning transform predictor 34 (itself exemplarily being denoted by β) are exemplarily denoted by β(v̂) in harmony with equation (1.4.5) described further in the disclosure, the predicted sample filling is exemplarily denoted by p and the reconstructed signal, or reconstructed transform coefficients, are denoted by ŝ = β(v̂) + p, in harmony with equation (1.4.3) described further in the disclosure. It is noted that the dequantized transform coefficients v̂ could have been obtained by an encoder using a further ML predictor (exemplarily denoted by α) but could also have been obtained by any other means, such as without using the further ML predictor (exemplarily denoted by α) or the like; hence the use of the notation v̂ in this embodiment may
[0213] filing version, rmnot be restricted or limited to the use of the further ML predictor (e.g. a) on a corresponding encoder side.
[0214] For example, the derivation of the quantization indices from data stream 14 may be performed by decoding binary representations of the quantization indices from the data stream 14 using entropy decoding followed by a debinarization of the binary representations to obtain the quantization indices.
[0215] Additionally, the derivation process of the transform coefficients may optionally involve a process referred to as dependent dequantization or trellis-coded dequantization, in which, depending upon the sign, or parities, of the previously decoded transform coefficients, the decoder switches among a set of dequantizers.
[0216] For instance, as shown in Fig. 4 by way of dashed lines, the decoder 50 may optionally comprise an in-loop filtering module 68 configured to filter a portion of the reconstructed signal as formed by the combiner 64 to obtain a filtered reconstructed signal for a purpose of reducing compression artefacts in the prediction module 14. Further, the decoder 50 may optionally comprise an out-of-loop filtering module 70 configured to receive the reconstructed signal at the output of the prediction loop and filter the reconstructed signal for a purpose of improving perceptual picture quality of the reconstructed signal. The reconstructed signal as filtered by the out of loop filtering module 70 is finally provided as a reconstructed signal, namely picture 12’.
[0217] The present application now proceeds further with description of an implementation of the encoder 10 shown in Fig. 5, in harmony with (e.g. corresponding to, or fitting) the decoder of Fig. 4 in accordance with embodiments of the second aspect. It is noted that this encoder 10 represents implementations which are configurable to fit the decoder 50 of Fig. 4.
[0218] As previously indicated, the encoder 10 may be a video encoder or a picture encoder which, in addition to the details set out with respect to the decoder of Fig. 4, comprises one or more of the functionalities described above with respect to Fig. 1 to 3. In particular, details of the decoder 50 may be configured so as to be transferred on to the encoder of Fig. 5. Therefore, such features and details, which have already been described for the decoder 50 of Fig. 4, are not repeated herein for the sake of brevity and conciseness of the disclosure. Instead, the description of the encoder 10 of Fig. 5 is aimed at providing features and details which may not be configured to be transferable on to the encoder 10.
[0219] filing version, rmThe encoder 10 for block based encoding of a predetermined block 13 of a picture 12 (both of which are shown in Fig. 5) is configured to predict 14 a predicted sample filling of the predetermined block 13. This prediction 14 may be performed using an intra-prediction mode which the encoder 10 encodes into the data stream 40, such as encoding an index of the intraprediction mode into the data stream 40. This prediction has been denoted by way of a prediction module 14 in Figure 5. In accordance with some embodiments, this intra-prediction mode may be a known or existing intra-prediction mode available to the encoder. Alternatively, in accordance with some embodiments, a new intra-prediction mode, such as a learned intraprediction mode, may be used for predicting the predicted sample filling by using a machine learning prediction predictor 36, denoted by p in block 36 of Fig. 5. This prediction 14 of the predicted sample filling using the ML prediction predictor 36 is analogous to the ML prediction predictor 36 used by the decoder 50 of Fig. 4, and thus reference is made to those details being adapted and applied on to the encoder 10, such details are not repeated herein. Additionally or alternatively, the encoder 10 may predict the predicted sample filling of the predetermined block 13 using inter-prediction.
[0220] With respect to the different intra prediction modes available to the encoder 10, the encoder 10 may predict the predicted sample filling of the predetermined block 13 by selecting a selected intra-prediction mode out of the different intra-prediction modes signalled in the data stream 40, and the encoder may predict the predicted sample filling using the selected intraprediction mode. For example, the different intra-prediction modes may include one or more of: a planar mode, a DC mode, a plurality of angular modes, one or more matrix-based intra prediction modes, according to each of which the predicted sample filling is obtained by a multiplication between a matrix associated with the respective matrix-based intra prediction mode and a vector obtained from sample values in a neighbourhood of the predetermined block, and one or more intra-subpartition modes, according to each of which the predicted sample filling is obtained by subdividing the predetermined block into subpartitions and intrapredicting the subpartitions sequentially with correcting a subpartition using the sample domain prediction residual before intra-predicting a next subpartition.
[0221] The encoder 10 is further configured to encode an analysis transform block 20 of a prediction residual for the predetermined block 13 into the data stream 40. The analysis transform block 20 is denoted by T as an example in Fig. 5. The analysis transform block 20 may relate to an analysis transform (or transformation or mapping) (e.g. or a forward transform) of the prediction residual of the predetermined block 13. Encoding the analysis transform block 20 means that transform coefficients of the analysis transform block 20, which are transmitted in the data stream 40 and available to the encoder are encoded into the data stream 40.
[0222] filing version, rmThe analysis transform 20 is for reconstructing the predetermined block 13. The encoder 20 might actually perform the reconstruction of the predetermined block 13, namely in its prediction loop and additionally, or alternatively, for rate-distortion, R / D, optimization purposes.
[0223] The encoder 10 performs the reconstruction of the predetermined block 13 by subjecting a synthesis transform block 32 to a machine learning transform predictor 34, which has been denoted by β in Fig. 5 as an example. It should be noted that block 34 in Fig. 5 denotes that the corresponding transform (e.g. transformation or mapping) of the synthesis transform block 32 might be implemented by the ML transform predictor 34 rather than influencing or existing in addition to the transforms (or, for example, transformations or mappings) of the synthesis transform block 32 to which the ML transform predictor 34 is attached to.
[0224] The ML transform predictor 34 is configured to transform 35 (e.g. to carry out a transform or transformation or a mapping of) of the synthesis transform block 32 from a synthesis transform domain to a sample domain to obtain a sample domain prediction residual.
[0225] Alternatively, the ML transform predictor 34 is configured to perform a transform transition 39 of the synthesis transform block 32 from the synthesis transform domain to the sample domain to obtain the sample domain prediction residual. The transform transition 39 meaning a proper subset of one or more transformation stages (e.g. or transformations or mappings) out of a sequence (e.g. or a chain) of transform stages (e.g. or transformations or mappings) leading from the synthesis transform domain to the sample domain (e.g. culminating in the sample domain). It is indicated here that the two blocks, i.e. the transform 35 and the transform transition 39, have been illustrated using dashed lines, without preferring one over the other, for the purpose of representing the alternatives which the ML transform predictor 34 subjects the synthesis transform block 32 to.
[0226] The synthesis transform domain here relates to the domain of the transform coefficients signalled in the data stream 40 representing (e.g. or constituting, or forming) the synthesis transform block 32. The sample domain here relates to the domain of the samples of the predetermined block 13 reconstructable by the decoder 50.
[0227] The encoder 10 then reconstructs the predetermined block 13 by correcting the predicted sample filling of the predetermined block 13 using the sample domain prediction residual. In particular, as seen in Fig. 5, the sample domain prediction residual obtained by, or as a result of, the synthesis transform block 32, which is accompanied by the ML transform predictor 34
[0228] filing version, rmimplementing it, is combined with the predicted sample filling obtained by, or as a result of, the prediction module 14 (which is optionally accompanied by a machine learning prediction predictor 36, details of which will be summarily described later in this disclosure since they have already been described for the decoder 50 of Fig. 4 and thus can be easily configured to be transferable onto the encoder 10 of Fig. 5) by a combiner or a combiner unit 64, thereby correcting the predicted sample filling.
[0229] For instance, in accordance with embodiments, the encoder 10 may subject the prediction residual for the predetermined block 13 (e.g. obtained by, or as a result, of the subtracter / prediction residual former 16) to a further machine learning predictor 38, which has been denoted by α in Fig. 5 as an example and further in the disclosure. For example, parameters of the machine learning transform predictor 38 may be denoted by ψaelsewhere in the disclosure, regardless of its specific implementation. The further ML predictor 38 may be configured to transform 57 the prediction residual from the sample domain to an analysis transform domain to obtain the analysis transform block 20 representing a transform domain prediction residual. Alternatively, the further ML predictor 38 may be configured to perform a transform transition 59 of the prediction residual from the sample domain to the analysis transform domain to obtain the analysis transform block 20 representing the transform domain prediction residual.
[0230] Before proceeding further with describing more details relating to Fig. 5, it should be noted that the ML transform predictor 34 is merely present in the prediction loop 37 of the encoder 10 and thus, having been already described in detail for the decoder of Fig. 4, repeating details of the ML transform predictor 34 as configured to be transferable onto the encoder are omitted for the sake of brevity and conciseness of the disclosure. Therefore, in describing Fig. 5, the present application rather focuses on the further ML transform predictor 38 of the encoder 10 which by virtue of being absent in the decoder 50 of Fig. 4 has not yet been described.
[0231] In accordance with embodiments, the further ML transform predictor 38 may be non-linear with respect to the analysis transform block 20. This means, for example, that the further ML transform predictor 38 may be implemented using concepts which exploit a non-linear dependence on the analysis transform block 20. In particular, the further ML transform predictor 38 may be implemented using concepts which exploit a non-linear dependence on the transform coefficients (e.g. a set of transform coefficients) of the analysis transform block 20. An example of such concepts is provided in the form of neural networks further in the disclosure, which may optionally be trainable using training data.
[0232] filing version, rmFurther, in accordance with embodiments, the further ML transform predictor 38 may be configured to additionally take as input one or more previously encoded samples neighbouring the predetermined block 13 (e.g. in other words, reference samples available to the encoder 10). In other words, the inputs (or for example, attributes, or equivalently, arguments) of the further ML transform predictor 38 may comprise both the one or more previously encoded samples neighbouring the predetermined block 13 and the transform coefficients of the analysis transform block 20.
[0233] According to embodiments, the further ML transform predictor 38 may be non-linear with respect to the one or more previously encoded samples neighbouring the predetermined block 13. That is, the further ML transform predictor 38 may be configured to combine the one or more previously encoded samples neighbouring the predetermined block 13 and the analysis transform block 20 in a non-linear manner. This means, for example, that the further ML transform predictor 38 may be implemented using concepts which exploit a non-linear dependence on both the already encoded samples in the neighborhood of the predetermined block 13 and the transform coefficients of the analysis transform block 20. An example of such concepts is provided in the form of neural networks further in the disclosure, which may optionally be trainable using training data.
[0234] Further, in accordance with embodiments, the further ML transform predictor 38 may be configured to additionally take as input the predicted sample filling. In other words, the inputs (or for example, attributes, or equivalently, arguments) of the further ML transform predictor 38 may comprise the predicted sample filling, the one or more previously encoded sample neighbouring the predetermined block 13 and the transform coefficients of the analysis transform block 20.
[0235] According to embodiments, the further ML transform predictor 38 may be non-linear with respect to the the predicted sample filling. That is, the further ML transform predictor 38 may be configured to combine the predicted sample filling, the one or more previously encoded samples neighbouring the predetermined block 13 and the analysis transform block 20 in a non-linear manner. This means, for example, that the further ML transform predictor 38 may be implemented using concepts which exploit a non-linear dependence on the predicted sample filling, the already encoded samples in the neighborhood of the predetermined block and the transform coefficients of the analysis transform block 20. An example of such concepts is provided in the form of neural networks further in the disclosure, which may optionally be trainable using training data.
[0236] filing version, rmIn accordance with embodiments, the analysis transform block 20 may be configured to sequentially transform (e.g. perform a plurality of transformations or mappings of the analysis transform block in a chain-wise or successive manner, or in other words, one after another) from the sample domain to an intermediate analysis transform domain. In addition, or as an alternative, to these embodiments, the analysis transform block 20 may be configured to sequentially transform (e.g. perform a plurality of transformations or mappings of the analysis transform block in a chain-wise or successive manner, or in other words, one after another) from the intermediate analysis transform domain to the analysis transform domain. Further, additionally or alternatively, in accordance with embodiments, the analysis transform domain may be configured to sequentially transform from a first intermediate analysis transform domain to a second intermediate analysis transform domain in the case where more than one intermediate analysis transform domain may be used, or preferred. For example, this may be the case wherein the analysis transform block 20 is decoded by making use of more than two analysis transforms, such as a scenario where a primary transform, a secondary transform and a tertiary transform are used for the same.
[0237] In accordance with embodiments, combinations of the aforementioned analysis transform domains, that is, the analysis transform domain and / or the intermediate analysis transform domain(s) may be configured to be at least one of (e.g. or any of): a DCT domain, a DST domain, a LFNST domain, a NSPT domain, a linear transform domain and an orthogonal transform domain.
[0238] Further, according to embodiments, transformations (e.g. or transforms, or mappings) relating (or, e.g. bridging) the analysis transform domain to the intermediate analysis transform domain(s) and / or the intermediate analysis transform domain(s) to the sample domain may be configured to be a linear transform or a orthogonal transform or both a linear and orthogonal transform. Further, for example, in embodiments wherein more than two transformations may be used, any of the intermediate analysis transformations may be configured to be a linear transform or a orthogonal transform or both a linear and orthogonal transform.
[0239] Moreover, the size of the transformations associated with the analysis transform block 20 may be such that it corresponds to the size of the analysis transform block 20 in terms of samples, i.e. the number of transform coefficients may equal the number of samples in block 13, but this may also be different. In other words, the number of transform coefficients of the analysis transform block 20 may be equal to a number (e.g. a count) of samples of the prediction residual formed by, or as a result of, the subtracter / prediction residual former 16 (shown in Fig.
[0240] 5). Alternatively, the size of the transformations may be such that it corresponds to that of a
[0241] filing version, rmsub-block (e.g. a sub-portion) of the analysis transform block 20, or for example, a sub-block (e.g. or a sub-portion) of the predetermined block 13.
[0242] So far, embodiments have been concentrated on the predetermined block 13 which may have associated therewith one or more of the following characteristics such as:
[0243] a) size of block 13 such as x samples wide and y sample high, and
[0244] b) prediction mode such as a selected intra prediction mode in case of block 13 being an intra predicted block, or an inter prediction mode in which case one or more motion vectors might be signaled in the data stream 40 for block 13, and
[0245] c) the type of the transformation (e.g. or transformation or mapping) of the analysis transform block 20 in case of, for instance, the encoder 10 supporting more than just one transformation type for transforming 35 or performing the transform transition 39 of the prediction residual from the sample domain to the analysis transform domain to obtain the analysis transform block 20 representing the transform domain prediction residual, and
[0246] d) the one or more previously encoded samples neighbouring the predetermined block 13 (e.g. or one or more reference samples, which may be available to the encoder 50 in the picture buffer).
[0247] Therefore, embodiments are presented according to which the above characteristics may be varied resulting in different implementations of the further machine learning transform predictor 38.
[0248] According to embodiments, the encoder 10 may be configured to support different block sizes for the predetermined block 13 and to obtain the analysis transform block 20 representing the transform domain prediction residual by varying a number of parameters defining the further ML transform predictor 38 and the parameters (e.g. meaning values of the parameters) defining the further ML transform predictor 38, depending on a block size of the predetermined block 13. In other words, for example, the encoder 10 may obtain the analysis transform block 20 representing the transform domain prediction residual by selecting a block size among the different sizes (e.g. and supported by the decoder) for the predetermined block 13, and thus, varies the parameters, and a number or a count there of, associated with further ML transform predictor 38 accordingly.
[0249] According to embodiments, the encoder 10 may be configured to support different intra prediction modes for the predetermined block 13 signalled in the data stream 40 and to obtain the analysis transform block 20 representing the transform domain prediction residual by filing version, rmvarying a number of parameters defining the further ML transform predictor 38 and the parameters (e.g. meaning values of the parameters) defining the further ML transform predictor 38, depending on an intra prediction mode of the predetermined block 13. In other words, for example, the encoder 10 may obtain the analysis transform block 20 representing the transform domain prediction residual by selecting an intra prediction mode among the different intra prediction modes (e.g. signalled in the data stream 40 and thus, supported by the encoder) for the predetermined block 13, and thus, varies the parameters, and a number or a count there of, associated with further ML transform predictor 34 accordingly. Further, the encoder 10 may be configured to support one or more groups of intra-prediction modes into which the different intra prediction modes are grouped for the predetermined block 13, wherein the manner in which the one or more intra-prediction groups are grouped out of the different intra-prediction modes may be signalled in the data stream 40 along with the different intra prediction modes. The encoder 10 may then be configured to obtain the analysis transform block 20 representing the transform domain prediction residual by varying a number of parameters defining the further ML transform predictor 38 and the parameters (e.g. meaning values of the parameters) defining the further ML transform predictor 38, depending on a group of intra prediction modes of the predetermined block 13. In other words, for example, the encoder 10 may obtain the analysis transform block 20 representing the transform domain prediction residual by selecting a group of intra prediction mode among the different groups of intra prediction modes (e.g. signalled in the data stream 40 and thus, supported by the encoder) for the predetermined block 13, and thus, varies the parameters, and a number or a count there of, associated with further ML transform predictor 38 accordingly.
[0250] According to embodiments, the encoder 10 may be configured to obtain the analysis transform block 20 representing the transform domain prediction residual by varying a number of parameters defining the ML transform predictor 38 and the parameters defining the ML transform predictor 38, depending on the one or more previously encoded samples neighbouring the predetermined block 13. In other words, for example, the encoder 10 may obtain the analysis transform block 20 representing the transform domain prediction residual by selecting one or more previously encoded samples among the one or more previously encoded samples neighbouring the predetermined block 13, and thus, varies the parameters, and a number or a count there of, associated with ML transform predictor 34 accordingly.
[0251] With respect to the transform domain prediction residual, the decoder 50 operates as follows: the analysis transform block representing the transform domain prediction residual may be obtained using the further M L transform predictor 38, which is implemented by a corresponding or associated neural network: That is, as shown in Fig. 6, the further ML transform predictor 38 (e.g. a) is associated or implemented by a neural network 41.
[0252] filing version, rmIn accordance with embodiments, the further ML transform predictor 38 (e.g. a) may be implemented by: an initial analysis transform (for example denoted using Aolater in the disclosure; and represented by 43a in Fig. 15 later in the disclosure), a neural network 41 and a combiner unit (represented by 43f in Fig. 15 later in the disclosure). The initial analysis transform may be configured to receive a sample domain prediction residual (represented by 43b in Fig. 15 later in the disclosure) for the predetermined block 13 (for example, this prediction residual may be represented by s-p), and to obtain an initial analysis transform domain prediction residual (e.g. for example denoted using u = A0(s-p) later in the disclosure) based on the sample domain prediction residual for the predetermined block 13. The neural network 41 may comprise an initial non-linear fully-connected layer (represented by 43c in Fig.
[0253] 15 later in the disclosure) and one or more succeeding layers (represented by 43d in Fig. 15 later in the disclosure). The neural network 41 may be configured to receive the sample domain prediction residual as inputs of the initial layer, receive an output of the initial layer as an input of the one or more succeeding layers.
[0254] The combiner unit may be configured to combine the initial analysis transform domain prediction residual and an output of a final layer (represented by 43e in Fig. 15 later in the disclosure) among the one or more succeeding layers to obtain the analysis transform block representing the transform domain prediction residual (e.g. the analysis transform domain prediction residual). For example, the combiner unit may simply add, or for example average, the initial analysis transform domain prediction residual and the output of a final layer among the one or more succeeding layers. In examples, the combiner unit, one or more thereof, may even combine the the initial analysis transform domain prediction residual and an output of a final layer among the one or more succeeding layers as per a weighted averaging operation, wherein weights may be related, or given, by additional or existing parameters of the NN 41.
[0255] The neural network 41 may, for instance, also be referred to as the analysis transform neural network 41 to further distinguish it from other neural networks described in this disclosure.
[0256] The attributes of the neural network 41 or, in different terms, the input(s) of the initial layer of the neural network 41, may comprise:
[0257] 1) one or more samples of the prediction residual (i.e. the sample domain prediction residual);
[0258] further, 2) the predicted sample filling of the predetermined block 13;
[0259] further, 3) the one or more previously encoded samples neighbouring the predetermined block
[0260] filing version, rm13. In other words, reference samples or already encoded samples available to the encoder in the picture buffer may be additionally taken as input by the neural network 41. A number (or a count) of the one or more previously encoded samples neighbouring the predetermined block which is input into the neural network 41 may vary anywhere from one previously encoded sample to all previously encoded samples of the one or more previously encoded samples neighbouring the predetermined block 13;
[0261] further, 4) a quantization parameter of the predetermined block 13 such as the quantization step size based on which the encoder quantizes or scales the transform coefficients. This optional input, shown as being part of round block 24 in Fig. 5, of the neural network 41 implementing the further ML predictor 38 has been illustrated with a dashed line. The quantization parameter may be configured to adjust the granularity at which the quantizer, or quantization block, 26, quantizes (e.g. scales) the transform coefficients of the analysis transform block 20. For instance, this quantization 26 may be performed prior to entropy coding 36 as shown in Fig. 5; and
[0262] further, 5) an index of the selected intra-prediction mode selected from a plurality of intraprediction modes (or different intra-prediction modes). This optional input, shown as being part of round block 24 in Fig. 5, of the neural network 41 implementing the further ML predictor 38 has been illustrated with a dashed line.
[0263] It is noted that both the options 4) and 5) have been been indicated in Fig. 5 by way of being depicted as schematically contained within a block / module 24, wherein the block / module 24 is connected (using a solid line) to the quantizer 26. The optional nature of the QP and the intra mode as inputs to the implementations of the ML predictors 34, 38 (e.g. NNs 51, 41) is denoted by the use of dashed lines in connection with the blocks 34, 38. Hence, these options, both individually or in combination, constitute distinct embodiments compared to the one exemplarily illustrated in Fig. 5.
[0264] In accordance with embodiments, the final layer of the one or more succeeding layers of the neural network 41 is a linear layer. It is to be understood that linearity of the final layer relates to the attributes, or inputs, of the neural network 41 implementing the further ML transform predictor 38 such as any of, or at least one of, the one or more samples of the prediction residual (i.e. the set of samples of the sample domain prediction residual), the predicted sample filling of the predetermined block 13, the one or more previously encoded samples neighbouring the predetermined block 13, the quantization parameter of the predetermined block 13, and the the index of the selected intra-prediction mode.
[0265] filing version, rmFurther, in accordance with embodiments, the the neural network 41 may be configured to receive the one or more samples of the prediction residual (i.e. the set of samples of the sample domain prediction residual) concatenated (e.g. meaning, joining these together in sequence, or as a single entity) with the predicted sample filling as inputs of the initial layer of the neural network 41. Even further, in accordance with embodiments, the neural network 41 may be configured to receive any, or at least one, of the following: the one or more previously encoded samples neighbouring the predetermined block 13, the quantization parameter of the predetermined block 13, and the index of a selected intra-prediction mode selected from a plurality of intra-prediction modes concatenated (e.g. meaning, joining any of these together in sequence, or as a single entity) with the above described inputs, i.e. the predicted sample filling and the one or more samples of the prediction residual (i.e. the set of samples of the sample domain prediction residual), as inputs of the initial layer of the neural network 41. For instance, further to the above options, additional reconstruction information obtainable by the encoder 10 may be taken as inputs of the initial layer of the neural network 41 along with examples of the above mentioned inputs. For instance, the additional reconstruction information obtainable by the encoder 10 may be concatenated along with examples of the above mentioned inputs of the initial layer of the neural network 41.
[0266] According to embodiments, the initial analysis transform, or transformation or mapping, (e.g. A0) may be an orthogonal transform. In other words, the further ML transform predictor 34 may be initialized by an orthogonal transform to obtain the analysis transform block the transform domain prediction residual. Additionally, In accordance with embodiments, the initial synthesis transform may be at least one of, or any of, a DCT, a DST, a LFNST, a NSPT, and a linear transform.
[0267] As another optional feature, the analysis transform NN 41 in the above description comprises a non-linear activation function, as an example a rectified linear unit (ReLu). It is to be noted that other forms which introduce any non-linearities may be used in combination with or without the ReLu.
[0268] For instance, it holds in general, according to embodiments, the activation functions in the layers of the analysis transform NN 41 may include non-linear functions such as leaky ReLu, tanh, binary step function, SELU, ELU, sigmoid / logistic and / or parametric ReLu. This means, for example, that a first grouping of the layers of the NN 41 may comprise non-linear activation functions different from a second grouping of the layers of the NN 41. As another optional feature, it also holds in general, that any of the layers of the NN 41 may comprise more than one non-linear activation function. Additionally or alternatively, as an instantiation, some of the
[0269] filing version, rmlayers, or some neurons among the layer, of the NN 41 may even comprise an identity function as the activation function.
[0270] According to embodiments, the encoder 10 may support one or more of a number of different initial analysis transforms for the further ML transform predictor 38, wherein each initial synthesis transform is related to a neural network 41ᵢ implementing the further ML transform predictor 38 and may obtain the analysis transform block 20 representing the transform domain prediction residual (e.g. the analysis transform domain prediction residual) by selecting and using a corresponding neural network 41ᵢ related to an initial analysis transform among the different initial analysis transforms of the predetermined block 13. In other words, the encoder 10 may support different neural networks 41ᵢ wherein the index i may range from anywhere including one to including a maximal number of different initial analysis transforms supported by the encoder 10. The encoder 10 may perform a selection among the different neural networks, wherein the different neural networks are classified according to the different initial analysis transform of the ML analysis transform predictor 38. The different neural networks may be further classified according to a number (e.g. a count of) of parameters defining the NN 41ᵢ and values of the parameters of the NN 41ᵢ which the encoder 10 may be configured to vary.
[0271] According to embodiments, the encoder 10 may support one or more of a number of different block sizes for the predetermined block 13, wherein each block size is related to a neural network 41ᵢ implementing the further ML transform predictor 38 and may obtain the analysis transform block 20 representing the transform domain prediction residual (e.g. the analysis transform domain prediction residual) by selecting and using a corresponding neural network 41ᵢ related to a block size among the different block sizes of the predetermined block 13. In other words, the encoder 10 may support different neural networks 41ᵢ wherein the index i may range from anywhere including one to including a maximal number of different block sizes of the predetermined block 13 supported by the encoder 10. The encoder 10 may perform a selection among the different neural networks, wherein the different neural networks are classified according to the different block sizes of the predetermined block 13. The different neural networks may be further classified according to a number (e.g. a count of) of parameters defining the NN 41ᵢ and values of the parameters of the NN 41ᵢ which the encoder 10 may be configured to vary.
[0272] Further, in the case of a block size equalling a mutually transposed block size among the different block sizes of the predetermined block 13, the encoder 10 may obtain the analysis transform block 20 representing the transform domain prediction residual (e.g. the analysis transform domain prediction residual) by transposing the input of the initial layer and the output of the final layer of a corresponding neural network 41ᵢ related to the mutually transposed block
[0273] filing version, rmsize of the predetermined block 13, and using the parameters of the neural network 41ᵢ related to the mutually transposed block size of the predetermined block 13.
[0274] Further, in the case of the block size exceeding a predetermined block size value, the corresponding neural network 41ᵢ takes, as an input, a sub-portion (e.g. a sub-block, or a subset) of the prediction residual (i.e. the sample domain prediction residual). In other words, the input of the NN 41ᵢ in such an instance would be a subset of the set of samples of the sample domain prediction residual.
[0275] For example, in case of a sub-portion of the analysis transform block 20 (e.g. a N x N subblock of the analysis transform block 20) being taken as input to the neural network 41ᵢ, the neural network 41ᵢ may be configured to output, such as an output of its final layer, corresponding to the inputted sub-portion of the analysis transform block 20. Further, in addition to this, the combiner unit, associated with the neural network 41ᵢ implementing the further ML transform predictor 38, may be configured to combine the initial analysis transform domain prediction residual and the output based on the inputted sub-portion of the analysis transform block to obtain the analysis transform block. In this regard, for example, it may be the NN 41ᵢ may be configured to output only based on the inputted sub-portion of the analysis transform block, which then the combiner unit may combine with the initial analysis transform domain prediction residual to obtain the analysis transform block.
[0276] For example, the sub-portion of the analysis transform block 20 may comprise a N x N subblock of the analysis transform block 20, with the N x N sub-block of the analysis transform block comprising a zero frequency or DC transform coefficient. It may be that the sub-portion of the synthesis transform block 20 is obtained by configuring, e.g. setting, transform coefficients associated with a set of specific frequencies, each of which exceeds a predetermined frequency threshold associated with the transform block, to zero. In this regard, transform coefficients associated with the set of specific frequencies which are set to zero may be referred to as high frequency transform coefficients, while the transform coefficients associated with the sub-portion may be referred to as low frequency transform coefficients. The predetermined frequency threshold which may help demarcate, or distinguish, or categorize, the low-frequency and high-frequency transform coefficients, may be varied across the NN 41ᵢ depending on application or use-case. In other words, the sub-portion of the analysis transform block 20 may correspond to a set of transform coefficients being lowest in terms of frequency, the set of transform coefficient being derived from a super set of transform coefficients subsuming the set of transform coefficients being lowest in terms of frequency. For example, the sub-portion may be 2x2 or 4x4 or 8x8 or 16x16 block comprising the lowest frequency coefficients, such sub-portions may comprise non-square, or rectangular, blocks additionally or alternatively.
[0277] filing version, rmAccording to embodiments, the encoder 10 may support one or more of a number of different intra-prediction modes for the predetermined block 13 or one or more of a group of intraprediction modes into which the different intra-prediction modes are grouped, both of which are related to a neural network 41ᵢ implementing the further ML transform predictor 38 and may obtain the analysis transform block 20 representing the transform domain prediction residual (e.g. the analysis transform domain prediction residual) by selecting and using a corresponding neural network 41ᵢ related to an intra-prediction mode, or a group of intraprediction modes, among the different intra-prediction modes, or the groups thereof, of the predetermined block 13. In other words, the encoder 10 may support different neural networks 41ᵢ wherein the index i may range from anywhere including one to including a maximal number of different intra-prediction modes or a maximal number of different groups of intra-prediction modes of the predetermined block 13 supported by the encoder 10. The encoder 10 may perform a selection among the different neural networks 41ᵢ, wherein the different neural networks 41ᵢ are classified according to the different intra-prediction modes or groups thereof of the predetermined block 13. The different neural networks 41 j may be further classified according to a number (e.g. a count of) of parameters defining the NN 41ᵢ and values of the parameters of the NN 41ᵢ which the encoder 10 may be configured to vary.
[0278] In other words, it may be that, with respect to obtaining the analysis transform block 20 representing the transform domain prediction residual (e.g. the analysis transform domain prediction residual), one or more of the following may be true:
[0279] 1) the encoder 10 may manage and comprise a set of NNs 41ᵢ (and possibly associated parameters and a number or count thereof) for each block size or each of different groups of block sizes into which all supported block sizes are grouped, and the encoder 10, when obtaining (or e.g., computing) the analysis transform block 20 representing the transform domain prediction residual (e.g. the analysis transform domain prediction residual), firstly selects the corresponding - the one fitting to the size of block 12 - set of NNs 41ᵢ (and possibly associated parameters and a number or count thereof) before acting and behaving as described above with respect to that set of NNs 41ᵢ (and possibly associated parameters and a number or count thereof). That is, for instance, the encoder 10 may select NNs 41ᵢ using a predetermined criterion which is associated with the supported block sizes of the block 13.
[0280] a. In accordance with embodiments, the grouping of the blocks may group mutually transposed blocks, i.e. blocks and correspondingly transposed blocks comprising the same size, together. In other words, blocks of shape Wx H and H x W, may be grouped together, where W and H denote dimensions of the
[0281] filing version, rmblock, e.g. with W being the width of one block in samples and H being the height of this block in samples. Without loss of generality, in case of the block shape W x H, the synthesis transform NN 41ᵢ would process its inputs and outputs without additional operations whereas in case of the transposed block shape H x W, the inputs, namely the set of transform coefficients and the predicted sample filling 18 (i.e. the prediction signal) would be transposed before application onto the NN, thereby allowing the analysis transform block 20 representing the transform domain prediction residual (e.g. the analysis transform domain prediction residual) obtained (e.g. computed) by the NN 41ᵢ for the block shape W x H to be used, and subsequently, the output of the NN would be transposed back into the shape H x W thus obtaining (e.g. computing) the analysis transform block 20 representing the transform domain prediction residual (e.g. the analysis transform domain prediction residual) for the transposed block shape H x W.
[0282] b. In accordance with embodiments, if the block size selected by the encoder 10 is larger, or greater, than a predetermined block size value, the encoder 50 may take a sub-portion (e.g. or sub-block) of the prediction residual (e.g. or a subset of the samples of the sample domain prediction residual). For instance, this subportion of the prediction residual may correspond to the predetermined block size value. Equivalently speaking, a sub-portion of the analysis transform block may be taken by the encoder 50, for example, this may correspond to a subset of the set of transform coefficients of the analysis transform block 20.
[0283] ) the encoder 10 may use (and possibly select), for instance, the same set of NNs (and possibly associated parameters and a number or count thereof) irrespective of the prediction mode of block 13, but it could, naturally, be that the encoder 10 may manage and comprise a set of NNs 41ᵢ (and possibly associated parameters and a number or count thereof) for each intra-prediction mode or each of different groups of intraprediction modes into which all supported intra-prediction mods are grouped, and the encoder 10, when obtaining (e.g. computing) the analysis transform block 20 representing the transform domain prediction residual (e.g. the analysis transform domain prediction residual), may firstly select the corresponding - the one fitting to the intra prediction mode of block 13 - set of NNs (and possibly associated parameters and a number or count thereof) before acting and behaving as described above with respect to that set of NNs (and possibly associated parameters and a number or count thereof) such as by use of the index of the selected intra-prediction mode for block 13 which index might be signaled in the data stream.
[0284] filing version, rm3) the encoder 10 may use (and possibly select), for instance, the same set of NNs (and possibly associated parameters and a number or count thereof) irrespective of the transform type of the initial analysis transform, but it could, naturally, be that the encoder 10 manages and comprises a set of NNs 41ᵢ (and possibly associated parameters and a number or count thereof) for each transform type or each of different groups of transform types into which all supported transform types are grouped, and the encoder, when obtaining (e.g. computing) the analysis transform block 20 representing the transform domain prediction residual (e.g. the analysis transform domain prediction residual), may firstly select the corresponding - the one fitting to the transform type of the initial synthesis transform - set of NNs (and possibly associated parameters and a number or count thereof) before acting and behaving as described above with respect to that set of NNs (and possibly associated parameters and a number or count thereof).
[0285] 4) Naturally, it may be that the encoder 10 obtains (e.g. computes) the analysis transform block 20 representing the transform domain prediction residual (e.g. the analysis transform domain prediction residual) only to a subset of the possible block characteristics a - d listed above, such as only to blocks in a certain range of sizes, the initial synthesis transform is of one or more of certain transform types, and blocks of one or more intra-prediction modes. Moreover, inter-predicted blocks may or may not be subject to correction of the predicted sample filling using the sample domain prediction residual.
[0286] With respect to the above discussion, parameters of the NN 41ᵢ may comprise a number (or a count) of hidden layers (i.e. the one or more succeeding layers), and weights, and a number or a count thereof, and biases, and a number or a count thereof.
[0287] In accordance with embodiments, the number (or count) of transform coefficients in the set of transform coefficients of the analysis transform block 20 representing the transform domain prediction residual may be equal to a number of samples of the prediction residual (i.e. the sample domain prediction residual).
[0288] In accordance with embodiments, the set of transform coefficients of the analysis transform block 20 representing the transform domain prediction residual may be encoded by the encoder 10 into the data stream 40 based on a scan order. For example, the scan order defined within, or for encoding of, the analysis transform block 20 may be a diagonal scan order starting at the highest frequency component, e.g., a bottom right position in the analysis transform block 20 and at the DC component, which may be positioned at the top left of the analysis transform block 20. In other words, the scan order may roughly traverse the set of transform coefficients from higher to lower frequency, so that transform coefficients associated with lower filing version, rmfrequency components may be listed further down in the scan order. However, the scan order may not necessarily follow strictly an order set up by the frequency, in particular, as transform coefficients may be associated with vertical and horizontal frequency components, there may be multiple transform coefficients, for which their order according to higher or lower frequency is not well defined.
[0289] Hitherto, in the description of the encoder 50 as exemplarily illustrated in Fig. 5 has been restricted to known or existing intra-prediction modes. The prediction module 14 for predicting the predicted sample filling of the predetermined block 13 may be carried out using the ML prediction predictor 36 which may be implemented by the neural network 61 as illustrated in Fig. 6. Since such a neural network has already been described for implementing the ML prediction predictor for the decoder 50 of Fig. 4, reference is made to those details being adapted so as to be applicable on to the encoder 10 here of Fig. 5, therefore such details are not repeated herein.
[0290] So far optional details of the encoder 10 such as the entropy encoder 31 and the in-loop filter 68 of the prediction loop 37 as shown in Fig. 5, have not been described, the present application proceeds further with description of such details.
[0291] For instance, prior to the entropy encoding 31, the quantizer 26, may quantize the transform coefficients of the analysis transform block 20, thereby obtaining quantized transform coefficients. The quantized transform coefficients may further be used for deriving a set of quantization indices. In particular, the quantization indices may be represented by signs and absolute values. For example, the quantization 26 may include a scaling, and optionally, a rounding operation.
[0292] The encoder 10 of Fig. 5 further comprises an entropy encoder 31, which entropy encodes the set of quantization indices into the data stream 40. For example, entropy coding the set of quantization indices into the data stream 40 may be performed by encoding binary representations of the set of quantization indices into the data stream 40 followed by subjecting the binary representations of the set of quantization indices to an entropy coding scheme. For example, such an entropy coding scheme may involve a process referred to as dependent quantization or trellis-coded quantization, in which, depending upon the sign, or parities, of the previously encoded transform coefficients (e.g. the quantized transform coefficients), the encoder switches among a set of quantizers.
[0293] It is also indicated in Fig. 5 that the prediction residual (i.e. the sample domain prediction residual) is exemplarily denoted by s - p; the transform coefficients of the analysis transform block 20 being subjected to the ML transform predictor 38 (itself being exemplarily denoted by a) are denoted by v, in harmony with equation (1.4.1) further in the disclosure; quantized
[0294] filing version, rmtransform coefficients obtained by the quantizer 26 are exemplarily denoted by Q(v, A) where in the QP is exemplarily denoted by Δ; dequantized transform coefficients are exemplarily denoted by v, in harmony with equation (1.4.2) and similar to equation (1.1.4); the transform coefficients of the synthesis transform block 32 being subjected to the ML transform predictor 34 (itself being exemplarily denoted by p) are exemplarily denoted by p(v), in harmony with equation (1.4.5) described further in the disclosure; the predicted sample filling is exemplarily denoted by p and the reconstructed signal, or reconstructed transform coefficients, are denoted by s in harmony with equation (1.4.3) described further in the disclosure.
[0295] For instance, as shown in Fig. 4 by way of dashed lines, the decoder 50 may optionally comprise an in-loop filtering module 68 configured to filter a portion of the reconstructed signal as formed by the combiner 64 to obtain a filtered reconstructed signal for a purpose of reducing compression artefacts in the prediction module 14.
[0296] The present application now proceeds further with description of an implementation of the encoder 90 illustrated in Fig. 7 in accordance with embodiments of the second aspect. In the following, embodiments of the invention according to the second aspect are described, which may optionally be implemented with respect to Fig. 1, Fig. 2 and / or Fig. 3, wherein the features described earlier in the disclosure may be combined with the embodiments described below individually or in combination with each other.
[0297] Fig. 7 shows an embodiment of an encoder 90 for block based encoding of a predetermined block 13 of a picture 12. As said, the encoder 90 may be a video encoder or a picture encoder which comprises one or more of the functionalities described above with respect to Fig. 1 to 3.
[0298] In line with what has been already been stated earlier, this picture 12 may either be a part of a video sequence or be a part, or possibly a whole, of an image such as a standalone still image. The block 13 might be an inter-predicted block or an intra-predicted block. As already mentioned, the encoder 90 performs the encoding in a block-wise or block-based manner. That is, the encoder 90 subdivides the picture 12 into one or more blocks, in units of which the encoder 90 decodes the picture 12 into the data stream 40. Generally, the subdivision may end up into one or more blocks of constant size such as an array of blocks arranged in rows and columns or into one or more blocks of different block sizes such as derived by use of a hierarchical multi-tree subdivisioning applied to the whole picture area of picture 12, or derived from a pre-partitioning of picture 12 into an array of tree blocks which are then subject to multitree subdivisioning. The hierarchical subdivision information may be signalled in the data stream 40. It is noted that these examples shall not be considered as excluding other possible approaches 10 of sub divisioning the picture 12 into one or more blocks. Thus, the
[0299] filing version, rmpredetermined block 13 which the encoder 90 encodes may be formed from the picture 12 using such or similar approaches.
[0300] Note that the predetermined block 12 may by a quadratic block or a rectangular or non-square block, while any other shape may be used as well.
[0301] The encoder 90 predicts a predicted sample filling of the predetermined block 13 using an an intra-prediction mode which the encoder 90 inserts into the data stream 40, such as encoding an index of the intra-prediction mode into the data stream 40. In particular, the predicted sample filling of the predetermined block 13 might be predicted by selecting an intra-prediction mode out of a plurality of intra-prediction modes which the encoder 90 supports. Examples of the plurality of intra-prediction modes supported by the encoder 90 include, in accordance with embodiments, one or more of a DC mode, a planar mode, a plurality of angular modes, one or more matrix-based intra prediction (MIP) modes, according to each of which the predicted sample filling is obtained by a multiplication between a matrix associated with the respective matrix-based intra prediction mode and a vector obtained from sample values in a neighborhood of the predetermined block, and one or more intra-subpartition modes, according to each of which the predicted sample filling is obtained by subdividing the predetermined block into subpartitions and intra-predicting the subpartitions sequentially with correcting a subpartition using the prediction residual before intra-predicting a next subpartition. Any combination of the just-mentioned modes shall form the supported set of intra-prediction modes of the encoder 90.
[0302] Note that, in any of the just-described intra-prediction modes, one or more previously encoded samples neighboring the predetermined block 13 based on which the predicted sample filling is predicted need not to be directly adjacent to the predetermined block 13. Rather, the neighborhood, or equivalently described a region in the vicinity of the predetermined block 13, within which the encoded samples are located based on which the predicted sample filling is predicted may also cover or merely cover the one or more previously encoded samples which are not directly adjacent to the block boundary. The previously encoded samples based on which the intra-prediction is performed may be located using a certain criterion such as, for example, one locating same in terms of a specific maximum distance from the block boundary (e.g. boundary of the predetermined block 13) or the like. That is, for instance, the neighborhood may denote a template of previously encoded samples each having a distance from the block boundary of block 13 which is equal to or smaller than a specific distance and this specific distance may be defined by default or may be transmitted in the data stream 40.
[0303] filing version, rmThe one or more intra-subpartition modes may subdivide the predetermined block 13 into subpartitions along a first direction (e.g. a horizontal direction) and / or along a second direction (e.g. a vertical direction). In case of subdividing only along one direction, the subpartitions might be as wide as the block 13 in a direction perpendicular to the direction of subdividing the block 13 into the partitions. The sub-partitions might then be intra-predicted and corrected using the block’s prediction residual sequentially, thereby improving the intra predictor for subsequently intra predicted sub-partitions.
[0304] Again, the illustrations provided in Fig. 7 are merely an example. The encoder 90 might support further intra-prediction modes or fewer or a different collection. Further, it is feasible that the encoder 90 may select more than one intra-prediction mode for predicting the predicted sample filling so as to perform the prediction using multiple hypotheses. For the selection of the intra prediction mode, the encoder 90 may encode an index into the data stream 40 for block 13 which is indicative of the selected intra prediction mode.
[0305] The predicted sample filling is generated using the selected intra-prediction mode based on the values of already encoded samples (e.g. the one or more previously encoded sample, or equivalently also referred to as reference samples; e.g. these encoded samples or reference samples may be available to the encoder 90 via the picture buffer) in the neighborhood of the predetermined block 13. As described, these samples in area may directly abut the circumference, or boundary, of block 13, but this may additionally or alternatively comprise other samples. The already encoded samples in the area may be adjacent, or located next, to the predetermined block along a left-hand boundary of the predetermined block 13 and a top boundary of the predetermined block 13.
[0306] Again, concepts of encoder 90 with respect to an intra-predicted block have been described so far, these concepts would be readily transferable onto inter-predicted blocks, so as to be configured to apply these concepts onto such inter-predicted block alternatively or additionally. Further, the encoder 90 may use intra-prediction in combination with inter-prediction to predict the predicted sample filling 18. For example, the encoder 90 may be configured to combine a predictor resulting from intra-prediction with one obtained by inter-prediction. Details of such a predictor such as a ML prediction predictor for intra-prediction will be described later in this disclosure. For inter-prediction, the encoder 90 may be configured to encode information on motion vector(s) for block 13 into the data stream 40 so as to perform motion-compensated prediction of block 13.
[0307] filing version, rmThe encoder 90 encodes an analysis transform block 20 of a prediction residual for the predetermined block 13 into the data stream 40. The analysis transform block 20 is denoted by T as an example in Fig. 7. The analysis transform block 20 may relate to an analysis transform (or transformation or mapping) (e.g. or a forward transform) of the prediction residual (i.e. the sample domain prediction residual) of the predetermined block 13. The prediction residual for the predetermined block 13 may be formed by the subtractor (or prediction residual former) 16 using the predicted sample filling (e.g. p) and samples of the original signal (e.g. s). Encoding the analysis transform block 20 means that transform coefficients of the analysis transform block 20, which are transmitted in the data stream 40 and available to the encoder are encoded into the data stream 40.
[0308] The encoding of the analysis transform block 20 is performed by encoding the transform coefficients of the analysis transform block 20 of the prediction residual by subjecting the analysis transform block 20, accompanied with the predicted sample filling, to a machine learning reductive offset predictor 21 (e.g. reduction function, which may be denoted by V later in the disclosure) so as to obtain a reductive offset block (e.g. denoted by g in Fig. 7 and later in the disclosure).
[0309] Encoding the transform coefficients of the analysis transform block 20 of the prediction residual is further comprises correcting each transform coefficient of the analysis transform block 20 (e.g. a transform coefficient t / ,of the set of transform coefficients u) based on a corresponding absolute value of the reductive offset block (e.g. abs(g) )and the sign of the transform coefficient (e.g. sgn(u) where sgn denotes a sign (or signum) function) to obtain a corresponding corrected transform coefficient of a modified transform block 22 lower in magnitude (e.g. a corrected transform coefficient v, of the set of transform coefficients v). Further, the encoder 90 encodes the modified transform block 22 into the data stream 40. By this measure, the encoder 90 improves the analysis transform block 20 by obtaining and encoding the modified transform block 22, each transform coefficient of the latter being lower in magnitude than that of the former, into the data stream 40. In other words, since the transform coefficients of the analysis transform block 20 are corrected on the basis of their signs and absolute value of the obtained reductive offset block, transform coefficients of the modified transform block 22 being smaller in magnitude are obtained, thereby reducing the bitrate cost of encoding the modified transform block 22 as compared to the analysis transform block 20.
[0310] Further, as part of a rate-distortion, R / D optimizer, and / or a prediction loop 37 of the encoder 90, the encoder 90 subjects the modified transform block 22, accompanied with the predicted
[0311] filing version, rmsample filling, to a machine learning improvement predictor 30 so as to obtain an additive offset block, for example denoted by f in Fig. 7 and elsewhere in the disclosure. The encoder 90 forms a sum of the modified transform block 22 and the additive offset block to obtain an improved transform block 23. The encoder 90 subjects the improved transform block 23 to a synthesis transform block 32, denoted by T-1in Fig. 7, and corrects the predicted sample filling using the prediction residual. In other words, the adder / combiner 64 may simply combine (e.g. add, or may even perform an averaging, or a weighted sum) of the prediction residual as an output of the the synthesis transform block 32 and the predicted sample filling to obtain a corrected predicted sample filing. This corrected predicted sample filling may then be provided to the picture buffer, as shown in Fig. 7.
[0312] In accordance with embodiments, either one of the ML reductive offset predictor 21 and the ML improvement predictor 30 or both of the ML reductive offset predictor 21 and the ML improvement predictor 30 may additionally take as input a quantization parameter, QP, of the predetermined block 13. In other words, attributes of the ML reductive offset predictor 21 may comprise the quantization parameter of the predetermined block 13 in addition to the set of transform coefficients (e.g. one or more transform coefficients of the set of transform coefficients) of the analysis transform block 20 while attributes of the ML improvement predictor 30 may comprise the quantization parameter of the predetermined block 13 in addition to the set of transform coefficients (e.g. the set of reconstructed transform coefficients; e.g. one or more transform coefficients of the set of transform coefficients) reconstructed as part of the prediction loop 37, e.g. for reconstructing the predetermine block 13.
[0313] It should be noted that block 21 and the block 30 in Fig. 7 denote that the corresponding transforms / transformations of the modified transform block 22 and the improved transform block 23 might be implemented by the corresponding ML reductive offset predictor 21 and the corresponding ML improvement predictor 30 rather than influencing or existing in addition to the transforms (or, for example, transformations or mappings) of the modified transform block 22 and the improved transform block 23 to which the corresponding ML reductive offset predictor 21 and the corresponding ML improvement predictor 30 are attached to.
[0314] It should be noted that on the one hand, the ML reductive offset predictor 21 subjects the analysis transform block 20 to obtain the modified transform block 22 being modified (or reduced or corrected) in a reductive manner meaning that the transform coefficients are smaller in magnitude (or value) while on the other hand, the ML improvement predictor 30 subjects the reconstructed transform coefficients, before they are subjected to the synthesis transform block 32, to obtain an improved transform block being improved (or modified or
[0315] filing version, rmcorrected) in an additive manner meaning that the reconstructed transform coefficients are greater (or larger) in magnitude (or value).
[0316] Further, in accordance with embodiments, the ML reductive offset predictor 21 may be configured to additionally take as input any, or at least one, of: the one or more previously encoded samples neighbouring the predetermined block 13, the predicted sample filling, and the index of the selected intra prediction mode out of a plurality of intra-prediction modes. In other words, the inputs (or for example, attributes, or equivalently, arguments) of the ML reductive offset predictor 21 may comprise any of, or at least one of: the one or more previously encoded samples neighbouring the predetermined block 13, the predicted sample filling, and the index of the selected intra prediction mode out of a plurality of intra-prediction modes.
[0317] According to embodiments, the ML reductive offset predictor 21 may be non-linear with respect to any, or at least one, of: the analysis transform block 20 (i.e. with respect to one or more of the set of transform coefficients of the analysis transform block 20), the predicted sample filling, the quantization parameter of the predetermined block, one or more previously encoded samples neighbouring the predetermine block 13 and the index of the selected intra-prediction mode. That is, the ML reductive offset predictor 21 may be configured to combine the analysis transform block 20 (i.e. the one or more of the set of transform coefficients of the analysis transform block 20), the predicted sample filling, the quantization parameter of the predetermined block, the one or more previously encoded samples neighbouring the predetermine block 13 and the index of the selected intra-prediction mode in a non-linear manner. This means, for example, that the ML reductive offset predictor 21 may be implemented using concepts which exploit a non-linear dependence between any, or at least one, of: the analysis transform block 20 (i.e. the one or more of the set of transform coefficients of the analysis transform block 20), the predicted sample filling, the quantization parameter of the predetermined block, the one or more previously encoded samples neighbouring the predetermine block 13 and the index of the selected intra-prediction mode. An example of such concepts is provided in the form of neural networks further in the disclosure, which may optionally be trainable using training data.
[0318] Further, in accordance with embodiments, the ML improvement predictor 30 may be configured to additionally take as input any, or at least one, of: the one or more previously encoded samples neighbouring the predetermined block 13, the predicted sample filling, and the index of the selected intra prediction mode out of a plurality of intra-prediction modes. In other words, the inputs (or for example, attributes, or equivalently, arguments) of the ML improvement predictor 30 may comprise any of, or at least one of: the one or more previously
[0319] filing version, rmencoded samples neighbouring the predetermined block 13, the predicted sample filling, and the index of the selected intra prediction mode out of a plurality of intra-prediction modes.
[0320] According to embodiments, the ML improvement predictor 30 may be non-linear with respect to any, or at least one, of: the analysis transform block 20 (i.e. with respect to one or more of the set of transform coefficients of the analysis transform block 20), the modified transform block 22 (i.e. with respect to one or more of the set of modified (or reduced) transform coefficients of the modified transform block 22), the predicted sample filling, the quantization parameter of the predetermined block, one or more previously encoded samples neighbouring the predetermine block 13 and the index of the selected intra-prediction mode. That is, the ML reductive offset predictor 21 may be configured to combine the analysis transform block 20 (i.e. the one or more of the set of transform coefficients of the analysis transform block 20), the modified transform block 22 (i.e. the one or more of the set of modified (or reduced) transform coefficients of the modified transform block 22), the predicted sample filling, the quantization parameter of the predetermined block, the one or more previously encoded samples neighbouring the predetermine block 13 and the index of the selected intra-prediction mode in a non-linear manner. This means, for example, that the ML reductive offset predictor 21 may be implemented using concepts which exploit a non-linear dependence between any, or at least one, of: the analysis transform block 20 (i.e. the one or more of the set of transform coefficients of the analysis transform block 20), the modified transform block 22 (i.e. the one or more of the set of modified (or reduced) transform coefficients of the modified transform block 22), the predicted sample filling, the quantization parameter of the predetermined block, the one or more previously encoded samples neighbouring the predetermine block 13 and the index of the selected intra-prediction mode. An example of such concepts is provided in the form of neural networks further in the disclosure, which may optionally be trainable using training data.
[0321] So far, embodiments have been concentrated on the predetermined block 13 which may have associated therewith one or more of the following characteristics such as:
[0322] a) size of block 13 such as x samples wide and y sample high, and
[0323] b) prediction mode such as a selected intra prediction mode in case of block 13 being an intra predicted block, or an inter prediction mode in which case one or more motion vectors might be signaled in the data stream 40 for block 13, and
[0324] c) the type of the transformation (e.g. or transformation or mapping) of the analysis transform block 20 and of the synthesis transform block 32 in case of, for instance, encoder 90 supporting more than just one transformation type of: the analysis
[0325] filing version, rmtransform block 20 to obtain the set of transform coefficients representing the transform domain prediction residual and the synthesis transform block 32 to obtain the sample domain prediction residual, and
[0326] d) the one or more previously encoded samples neighbouring the predetermined block 13 (e.g. or one or more reference sample, which may be available to the encoder 90 in the picture buffer).
[0327] Therefore, in the following, embodiments are presented according to which the above characteristics may be varied resulting in different implementations of the machine learning reductive offset predictor 21 and / or the machine learning improvement predictor 30.
[0328] According to embodiments, the encoder 90 may be configured to support different block sizes for the predetermined block 13 and for obtaining the modified transform block 22 by varying a number of parameters defining the ML reductive offset predictor 21 and the parameters defining the ML reductive offset predictor 21, depending on a block size of the predetermined block 13. In other words, for example, the encoder 90 may obtain the modified transform block 22 by selecting a block size among the different sizes (e.g. and supported by the encoder 90) for the predetermined block 13, and thus, varies the parameters, and a number or a count there of, associated with the ML reductive offset predictor 21 accordingly.
[0329] According to embodiments, the encoder 90 may be configured to support different intra prediction modes for the predetermined block 13 signalled in the data stream 40 and for obtaining the modified transform block 22 by varying a number of parameters defining the ML reductive offset predictor 21 and the parameters defining the ML reductive offset predictor 21, depending on an intra prediction mode of the predetermined block 13. In other words, for example, the encoder 90 may obtain the modified transform block 22 by selecting an intra prediction mode among the different intra prediction modes (e.g. signalled in the data stream 40 and thus, supported by the encoder 90) for the predetermined block 13, and thus, varies the parameters, and a number or a count there of, associated with the ML reductive offset predictor 21 accordingly. Further, the encoder 90 may be configured to support one or more groups of intra-prediction modes into which the different intra prediction modes are grouped for the predetermined block 13, wherein the manner in which the one or more intra-prediction groups are grouped out of the different intra-prediction modes may be signalled in the data stream 40 along with the different intra prediction modes. The encoder 90 may then configured to obtain the modified transform block 22 by varying a number of parameters defining the ML reductive offset predictor 21 and the parameters defining the ML reductive offset predictor 21, depending on a group of intra prediction modes of the predetermined block 13. In other words, for example, the encoder 90 may obtain the modified transform block 22 by selecting a group of intra prediction mode among the different groups of intra prediction modes (e.g. signalled in filing version, rmthe data stream 40 and thus, supported by the decoder) for the predetermined block 13, and thus, varies the parameters, and a number or a count there of, associated with the ML reductive offset predictor 21 accordingly.
[0330] According to embodiments, the decoder 90 may be configured to obtain the modified transform block 22 by varying a number of parameters defining the ML reductive offset predictor 21 and the parameters defining the ML reductive offset predictor 21, depending on the one or more transform (e.g. transformation, or mapping) types of the analysis transform block 20. In other words, for example, the encoder 90 may obtain the modified transform block 22 by selecting one or more transform (e.g. transformation, or mapping) types of the analysis transform block 20, and thus, varies the parameters, and a number or a count there of, associated with the ML reductive offset predictor 21 accordingly.
[0331] According to embodiments, the decoder 90 may be configured to obtain the modified transform block 22 by varying a number of parameters defining the ML reductive offset predictor 21 and the parameters defining the ML reductive offset predictor 21, depending on the one or more previously encoded samples neighbouring the predetermined block 13. In other words, for example, the encoder 90 may obtain the modified transform block 22 by selecting one or more of previously encoded samples among the one or more previously encoded samples neighbouring the predetermined block 13, and thus, varies the parameters, and a number or a count there of, associated with the ML reductive offset predictor 21 accordingly.
[0332] According to embodiments, the encoder 90 may be configured to support different block sizes for the predetermined block 13 and for obtaining the improved transform block 23 by varying a number of parameters defining the ML improvement predictor 30 and the parameters defining the ML improvement predictor 30, depending on a block size of the predetermined block 13. In other words, for example, the encoder 90 may obtain the improved transform block 23 by selecting a block size among the different sizes (e.g. and supported by the encoder 90) for the predetermined block 13, and thus, varies the parameters, and a number or a count there of, associated with the ML improvement predictor 30 accordingly.
[0333] According to embodiments, the encoder 90 may be configured to support different intra prediction modes for the predetermined block 13 signalled in the data stream 40 and and for obtaining the improved transform block 23 by varying a number of parameters defining the ML reductive offset predictor 21 and the parameters defining the ML improvement predictor 30, depending on an intra prediction mode of the predetermined block 13. In other words, for example, the encoder 90 may obtain the improved transform block 23 by selecting an intra prediction mode among the different intra prediction modes (e.g. signalled in the data stream 40 and thus, supported by the decoder) for the predetermined block 13, and thus, varies the filing version, rmparameters, and a number or a count there of, associated with the ML reductive offset predictor 21 accordingly. Further, the encoder 90 may be configured to support one or more groups of intra-prediction modes into which the different intra prediction modes are grouped for the predetermined block 13, wherein the manner in which the one or more intra-prediction groups are grouped out of the different intra-prediction modes may be signalled in the data stream 40 along with the different intra prediction modes. The encoder 90 may then configured to obtain the improved transform block 23 by varying a number of parameters defining the ML improvement predictor 30 and the parameters defining the ML improvement predictor 30, depending on a group of intra prediction modes of the predetermined block 13. In other words, for example, the encoder 90 may obtain the improved transform block 30 by selecting a group of intra prediction mode among the different groups of intra prediction modes (e.g. signalled in the data stream 40 and thus, supported by the encoder) for the predetermined block 13, and thus, varies the parameters, and a number or a count there of, associated with the ML improvement predictor 30 accordingly.
[0334] According to embodiments, the decoder 90 may be configured to obtain the improved transform block 23 by varying a number of parameters defining the ML improvement predictor 30 and the parameters defining the ML improvement predictor 30, depending on the one or more transform (e.g. transformation, or mapping) types of the analysis transform block 20 and / or the modified transform block 22. In other words, for example, the encoder 90 may obtain the improved transform block 23 by selecting one or more transform (e.g. transformation, or mapping) types of the analysis transform block 20 and / or the modified transform block 22, and thus, varies the parameters, and a number or a count there of, associated with the ML improvement predictor 30 accordingly.
[0335] According to embodiments, the decoder 90 may be configured to obtain the improved transform block 23 by varying a number of parameters defining the ML improvement predictor 30 and the parameters defining the ML improvement predictor 30, depending on the one or more previously encoded samples neighbouring the predetermined block 13. In other words, for example, the encoder 90 may obtain the improved transform block 23 by selecting one or more of previously encoded samples among the one or more previously encoded samples neighbouring the predetermined block 13, and thus, varies the parameters, and a number or a count there of, associated with the ML improvement predictor 30 accordingly.
[0336] With respect to the obtaining the modified transform 22, the encoder 90 operates as follows: the transform coefficients (e.g. the corrected transform coefficients, the reduced transform coefficients, the modified transform coefficients) of the modified transform block 22 are obtained based on the reductive offset block, and the sign of the transform coefficients of the analysis transform block, wherein the reductive offset block is obtained using the ML reductive
[0337] filing version, rmoffset predictor 21, which is implemented by a corresponding or associated neural network: That is, as shown in Fig. 8, the ML transform predictor 34 is associated or implemented a neural network 91.
[0338] In accordance with embodiments, the ML reductive offset predictor 21 may be implemented by a neural network 91 comprising an initial non-linear fully-connected layer and one or more succeeding layers. The neural network 91 may be configured to receive the analysis transform block 20, the predicted sample filling, the quantization parameter and the index of the selected intra-prediction mode as inputs of the initial layer, receive an output of the initial layer as an input of the one or more succeeding layers. By this measure, output of the NN 91 implementing the ML reductive offset predictor 21 provides the reductive offset block. The neural network 91 may, for instance, also be referred to as the reductive offset neural network 91 to further distinguish it from other neural networks described in this disclosure.
[0339] The attributes of the neural network 91 or, in different terms, the input(s) of the initial layer of the neural network 91, may comprise:
[0340] 1) the set of transform coefficients of the analysis transform block 20;
[0341] further, 2) the predicted sample filling of the predetermined block 13;
[0342] further, 3) the one or more previously encoded samples neighbouring the predetermined block 13. In other words, reference samples or already decoded samples available to the encoder in the picture buffer may be additionally taken as input by the neural network. A number (or a count) of the one or more previously decoded samples neighbouring the predetermined block which is input into the neural network may be vary from anywhere from one previously decoded sample to all previously decoded samples of the one or more previously decoded samples neighbouring the predetermined block 13;
[0343] further, 4) a quantization parameter of the predetermined block 13 such as the quantization step size based on which the encoder may quantize or scale the transform coefficients. The quantization parameter may be configured to adjust the granularity at which the quantizer, or quantization block, 26, quantizes (e.g. scales) the transform coefficients; and
[0344] further, 5) an index of the selected intra-prediction mode selected from a plurality of intraprediction modes (or different intra-prediction modes).
[0345] filing version, rmIt is noted that both the options 4) and 5) have been been indicated in Fig. 5 by way of being depicted as schematically contained within a block / module 24, wherein the block / module 24 is connected (using a solid line) to the quantizer 26, the modified transform block 22 and the improved transform block 23. The optional nature of the QP and the intra mode as inputs to the implementations of the ML predictors 21, 30 (e.g. NNs 91, 92) could be denoted by the use of dashed lines in connection with the blocks 21, 30, which is not shown in Fig. 7. However, the use of the solid black connecting the block / module 24 to the block 22 and the block 23 illustrates the QP and / or the intra prediction mode as possible inputs of the ML reductive offset predictor 21 (and hence, for example, of the NN 91) and the ML improvement predictor 30 (and hence, for example, of the NN 93) respectively.
[0346] In accordance with embodiments, the final layer of the one or more succeeding layers of the neural network 91 is a linear layer. It is to be understood that linearity of the final layer relates to the attributes, or inputs, of the neural network 91 implementing the ML reductive offset predictor 21 such as any of the set of transform coefficients of the analysis transform block 20, the predicted sample filling of the predetermined block 13, the one or more previously encoded samples neighbouring the predetermined block 13, the quantization parameter of the predetermined block 13, and the the index of the selected intra-prediction mode.
[0347] According to embodiments, transform (e.g. or transformation or mapping) of the synthesis transform block may be an orthogonal transform. In other words, the transform of the synthesis transform block relating, or bridging, from the sample domain prediction residual to the transform domain prediction residual may be an orthogonal transform. Additionally, In accordance with embodiments, the transform of the synthesis transform block may be at least one of, or any of, a DCT, a DST, a LFNST, a NSPT, and a linear transform. For instance, when speaking of the sample domain prediction residual, it may be understood as a set, or a collection, of samples representing the prediction residual formed by the subtractor / prediction residual signal former 16 subtracting from the original signal in terms of samples the predicted sample filling; and when speaking of the transform domain prediction residual (in the case wherein the transform of the analysis transform relates the sample domain prediction residual to a transform (or an analysis transform) domain prediction residual), it may be understood as a set of transform coefficients, a number or count of the set equalling a number or count of samples of the sample domain prediction residual, representing the transform domain prediction residual resulting from the analysis transform block 20.
[0348] filing version, rmAs another optional feature, the reductive offset NN 91 in the above description comprises a non-linear activation function, as an example a rectified linear unit (ReLu). It is to be noted that other forms which introduce any non-linearities may be used in combination with or without the ReLu.
[0349] For instance, it holds in general, according to embodiments, the activation functions in the layers of the reductive offset NN 91 may include non-linear functions such as leaky ReLu, tanh, binary step function, SELU, ELU, sigmoid / logistic and / or parametric ReLu. This means, for example, that a first grouping of the layers of the NN 91 may comprise non-linear activation functions different from a second grouping of the layers of the NN 91. As another optional feature, it also holds in general, that any of the layers of the NN 91 may comprise more than one non-linear activation function. Additionally or alternatively, as an instantiation, some of the layers, or some neurons among the layer, of the NN 91 may even comprise an identity function as the activation function.
[0350] According to embodiments, the encoder 90 may support one or more of a number of different transform types for the analysis transform block 20, wherein each different transform type for the analysis transform block 20 is related to a neural network 91jimplementing the ML reductive offset predictor 21 and may compute, or obtain, the reductive offset block by selecting and using a corresponding neural network 91jrelated to a transform type among the different transform types for the analysis transform block 20. In other words, the encoder 90 may support different neural networks 91jwherein the index / may range from anywhere including one to including a maximal number of different transform types supported by the encoder 90. The encoder 90 may perform a selection among the different neural networks, wherein the different neural networks are classified according to the different transform types for the analysis transform block 20. The different neural networks may be further classified according to a number (e.g. a count of) of parameters defining the NN 91jand values of the parameters of the NN 91jwhich the encoder 90 may be configured to vary.
[0351] According to embodiments, the encoder 90 may support one or more of a number of different block sizes for the predetermined block 13, wherein each block size is related to a neural network 91jimplementing the ML reductive offset predictor 21 and may compute, or obtain, the reductive offset block by selecting and using a corresponding neural network 91jrelated to a block size among the different block sizes of the predetermined block 13. In other words, the encoder 90 may support different neural networks 91jwherein the index / may range from anywhere including one to including a maximal number of different block sizes of the predetermined block 13 supported by the encoder 90. The encoder 90 may perform a selection among the different neural networks, wherein the different neural networks are classified
[0352] filing version, rmaccording to the different block sizes of the predetermined block 13. The different neural networks may be further classified according to a number (e.g. a count of) of parameters defining the NN 91jand values of the parameters of the NN 91jwhich the encoder 90 may be configured to vary.
[0353] Further, in the case of a block size equalling a mutually transposed block size among the different block sizes of the predetermined block 13, the decoder 90 may compute, or obtain, the reductive offset block by transposing the input of the initial layer and the output of the final layer of a corresponding neural network 91jrelated to the mutually transposed block size of the predetermined block 13, and using the parameters of the neural network 91jrelated to the mutually transposed block size of the predetermined block 13.
[0354] Further, in the case of the block size exceeding a predetermined block size value, the corresponding neural network 91jtakes, as an input, a sub-portion of the analysis transform block 20. In other words, the input of the NN 91jin such an instance would be a subset of the set of transform coefficients of the analysis transform block 20.
[0355] According to embodiments, the encoder 90 may support one or more of a number of different intra-prediction modes for the predetermined block 13 or one or more of a group of intraprediction modes into which the different intra-prediction modes are grouped, each of both of which is related to a neural network 91jimplementing the ML reductive offset predictor 21 and may compute, or obtain, the reductive offset block by selecting and using a corresponding neural network 91jrelated to an intra-prediction mode, or a group of intra-prediction modes, among the different intra-prediction modes, or the groups thereof, of the predetermined block 13. In other words, the encoder 90 may support different neural networks 91jwherein the index / may range from anywhere including one to including a maximal number of different intraprediction modes or a maximal number of different groups of intra-prediction modes of the predetermined block 13 supported by the encoder 90. The encoder 90 may perform a selection among the different neural networks, wherein the different neural networks are classified according to the different intra-prediction modes or groups thereof of the predetermined block 13. The different neural networks may be further classified according to a number (e.g. a count of) of parameters defining the NN 91jand values of the parameters of the NN 91jwhich the encoder 90 may be configured to vary.
[0356] In other words, it may be that, with respect to obtaining (or e.g. computing) the reductive offset block, one or more of the following may be true:
[0357] 1) the encoder 90 may manage and comprise a set of NNs 91j(and possibly associated parameters and a number or count thereof) for each block size or each of different
[0358] filing version, rmgroups of block sizes into which all supported block sizes are grouped, and the encoder 90, when obtaining (or e.g. computing), the reductive offset block, firstly may select the corresponding - the one fitting to the size of block 12 - set of NNs 91j(and possibly associated parameters and a number or count thereof) before acting and behaving as described above with respect to that set of NNs 91j(and possibly associated parameters and a number or count thereof). That is, for instance, the encoder 90 may select NNs 91jusing a predetermined criterion which is associated with the supported block sizes of the block 13.
[0359] a. In accordance with embodiments, the grouping of the blocks may group mutually transposed blocks, i.e. blocks and correspondingly transposed blocks comprising the same size, together. In other words, blocks of shape Wx H and H x W, may be grouped together, where W and H denote dimensions of the block, e.g. with W being the width of one block in samples and H being the height of this block in samples. Without loss of generality, in case of the block shape Wx H, the reductive offset NN 91ᵢ would process its inputs and outputs without additional operations whereas in case of the transposed block shape H x W, the inputs, namely the set of transform coefficients of the analysis transform block 20 and the predicted sample filling 18 (i.e. the prediction signal) would be transposed before application onto the NN, thereby allowing the reductive offset block obtained (e.g. computed) by the NN 91jfor the block shape W x H to be used, and subsequently, the output of the NN would be transposed back into the shape H x W thus obtaining (e.g. computing) the reductive offset block for the transposed block shape H x W.
[0360] b. In accordance with embodiments, if the block size selected by the encoder 90 is larger, or greater, than a predetermined block size value, the encoder 90 may take a sub-portion of the analysis transform block 20. For instance, this subportion of the analysis transform block 20 may correspond to the predetermined block size value. A sub-portion of the analysis transform block, for example, may correspond to a subset of the set of transform coefficients of, or representing, the analysis transform block 20.
[0361] ) the encoder 90 may use (and possibly select), for instance, the same set of NNs (and possibly associated parameters and a number or count thereof) irrespective of the prediction mode of block 13, but it could, naturally, be that the encoder 90 manages and comprises a set of NNs 91j(and possibly associated parameters and a number or count thereof) for each intra-prediction mode or each of different groups of intraprediction modes into which all supported intra-prediction mods are grouped, and the encoder 90, when obtaining (or e.g. computing) the reductive offset block, may firstly
[0362] filing version, rmselect the corresponding - the one fitting to the intra prediction mode of block 13 - set of NNs (and possibly associated parameters and a number or count thereof) before acting and behaving as described above with respect to that set of NNs (and possibly associated parameters and a number or count thereof) such as by use of the index of the selected intra-prediction mode for block 13 which index might be signaled in the data stream 40.
[0363] 3) the encoder 90 uses (and possibly selects), for instance, the same set of NNs (and possibly associated parameters and a number or count thereof) irrespective of the transform (or transformation) type of the analysis transform block 29, but it could, naturally, be that the encoder 90 manages and comprises a set of NNs 91j(and possibly associated parameters and a number or count thereof) for each transform type or each of different groups of transform types into which all supported transform types are grouped, and the encoder, when obtaining (or e.g. computing) the reductive offset block, may firstly select the corresponding - the one fitting to the transform type of the analysis transform block 20 - set of NNs (and possibly associated parameters and a number or count thereof) before acting and behaving as described above with respect to that set of NNs (and possibly associated parameters and a number or count thereof).
[0364] 4) Naturally, it may be that the encoder 90 obtains (or, e.g. computes) the reductive offset block only to a subset of the possible block characteristics a - d listed above, such as only to blocks in a certain range of sizes, the analysis transform is of one or more of certain transform types, and blocks of one or more intra-prediction modes. Moreover, inter-predicted blocks may or may not be subject to obtainment (or, e.g. computation) of the reductive offset block.
[0365] With respect to the above discussion, parameters of the NN 91jmay comprise a number (or a count) of hidden layers (i.e. the one or more succeeding layers), and weights, and a number or a count thereof, and biases, and a number or a count thereof.
[0366] With respect to the obtaining the improved transform block 23 based on the additive offset block, the encoder 90 operates as follows: the improved reconstructed transform coefficients of the improved transform block 23 are obtained based on the additive offset block, and the transform coefficients of the analysis transform block 20 and / or the modified transform block 22 (i.e. meaning the transform coefficients of the block 20 and / or the block 22 as reconstructed as part of the prediction loop 37), wherein the additive offset block is obtained using the ML improvement predictor 30, which is implemented by a corresponding or associated neural network: That is, as shown in Fig. 8, the ML improvement predictor 30 is associated or implemented a neural network 93.
[0367] filing version, rmIn accordance with embodiments, the ML improvement predictor 30 may be implemented by a neural network 93 comprising an initial non-linear fully-connected layer and one or more succeeding layers. The neural network 93 may be configured to receive the analysis transform block 20 and / or the modified transform block 22, the predicted sample filling, the quantization parameter and the index of the selected intra-prediction mode as inputs of the initial layer, receive an output of the initial layer as an input of the one or more succeeding layers. By this measure, output of the NN 93 implementing the ML improvement predictor 30 provides the additive offset block. The neural network 93 may, for instance, also be referred to as the improvement neural network 93 to further distinguish it from other neural networks described in this disclosure.
[0368] The attributes of the neural network 93 or, in different terms, the input(s) of the initial layer of the neural network 93, may comprise:
[0369] 1) the set of transform coefficients of the analysis transform block 20; and / or
[0370] 2) the set of transform coefficients of the modified transform block 22; and
[0371] further, 3) the predicted sample filling of the predetermined block 13;
[0372] further, 4) the one or more previously encoded samples neighbouring the predetermined block 13. In other words, reference samples or already decoded samples available to the encoder in the picture buffer may be additionally taken as input by the neural network. A number (or a count) of the one or more previously encoded samples neighbouring the predetermined block which is input into the neural network may be vary from anywhere from one previously encoded sample to all previously encoded samples of the one or more previously encoded samples neighbouring the predetermined block 13;
[0373] further, 5) a quantization parameter of the predetermined block 13 such as the quantization step size based on which the encoder may quantize or scale the transform coefficients. The quantization parameter may be configured to adjust the granularity at which the quantizer, or quantization block, 26, quantizes (e.g. scales) the transform coefficients; and
[0374] further, 6) an index of the selected intra-prediction mode selected from a plurality of intraprediction modes (or different intra-prediction modes).
[0375] filing version, rmBoth the option 5) and 6) have been indicated as contained in the block 24 which is shown connected to the improved transform block 23 in Fig. 7 to illustrate the possible inputs of the NN 93 implementing the ML improvement predictor 30.
[0376] In accordance with embodiments, the final layer of the one or more succeeding layers of the neural network 93 is a linear layer. It is to be understood that linearity of the final layer relates to the attributes, or inputs, of the neural network 93 implementing the ML improvement predictor 30 such as any of the set of transform coefficients of the analysis transform block 20, and / or the set of transform coefficients of the modified transform block 22, the predicted sample filling of the predetermined block 13, the one or more previously encoded samples neighbouring the predetermined block 13, the quantization parameter of the predetermined block 13, and the the index of the selected intra-prediction mode.
[0377] As another optional feature, the improvement NN 93 in the above description comprises a nonlinear activation function, as an example a rectified linear unit (ReLu). It is to be noted that other forms which introduce any non-linearities may be used in combination with or without the ReLu.
[0378] For instance, it holds in general, according to embodiments, the activation functions in the layers of the improvement NN 93 may include non-linear functions such as leaky ReLu, tanh, binary step function, SELU, ELU, sigmoid / logistic and / or parametric ReLu. This means, for example, that a first grouping of the layers of the NN 93 may comprise non-linear activation functions different from a second grouping of the layers of the NN 93. As another optional feature, it also holds in general, that any of the layers of the NN 93 may comprise more than one non-linear activation function. Additionally or alternatively, as an instantiation, some of the layers, or some neurons among the layer, of the NN 93 may even comprise an identity function as the activation function.
[0379] According to embodiments, the encoder 90 may support one or more of a number of different transform types for the analysis transform block 20, wherein each different transform type for the analysis transform block 20 is related to a neural network 93j implementing the ML improvement predictor 30 and may compute, or obtain, the additive offset block by selecting and using a corresponding neural network 93j related to a transform type among the different transform types for the analysis transform block 20. In other words, the encoder 90 may support different neural networks 93j wherein the index / may range from anywhere including one to including a maximal number of different transform types supported by the encoder 90. The encoder 90 may perform a selection among the different neural networks, wherein the different neural networks are classified according to the different transform types for the filing version, rmanalysis transform block 20. The different neural networks may be further classified according to a number (e.g. a count of) of parameters defining the NN 93ᵢ and values of the parameters of the NN 93ᵢ which the encoder 90 may be configured to vary.
[0380] According to embodiments, the encoder 90 may support one or more of a number of different block sizes for the predetermined block 13, wherein each block size is related to a neural network 93j implementing the ML improvement predictor 30 and may compute, or obtain, the additive offset block by selecting and using a corresponding neural network 93j related to a block size among the different block sizes of the predetermined block 13. In other words, the encoder 90 may support different neural networks 93j wherein the index / may range from anywhere including one to including a maximal number of different block sizes of the predetermined block 13 supported by the encoder 90. The encoder 90 may perform a selection among the different neural networks, wherein the different neural networks are classified according to the different block sizes of the predetermined block 13. The different neural networks may be further classified according to a number (e.g. a count of) of parameters defining the NN 93ᵢ and values of the parameters of the NN 93ᵢ which the encoder 90 may be configured to vary.
[0381] Further, in the case of a block size equalling a mutually transposed block size among the different block sizes of the predetermined block 13, the decoder 90 may compute, or obtain, the additive offset block by transposing the input of the initial layer and the output of the final layer of a corresponding neural network 93ᵢ related to the mutually transposed block size of the predetermined block 13, and using the parameters of the neural network 93ᵢ related to the mutually transposed block size of the predetermined block 13.
[0382] Further, in the case of the block size exceeding a predetermined block size value, the corresponding neural network 93ᵢ takes, as an input, a sub-portion of the analysis transform block 20. In other words, the input of the NN 93ᵢ in such an instance would be a subset of the set of transform coefficients of the analysis transform block 20.
[0383] According to embodiments, the encoder 90 may support one or more of a number of different intra-prediction modes for the predetermined block 13 or one or more of a group of intraprediction modes into which the different intra-prediction modes are grouped, each of both of which is related to a neural network 93ᵢ implementing the ML improvement predictor 30 and may compute, or obtain, the additive offset block by selecting and using a corresponding neural network 93ᵢ related to an intra-prediction mode, or a group of intra-prediction modes, among the different intra-prediction modes, or the groups thereof, of the predetermined block 13. In other words, the encoder 90 may support different neural networks 93ᵢ wherein the index i may range from anywhere including one to including a maximal number of different intra-prediction
[0384] filing version, rmmodes or a maximal number of different groups of intra-prediction modes of the predetermined block 13 supported by the encoder 90. The encoder 90 may perform a selection among the different neural networks, wherein the different neural networks are classified according to the different intra-prediction modes or groups thereof of the predetermined block 13. The different neural networks may be further classified according to a number (e.g. a count of) of parameters defining the NN 93ᵢ and values of the parameters of the NN 93ᵢ which the encoder 90 may be configured to vary.
[0385] In other words, it may be that, with respect to obtaining the additive offset block, one or more of the following may be true:
[0386] 1) the encoder 90 may manage and comprise a set of NNs 93ᵢ (and possibly associated parameters and a number or count thereof) for each block size or each of different groups of block sizes into which all supported block sizes are grouped, and the encoder 90, when computing, or obtaining, the additive offset block, firstly may select the corresponding - the one fitting to the size of block 12 - set of NNs 93ᵢ (and possibly associated parameters and a number or count thereof) before acting and behaving as described above with respect to that set of NNs 93ᵢ (and possibly associated parameters and a number or count thereof). That is, for instance, the encoder 90 may select NNs 93ᵢ using a predetermined criterion which is associated with the supported block sizes of the block 13.
[0387] a) In accordance with embodiments, the grouping of the blocks may group mutually transposed blocks, i.e. blocks and correspondingly transposed blocks comprising the same size, together. In other words, blocks of shape Wx H and H x W, may be grouped together, where W and H denote dimensions of the block, e.g. with W being the width of one block in samples and H being the height of this block in samples. Without loss of generality, in case of the block shape Wx H, the improvement NN 93ᵢ would process its inputs and outputs without additional operations whereas in case of the transposed block shape H x W, the inputs, namely the set of transform coefficients of the analysis transform block 20, and / or the set of transform coefficients of the modified transform block 22, and the predicted sample filling 18 (i.e. the prediction signal) would be transposed before application onto the NN, thereby allowing the additive offset block obtained by the NN 93ᵢ for the block shape W x H to be used, and subsequently, the output of the NN would be transposed back into the shape H x W thus obtaining the additive offset block for the transposed block shape H x W.
[0388] b) In accordance with embodiments, if the block size selected by the encoder 90 is larger, or greater, than a predetermined block size value, the encoder 90 may take a subportion of the analysis transform block 20 and / or of the modified transform block 22. For instance, this sub-portion of the analysis transform block 20 and / or of the modified filing version, rmtransform block 22 may correspond to the predetermined block size value. A subportion of the analysis transform block 20 and / or of the modified transform block 22, for example, may correspond to a subset of the set of transform coefficients of the analysis transform block 20 and / or a subset of the set of transform coefficients of the modified transform block 22.
[0389] 2) the encoder 90 may use (and possibly select), for instance, the same set of NNs (and possibly associated parameters and a number or count thereof) irrespective of the prediction mode of block 13, but it could, naturally, be that the encoder 90 manages and comprises a set of NNs 93ᵢ (and possibly associated parameters and a number or count thereof) for each intra-prediction mode or each of different groups of intra-prediction modes into which all supported intra-prediction mods are grouped, and the encoder 90, when computing, or obtaining, the additive offset block, may firstly select the corresponding - the one fitting to the intra prediction mode of block 13 - set of NNs (and possibly associated parameters and a number or count thereof) before acting and behaving as described above with respect to that set of NNs (and possibly associated parameters and a number or count thereof) such as by use of the index of the selected intra-prediction mode for block 13 which index might be signaled in the data stream 40.
[0390] 3) the encoder 90 uses (and possibly selects), for instance, the same set of NNs (and possibly associated parameters and a number or count thereof) irrespective of the transform (or transformation) type of the analysis transform block 20, but it could, naturally, be that the encoder 90 manages and comprises a set of NNs 93ᵢ (and possibly associated parameters and a number or count thereof) for each transform type or each of different groups of transform types into which all supported transform types are grouped, and the encoder, when computing the additive offset block, may firstly select the corresponding - the one fitting to the transform type of the analysis transform block 20 - set of NNs (and possibly associated parameters and a number or count thereof) before acting and behaving as described above with respect to that set of NNs (and possibly associated parameters and a number or count thereof).
[0391] 4) Naturally, it may be that the encoder 90 computes the additive offset block only to a subset of the possible block characteristics a - d listed above, such as only to blocks in a certain range of sizes, the analysis transform 20 is of one or more of certain transform types, and blocks of one or more intra-prediction modes. Moreover, inter-predicted blocks may or may not be subject to computation of the additive offset block.
[0392] With respect to the above discussion, parameters of the NN 93ᵢ may comprise a number (or a count) of hidden layers (i.e. the one or more succeeding layers), and weights, and a number or a count thereof, and biases, and a number or a count thereof.
[0393] filing version, rmFor instance, the encoder may encode the analysis transform block 20 and / or the modified transform block 22 of the prediction residual of the predetermined block 13 into the data stream 40 using entropy coding 31, as illustrated in Fig. 7.
[0394] It is also indicated in Fig. 7 that the prediction residual (i.e. the sample domain prediction residual) is exemplarily denoted by s - p; the transform coefficients of the analysis transform block 20 are denoted by u, in harmony with equation (1.1.1) further in the disclosure; modified, or e.g. corrected, transform coefficients representing the modified transform block 22 being subjected to the ML reductive offset predictor 21 (itself exemplarily denoted by g) are exemplarily denoted by v in harmony with equation (1.2.3); quantized transform coefficients obtained by the quantizer 26 are exemplarily denoted by Q(v, A) where in the QP is exemplarily denoted by Δ; dequantized transform coefficients are exemplarily denoted by v̂ := DQ∘Q(v, Δ); transform coefficients (e.g. corrected or improved transform coefficients) of the improved transform block 23 being subjected to the ML improvement predictor 30 (itself being exemplarily denoted by f) are exemplarily denoted by v̂ + f, in similarity to terms in parenthesis of equation (1.2.1) described further in the disclosure; the transform coefficients of, or representing, the synthesis transform block are exemplarily denoted by B(v̂ + f) in similarity to the first term of equation (1.2.1) described further in the disclosure; the predicted sample filling is exemplarily denoted by p and the reconstructed signal, or reconstructed transform coefficients, are denoted by ŝ := B(v̂ + f) + p in similarity to equation (1.2.1) described further in the disclosure.
[0395] Fig. 9 shows an implementation of the encoder 90 which combines already described embodiments according to the first aspect and just described embodiments according to the second aspect. In particular, the encoder 90 of Fig. 9, as compared to Fig. 8, additionally comprises the ML transform predictor 38, denoted by α, associated with the analysis transform block 20; the ML transform predictor 34, denoted by β, associated with the synthesis transform block 32; and the ML prediction predictor 36, denoted by p, associated with the prediction module / block 14. As already depicted in the encoder of Fig. 5, the functionality of the ML transform predictor 38 of transforming 57, or performing a transform transition 59 of, the analysis transform block 20 is depicted with dotted lines; and further, as already depicted in the decoder 50 of Fig. 4, the functionality of the ML transform predictor 34 of transforming 35, or performing a transform transition 39 of, the synthesis transform block 32 is depicted with dotted lines.
[0396] Since the additional three ML predictors 34, 46, 38 have already been extensively described in the disclosure, the application refrains from a repetition of such details, which may be configured to be applicable on to the encoder 90 of Fig. 9. However, it is emphasizes that Fig.
[0397] filing version, rm9 illustrates an embodiment comprising a maximal combination of the features of the first aspect and the second aspect. That is, all the ML predictors are simultaneously present in the embodiment according to Fig. 9. However, this embodiment may not restrict less maximal combination of features (e.g. the ML predictors) comprised in other embodiments, wherein some ML predictors which are present in Fig. 9 may be altogether absent in the other embodiments. For instance, in an embodiment, some of the ML predictors 34, 36, 38, 21, 30 may be present while some other of the ML predictors 34, 36, 38, 21, 30 may be absent. So far different embodiments relating to the first aspect and the second aspect of the invention by way of apparatuses such as an encoder and a decoder have been described in Figs. 1 to 9. Henceforth, embodiments relating to a third aspect are presented as depicted in Figs. 10 to 13. These embodiments pertain to implementations of a method of training a machine learning predictor, the likes of which have been earlier.
[0398] It is also indicated in Fig. 9 that the prediction residual (i.e. the sample domain prediction residual) is exemplarily denoted by s - p; the transform coefficients of, or representing, the analysis transform block 20 are denoted by u and defined as u: = α(s − p) being subjected to the ML transform predictor 38 (itself exemplarily denoted as α), in similarity to equation (1.4.1) described further in the disclosure; modified, or e.g. corrected, transform coefficients representing the modified transform block 22 being subjected to the ML reductive offset predictor 21 (itself exemplarily denoted by g) are exemplarily denoted by v and defined as v: = u - sgn(u). abs(g),in similarity to equation (1.2.3); quantized transform coefficients obtained by the quantizer 26 are exemplarily denoted by Q(v, A) where in the QP is exemplarily denoted by Δ; dequantized transform coefficients are exemplarily denoted by DQoQ(v, A); transform coefficients (e.g. corrected or improved transform coefficients) of the improved transform block 23 being subjected to the ML improvement predictor 30 (itself being exemplarily denoted by f) are exemplarily denoted by v̂ + f, in similarity to the term in parenthesis of equation (1.2.1) described further in the disclosure; the transform coefficients of the synthesis transform block 32 being subjected to the ML transform predictor 34 (itself being exemplarily denoted by β) are exemplarily denoted by β(v̂ + f), in similarity to equation (1.4.5) described further in the disclosure; the predicted sample filling is exemplarily denoted by p and the reconstructed signal, or reconstructed transform coefficients, are denoted by ŝ := β(v̂ + f) + p in similarity to equations (1.4.3) and (1.2.1) described further in the disclosure.
[0399] It is noted that options of the QP and the intra prediction mode have been indicated in Fig. 9 by way of being depicted as schematically contained within a block / module 24, wherein the block / module 24 is connected (using a solid line) to the quantizer 26, the modified transform block and the improved transform block 23. In addition to this, the optional nature of the QP and the intra mode as inputs to the implementations of the ML predictors 34, 38 (e.g. NNs 51,
[0400] filing version, rm41) is denoted by the use of dashed lines in connection with the blocks 34, 38. Hence, these options, both individually or in combination, constitute distinct embodiments represented by Fig. 9.
[0401] Figure 10 exemplarily shows a schematic block diagram of an implementation of a method of training a machine learning, ML, predictor 100 which is configured to be used for block based coding, i.e. block based encoding and / or block based decoding, of a predetermined block of a picture, such as the predetermined block 13 of the picture 12 in previously described figures.
[0402] The method comprises providing an initial set of parameters 200 of the ML predictor 100. This is depicted by way of blocks of the initial parameters 200 and of the ML predictor 100 being connected with each other.
[0403] The method further comprises subjecting the initial set of parameters 200 to a training stage 300 for obtaining an updated set of parameters 400 which are based on a dequantization function 500 and / or a bitrate cost model 600.
[0404] As depicted in Fig. 10, the training stage 300 comprises a plurality of training epochs 3040,1,2..., wherein each training epoch 304# is configured to update the corresponding set of parameters as inputs of a subsequent training epoch.
[0405] The method further comprises, as an epoch 304# of the training stage 300, computing a training error 328# according to a training objective 308#. The training objective 308# may depend on the output of the ML predictor 100, and / or the dequantization function 500, and / or the bitrate cost model 600. In the embodiment exemplarily presented in Fig. 10, the training objectives 3080,1,2depend on all three, i.e. the output of the ML predictor 100, and the dequantization function 500, and the bitrate cost model 600.
[0406] The method further comprises, as the epoch 304# of the training stage 300, updating the initial set of parameters 200 so as to obtain a subsequent set of parameters 348# related to the corresponding training error 328# being lower in values while leaving parameters of the bitrate cost model 600 unchanged. The updating of the initial set of parameters 200 may be performed by obtaining a reduced training error by, or as a result of using (e.g. based on) a training optimization scheme such as gradient descent, stochastic gradient descent, root mean square propagation, or ADAM optimization algorithm.
[0407] The method further comprises outputting the subsequent set of parameters 348# as inputs for a subsequent epoch 304#+i of the training stage 300.
[0408] filing version, rmBefore the specific instantiation of the method according to Fig. 10 is described in further detail, details of the dequantization function 500 and the bitrate cost model 600, both of which may be provided as inputs of the training objective 308#.
[0409] The dequantization function 500 is configured to, for each transform coefficient of a set of transform coefficients of a transform block of a prediction residual for the predetermined block, each transform coefficient having an original value, set a dequantized value of the transform coefficient to the original value of the transform coefficient if a quantized value of the transform coefficient is non-zero, and set a dequantized value of the transform coefficient to zero if a quantized value of the transform coefficient is zero, wherein the quantized value of the transform coefficient is derived by a quantizer based on a quantization parameter of the predetermined block. By this measure, the dequantization function 500 may retain original values of the transform coefficients during the training stage of the ML predictor 100 and thus, may provide a differentiable (e.g. smooth) approximation of the quantization.
[0410] The bitrate cost model 600 (e.g. a bitrate cost distribution) is configured to, for each transform coefficient of a set of transform coefficients of a transform block of a prediction residual for the predetermined block, obtain a bitrate cost associated with the transform block (i.e. bitrate cost associated with coding the transform block into / from the data stream) using a density function (e.g. a probability density function). The density function is configured to output a probability value associated with a quantization index of the transform coefficient (e.g. the quantization index obtained from a quantization of the transform coefficient). The attributes of the density function comprise a scan index of the transform coefficient, a last scan index associated with the transform block, which is indicative of a last non-zero-quantized transform coefficient; and a quantization parameter of the predetermined block. For instance, the last scan index associated with the transform block may be derived using a rate-distortion optimization quantization. As will be described further in the disclosure, the density function may be implemented using a neural network.
[0411] For instance, the density function may be obtained by computing a bitrate error function based on a difference between a cross entropy of the density function associated with the transform block, and a true bitrate cost depending on a code length of the last scan index. For example, the density function may be obtained from fitting the cross entropy to the true bitrate of transform coefficients using a suitable set of training data. This means that the density function may be obtained by computing the bitrate error function, which is itself computed on a basis of the difference between the cross entropy of the density function and the true bitrate cost. For example, here the bitrate error function may not be restricted or limited to a subtraction between the cross entropy and the true bitrate cost. For instance, the bitrate error function may be represented as a fraction whose numerator corresponds to the difference between the cross
[0412] filing version, rmentropy of the density function associated with the transform block and the true bitrate cost depending on a code length of the last scan index while the denominator corresponds to the true bitrate cost, thereby providing an instantiation wherein the bitrate error function may be given by the difference scaled by the true bitrate cost. Further, for instance, additionally, or alternatively, the bitrate error function may be represented as a fraction whose numerator corresponds to the difference and the denominator corresponds to the sum of the true bitrate cost and a positive integer, such as 1.
[0413] The attributes (e.g. inputs; e.g. arguments) of the density function may comprise one or more of the non-zero-quantized transform coefficients (i.e. values of the non-zero-quantized transform coefficients). In other words, the density function may be configured to take as input one or more of the non-zero-quantized transform coefficients (i.e. values of the non-zero-quantized transform coefficients)
[0414] For example, the attributes (e.g. inputs; e.g. arguments) of the density function may additionally comprise a number of non-zero-quantized transform coefficients. For example, a number meaning a count, or an overall enumerative measure of non-zero-quantized transform coefficients. In other words, a parameter indicating a cardinality of a set of the non-zero-quantized transform coefficients.
[0415] For example, in accordance with embodiments, each group of a plurality of quantization parameters associated with the predetermined block into which the plurality of quantization parameters is grouped may be related to a density function outputting a corresponding probability value depending on the corresponding group of quantization parameters. In other words, different density functions may be configured (e.g. obtained; e.g. trained) for different groups, or classes, of quantization parameters formed out of a set of quantization parameters for the predetermined block 13. For example, in situations with a limited number of training samples, different density functions for the different groups, or classes, of quantization parameters may be beneficial. For example, a density model for quantized transform block, with last scan index {0, 1,...,7} may be trained and be different to one trained for quantized transform blocks with last scan index {8,..., 15},
[0416] For example, in accordance with embodiments, each group of a plurality of scan indices associated with the transform block into which the transform block is grouped is related to a density function outputting a corresponding probability value depending on the corresponding group of scan indices. In other words, different density functions may be configured (e.g. obtained; e.g. trained) for different groups, or classes, of scan indices formed out of a set of scan indices for the transform block. For example, in situations with a limited number of training samples, different density functions for the different groups, or classes, of scan indices may be beneficial.
[0417] filing version, rmFor example, in accordance with embodiments, each group of a plurality of last scan indices associated with the transform block into which the transform block is grouped is related to a density function outputting a corresponding probability value depending on the corresponding group of last scan indices. In other words, different density functions may be configured (e.g. obtained; e.g. trained) for different groups, or classes, of last scan indices formed out of a set of last scan indices for the transform block. For example, in situations with a limited number of training samples, different density functions for the different groups, or classes, of last scan indices may be beneficial.
[0418] Describing in generality, the density function, as a parameterized probability density function, may be parameterized in terms of:
[0419] 1) scan indices of the transform coefficients of the transform block (i.e. quantized transform coefficients of the quantized transform block), or one or more groups of scan indices of transform coefficients into which the scan indices of the transform coefficients of the transform block is grouped into;
[0420] 2) last scan indices of the transform coefficients of the transform block (i.e. quantized transform coefficients of the quantized transform block), or one or more groups of last scan indices of transform coefficients into which the last scan indices of the transform coefficients of the transform block is grouped into;
[0421] 3) one or more quantization parameters of the predetermined block, one or more groups of quantization parameters into which the one or more quantization parameters may be grouped into;
[0422] 4) in addition or as an alternative to the aforementioned, one or more parameters derivable or obtainable from the quantized transform block or properties / characteristics thereof such as a total number of non-zero-quantized transform coefficients.
[0423] 5) Naturally, different sizes of the predetermined block (or equivalently the quantized transform block)
[0424] a. In accordance with embodiments, if the block size selected by the density function is larger, or greater, than a predetermined block size value, the density function may take a sub-portion of the quantized transform block. For instance, this sub-portion of the quantized transform block may correspond to the predetermined block size value. That is, the sub-portion may correspond to a subset of the set of quantized transform coefficients, wherein the set of transform coefficients represent the quantized transform block.
[0425] In accordance with embodiments, for each transform coefficient of the set of transform coefficients of the transform block of a prediction residual for the predetermined block, the density function may be implemented by a neural network.
[0426] filing version, rmThe neural network may comprise an initial non-linear fully connected layer and one or more succeeding layers. Further, the neural network may be configured to receive the transform coefficient, the scan index of the transform coefficient, the last scan index associated with the transform block and the quantization parameter of the predetermined as inputs of the initial layer, receive an output of the initial layer as an input of the one or more succeeding layers, and provide an output of a final layer among the one or more succeeding layers to obtain the probability value.
[0427] The application now reverts to description of the training stage 300 of the ML predictor 100 schematically depicted in Fig. 10. In the initial (or zeroth) epoch 3040of the training stage 300, the inputs of the initial (or zeroth) training objective 3080comprise: the initial set of parameters 200, the output of the ML predictor 100 which takes the initial set of parameters 200 as inputs, the dequantization function 500 and the bitrate cost model 600. The initial (or zeroth) training error 328o is computed according to the initial (or zeroth) training objective 3080and the initial set of parameters 200 are updated to obtain the initial (or zeroth) subsequent set of parameters 3480. The initial (or zeroth) subsequent set of parameters 348o may correspond to its training error being lower in value than the initial (or zeroth) training error 3280. In the course of updating of the initial parameters 200 to the initial (or zeroth) subsequent set of parameters 3480, the parameters of the bitrate cost model 600 may be left unchanged. The obtained initial (or zeroth) subsequent set of parameters 3480are then outputted (or provided) as inputs for a first subsequent epoch 304i of the training stage. This is depicted in Fig. 10 by way of a line connecting the initial (or zeroth) subsequent set of parameters 3480to the first training objective 308i of the first subsequent epoch 304i of the training stage 300.
[0428] In the first subsequent epoch 304i of the training stage 300, the inputs of the first training objective 308i depend on (or e.g. equivalently, comprise): the initial (or zeroth) subsequent set of parameters 348o, the output of the ML predictor 100 which takes the initial (or zeroth) subsequent set of parameters 3480as inputs, the dequantization function 500 and the bitrate cost model 600. The first training error 328i is computed according to the first training objective 308i and the initial (or zeroth) subsequent set of parameters 348o are updated to obtain the first subsequent set of parameters 348i. The first subsequent set of parameters 348i may correspond to its training error being lower in value than the first training error 328i. In the course of updating of the initial (or zeroth) subsequent set of parameters 3480to the first subsequent set of parameters 348i, the parameters of the bitrate cost model 600 may again be left unchanged. The obtained first subsequent set of parameters 348i are then outputted (or provided) as inputs for a second subsequent epoch 3042of the training stage. This is depicted in Fig. 10 by way of a line connecting the first subsequent set of parameters 348i to the second training objective 3082of the second subsequent epoch 3042of the training stage 300.
[0429] filing version, rmThe ellipsis next to the second subsequent epoch 3042 of the training stage 300 indicates that the training of the ML predictor may continue in an epoch wise manner as already described for the first epoch 304i. The training of the ML predictor 100 may eventually stop, or halt, according to a convergence criterion associated with the training error in a final epoch of the training stage. The obtained subsequent parameters of the final epoch may then be used to obtain the updated parameters 400 of the ML predictor 100. In other words, the ML predictor 100 may have the updated parameters 400 as its trained parameters (or optimal parameters). For instance, the method of training the ML predictor 100 may relate to the ML predictor 100 being a machine learning transform predictor, wherein the machine learning transform predictor may be configured to take, as input, a transform block of a prediction residual for the predetermined block, and / or a predicted sample filling of the predetermined block, and provide, as output, a sample domain prediction residual.
[0430] For instance, the method training the ML predictor 100 may relate to the ML predictor 100 being a machine learning transform predictor, wherein the machine learning transform predictor may be configured to take, as input, a sample domain prediction residual for the predetermined block and / or a predicted sample filling of the predetermined block, and to provide, as output, a transform block representing a transform domain prediction residual. For example, the method of training the ML predictor 100 may relate to the ML predictor 100 being the machine learning transform predictor, the method may further comprise: subjecting the transform block to a parametrization based on a skew symmetric matrix and the matrix exponentiation operation for obtaining orthogonality of the transform block.
[0431] For example, the method of training the ML predictor 100 may relate to the ML predictor 100 being the machine learning transform predictor, the method may further comprise initializing the transform block with an initial transform, wherein the initial transform is a DCT or DST, or a LFNST transform, or a NSPT transform or a linear transform, or an orthogonal transform or an identity transform. It holds in general that the transform block may be initialized with the initial transform corresponding to a family, or a collection, of any, or all, transforms configured to map, or bridge, or relate from a spatial (or sample) domain to a spectral (or frequency) domain. For example, with the aim of providing example, such a family, or collection, may include family of DCT transforms, family of DCT transforms DST, family of discrete Fourier transforms, LFNST, NPST, or family of data-trained linear transform, wherein the family of data-trained linear transforms with relatively, or substantially, good energy compaction capability and is suitable for compression, thereby providing a reasonable initial transform value.
[0432] filing version, rmIn addition, or as an alternative, to the above, the method of training the ML predictor 100 may relate to the ML predictor 100 being a machine learning prediction predictor may be configured to take, as input, values of one or more previously coded samples (e.g. previously encoded samples; e.g. previously decoded samples) neighbouring the predetermined block, and provide, as output, a predicted sample filling of the predetermined block.
[0433] For example, the method of training the ML predictor 100 may relate to the ML predictor 100 being the machine learning prediction predictor, the method may further comprise initializing the predicted sample filling with an initial predicted sample filling using a PLANAR intraprediction mode. However, this choice of intra-prediction mode may not merely be restricted to the planar mode but may also include other modes known to the person skilled in the art.
[0434] In addition, or as an alternative, to the above, the method of training the ML predictor 100 may relate to the ML predictor 100 being a machine learning reductive offset predictor may be configured to take, as input, an analysis transform block 20 of a prediction residual for the predetermined block and a predicted sample filling of the predetermined block, and provide, as output, a reductive offset block, wherein the reductive offset block is configured to correct each transform coefficient of the analysis transform block 20 based on corresponding absolute value of the reductive offset block and the sign of the transform coefficient for obtaining a corresponding corrected transform coefficient of a modified transform block lower in magnitude.
[0435] In addition, or as an alternative, to the above, the method of training the ML predictor 100 may relate to the ML predictor 100 being machine learning improvement predictor may be configured to take, as input, an analysis transform block 20 of a prediction residual for the predetermined block and a predicted sample filling of the predetermined block, and provide, as output, an additive offset block, wherein the additive offset block is configured to correct each transform coefficient of the transform block based on a corresponding value of the additive offset block for obtaining a corresponding improved transform coefficient of an improved transform block greater in magnitude.
[0436] For example, the ML predictor 100 may be any of: the ML transform predictor 34, the ML transform predictor 38, the ML prediction predictor 36, the ML reductive offset predictor 21 and the ML improvement predictor 30.
[0437] So far the method of training a single ML predictor has been described, the present application now details embodiments of the method for training more than one ML predictor. Further since details of training have already been described for the single ML predictor, such details are not
[0438] filing version, rmrepeated when describing Figs. 11 to 13. Instead, emphasis is placed on features of the method of training multiple ML predictors distinct from that of training the single ML predictor. Figure 11 exemplarily shows a schematic diagram relating to the method comprising training • a ML improvement predictor 102, denoted by f earlier and further in the disclosure, (e.g.
[0439] as a first ML predictor); and
[0440] • a ML reductive offset predictor 104, denoted by g earlier and further in the disclosure, (e.g. as a second ML predictor)
[0441] together so as to have a mutually shared training stage 302.
[0442] It should be noted that training objectives involved in the mutually shared training stage of the ML improvement predictor 102 and the ML reductive offset predictor 104 do not depend on either the bitrate cost model 600 or the dequantization function 500, forming a distinct embodiment.
[0443] The mutually shared training stage 302 is initialized with an initial (or zeroth) epoch 3040in which the ML improvement predictor 102 (along with its initial parameters) provides an input to the initial training objective 3120while the ML reductive offset predictor 104 provides no input to the initial training objective 3120. Thus, the training objective 312o depends on the additive offset block (e.g. f) but not on the reductive offset block (e.g. g). The initial set of subsequent parameters 3520are outputted as inputs to the first training objective 312i of the first subsequent epoch 304i. In the course of updating the initial parameters of the ML improvement predictor 102 to the initial set of subsequent parameters 3520, the parameters of the ML reductive offset predictor 104 are left unchanged.
[0444] The mutually shared training stage 302 proceeds further with a first subsequent epoch 304i in which the ML improvement predictor 102 provides an input to the first training objective 312i while the ML reductive offset predictor 104 provides still no input to the first training objective 312i. Thus, the training objective 312i still depends on the additive offset block (e.g. f) but not on the reductive offset block (e.g. g). The first set of subsequent parameters 352i are outputted as inputs to the second training objective 3122of the second subsequent epoch 3042and as inputs to the third training objective 3123of the third subsequent epoch 3043. In the course of updating the initial set of subsequent parameters 3520of the ML improvement predictor 102 to the first set of subsequent parameters 352i, the parameters of the ML reductive offset predictor 104 are left unchanged.
[0445] The mutually shared training stage 302 proceeds further with a second subsequent epoch 3042in which both the ML improvement predictor 102 and the ML reductive offset predictor 104 provide an input to the second training objective 3122, wherein the ML reductive offset predictor 104 provides its initial parameters 204 as input. Thus, the training objective 3122depends on
[0446] filing version, rmboth the additive offset block (e.g. f) and the reductive offset block (e.g. g). The second set of subsequent parameters 3522are outputted as inputs to the fourth training objective 3124 of the fourth subsequent epoch 3044. In the course of updating the initial set of parameters 204 of the ML reductive offset predictor 104 to the second set of subsequent parameters 3522, the parameters of the ML improvement predictor 102 are left unchanged.
[0447] The mutually shared training stage 302 proceeds further with a third subsequent epoch 304a in which the ML improvement predictor 102 provides an input to the third training objective 312a while the ML reductive offset predictor 104 provides no input to the third training objective 312a, wherein the ML improvement predictor 102 provides the first set of subsequent parameters 352i as input. Thus, the third training objective 312a depends on the additive offset block (e.g. f) but not on the reductive offset block (e.g. g). The third set of subsequent parameters 352a are outputted as inputs to the fourth training objective 3124of the fourth subsequent epoch 3044. In the course of updating the first set of subsequent parameters 352i of the ML improvement predictor 102 to the third set of subsequent parameters 3523, the parameters of the ML reductive offset predictor 104 are left unchanged.
[0448] The mutually shared training stage 302 proceeds further with a fourth subsequent epoch 3044in which both the ML improvement predictor 102 and the ML reductive offset predictor 104 provide an input to the fourth training objective 3124, wherein the ML reductive offset predictor 104 provides its second set of subsequent parameters 3522as input. Thus, the training objective 3124depends on both the additive offset block (e.g. f) and the reductive offset block (e.g. g). In the course of updating the second set of subsequent parameters 3522of the ML reductive offset predictor 104 to the fourth set of subsequent parameters 3524, the parameters of the ML improvement predictor 102 are left unchanged.
[0449] Thus, in accordance with embodiments, as part of training, the training objective may switch (e.g. alternate) between:
[0450] • a training objective 312i, 312a depending on the additive offset block (e.g. f) so as to (e.g. only) update parameters of the machine learning improvement predictor 102 in an epoch 304i, 304a of the mutually shared training stage 302 while leaving parameters of the machine learning reductive offset predictor 104 unchanged, and
[0451] • a training objective 3122, 3124depending on both the additive offset block (e.g. f ) and the reductive offset block (e.g. g) so as to (e.g. only) update the parameters of the machine learning reductive offset predictor 104 in a subsequent epoch 3042, 3044of the mutually shared training stage 302 while leaving parameters of the machine learning improvement predictor 102 unchanged.
[0452] At the culmination of the mutually shared training stage 302 which may be reached or arrived at in accordance to a convergence criterion (e.g. of the training error), obtained subsequent filing version, rmparameters of such a final epoch may then be used to obtain the updated parameters 402 of the ML improvement predictor 102 and the updated parameters 404 of the ML reductive offset predictor 104. In other words, the ML predictor 102 may have the updated parameters 402 as its trained parameters (or optimal parameters) and the ML predictor 104 may have the updated parameters 404 as its trained parameters (or optimal parameters).
[0453] Figure 12 exemplarily shows a schematic diagram relating to the method comprising training • a ML transform predictor 106 for block based encoding (i.e. for instance, subjecting an analysis transform block in an encoder), denoted by a earlier and further in the disclosure, (e.g. as a first ML predictor);
[0454] • a ML transform predictor 108 for block based decoding (i.e. for instance, subjecting a synthesis transform block in a decoder), denoted by / 3 earlier and further in the disclosure, (e.g. as a second ML predictor); and
[0455] • a ML prediction predictor 110, denoted by p earlier and further in the disclosure, (e.g.
[0456] as a third ML predictor);
[0457] together so as to have a mutually shared training stage 302.
[0458] It should be noted that in this embodiment some training objectives involved in the mutually shared training stage 302 do not depend on the bitrate cost model 600 and the dequantization function 500 while some other training objectives involved in the mutually shared training stage 302 do depend on both the bitrate cost model 600 and the dequantization function 500.
[0459] The mutually shared training stage 302 is initialized with an initial (or zeroth) epoch 3040in which all of the ML transform predictor for block based encoding 106 (along with its initial parameters 206), the ML transform predictor for block based decoding 108 (along with its initial parameters 208) and the ML prediction predictor 110 (along with its initial parameters 210) provide an input to the initial training objective 3140. Thus, the training objective 3140depends on the ML predictors 106, 108, 110 (i.e. the training objective 3140depends on a, / 3, p) as well as on the dequantization function 500 and the bitrate cost model 600. An initial set of subsequent parameters 3540are obtained and outputted as inputs to the first training objective 314i of the first subsequent epoch 304i. In the course of updating the initial parameters of the ML predictors 106, 108, 110 to the initial set of subsequent parameters 3540, the parameters of the bitrate cost model may be updated as per the initial set of subsequent parameters 3540or alternatively, may be left unchanged.
[0460] The mutually shared training stage 302 proceeds further with a first subsequent epoch 304i in which all of the ML predictors 106, 108, 110 again provide an input to the first training objective 314i, wherein the initial set of subsequent parameters 3540are taken as the inputs. Thus, the training objective 314i still depends on the ML predictors 106, 108, 110 (i.e. the training
[0461] filing version, rmobjective 314i depends on a, / 3, p) as well as on the dequantization function 500 and the bitrate cost model 600. A first set of subsequent parameters 354i are obtained and outputted as inputs to the second training objective 3142of the second subsequent epoch 3042. In the course of updating the initial parameters of the ML predictors 106, 108, 110 to the first set of subsequent parameters 354i, the parameters of the bitrate cost model may be updated as per the first set of subsequent parameters 354i or alternatively, may be left unchanged.
[0462] The mutually shared training stage 302 proceeds further with a second subsequent epoch 3042in which the ML predictors 108, 110 provide an input to the second training objective 3142while the ML predictor 106 does not provide an input to the second training objective 3142, wherein the first set of subsequent parameters 354i are taken as inputs. Thus, the training objective 3142depends on both the ML predictors 108, 110 (i.e. the training objective 3142depends on / 3, p) but not on the ML predictor 106, or the dequantization function 500 or the bitrate cost model 600. A second set of subsequent parameters 3542are obtained and outputted as inputs to the third training objective 3143of the third subsequent epoch 3043. In the course of updating the first set of subsequent parameters 354i of the ML predictors 108, 110 to the second set of subsequent parameters 3542, the parameters of the ML predictor 106 are left unchanged. Additionally, the parameters of the bitrate cost model 600 may also be left unchanged.
[0463] The mutually shared training stage 302 proceeds further with a third subsequent epoch 3043in which all of the ML predictors 106, 108, 110 again provide an input to the first training objective 314i, wherein the second set of subsequent parameters 3540are taken as the inputs of the ML predictors 108, 110 and the first set of subsequent parameters 354i are taken as the inputs of the ML predictor 106 (e.g. or also possibly of the bitrate cost model 600). Thus, the third training objective 3143depends on the ML predictors 106, 108, 110 (i.e. the training objective 3143depends on a, / 3, p) as well as on the dequantization function 500 and the bitrate cost model 600. A third set of subsequent parameters 3523are obtained and outputted as inputs to the fourth training objective 3144of the fourth subsequent epoch 3044. In the course of updating the first set of subsequent parameters 354i of the ML predictor 106 to the third set of subsequent parameters 3543and the second set of subsequent parameters 3542of the ML predictors 108, 110 to the third set of subsequent parameters 3543, the parameters of the bitrate cost model may be updated as per the third set of subsequent parameters 3543or alternatively, may be left unchanged.
[0464] The mutually shared training stage 302 proceeds further with a fourth subsequent epoch 3044in which the ML predictors 108, 110 provide an input to the fourth training objective 3144while the ML predictor 106 does not provide an input to the fourth training objective 3144, wherein the third set of subsequent parameters 354i are taken as inputs. Thus, the fourth training objective 3144depends on both the ML predictors 108, 110 (i.e. the training objective 3144
[0465] filing version, rmdepends on / 3, p) but not on the ML predictor 106, or the dequantization function 500 or the bitrate cost model 600. A fourth set of subsequent parameters 3544are obtained and may be outputted as inputs to a next training objective of a subsequent epoch. In the course of updating the third set of subsequent parameters 3543of the ML predictors 108, 110 to the fourth set of subsequent parameters 3542, the parameters of the ML predictor 106 are left unchanged. Additionally, the parameters of the bitrate cost model 600 may also be left unchanged.
[0466] Thus, in accordance with embodiments, as part of training, the training objective may switch (e.g. alternates) between:
[0467] • a training objective 314i, 3143depending on
[0468] o the machine learning transform predictor 106 (e.g. a) for block based encoding, o the machine learning transform predictor 108 (e.g. P) for block based decoding, o the machine learning prediction predictor 110 (e.g. p),
[0469] o the dequantization function 500 and / or the bitrate cost model 600
[0470] so as to update parameters of the corresponding machine learning predictors 106, 108, 110 in an epoch 304i, 3043of the mutually shared training stage 302; and
[0471] • a training objective 3142, 3144depending on
[0472] o the machine learning transform predictor 108 (e.g. ) for block based decoding, o the machine learning prediction predictor 110 (e.g. p),
[0473] so as to update parameters of the corresponding machine learning predictors 108, 110 in a subsequent epoch 3042, 3044of the mutually shared training stage 302 while leaving the parameters of the machine learning transform predictor 106 (e.g. a) for block based encoding unchanged.
[0474] At the culmination of the mutually shared training stage 302 which may be reached or arrived at in accordance to a convergence criterion (e.g. of the training error), obtained subsequent parameters of such a final epoch may then be used to obtain the updated parameters 406 of the ML predictor 106, the updated parameters 408 of the ML predictor 108 and the updated parameters 410 of the ML predictor 110. In other words, the ML predictor 106 may have the updated parameters 406 as its trained parameters (or optimal parameters), the ML predictor 108 may have the updated parameters 408 as its trained parameters (or optimal parameters) and the ML predictor 110 may have the updated parameters 410 as its trained parameters (or optimal parameters).
[0475] filing version, rmFigure 13 exemplarily shows another schematic diagram relating to the method comprising training
[0476] • a ML transform predictor 106 for block based encoding (i.e. for instance, subjecting an analysis transform block in an encoder), denoted by a earlier and further in the disclosure, (e.g. as a first ML predictor); and
[0477] • a ML transform predictor 108 for block based decoding (i.e. for instance, subjecting a synthesis transform block in a decoder), denoted by / 3 earlier and further in the disclosure, (e.g. as a second ML predictor);
[0478] together so as to have a mutually shared training stage 302.
[0479] It should be noted that in similarity to the embodiment presented in Fig. 12, this embodiment of Fig. 13 comprises: some training objectives involved in the mutually shared training stage 302 not depending on the bitrate cost model 600 and the dequantization function 500; and some other training objectives involved in the mutually shared training stage 302 depending on both the bitrate cost model 600 and the dequantization function 500.
[0480] The mutually shared training stage 302 is initialized with an initial (or zeroth) epoch 3060in which all of the ML transform predictor for block based encoding 106 (along with its initial parameters 206) and the ML transform predictor for block based decoding 108 (along with its initial parameters 208) provide an input to the initial training objective 3160. Thus, the training objective 3160depends on the ML predictors 106, 108 (i.e. the training objective 3160depends on a, P) as well as on the dequantization function 500 and the bitrate cost model 600. It should be noted that the training objective 3160also depends on the ML prediction predictor 110 providing an input to it, although the ML prediction predictor 110 in this embodiment is not part of training in the mutually shared training stage 302. In other words, the parameters of the ML prediction predictor stay unchanged and are not updated in this embodiment. An initial set of subsequent parameters 3560are obtained and outputted as inputs to the first training objective 3161 of the first subsequent epoch 304i. In the course of this epoch 304i, the initial parameters of the ML predictors 106, 108 are updated to the initial set of subsequent parameters 3540. The mutually shared training stage 302 proceeds further with a first subsequent epoch 3041in which all of the ML predictors 106, 108, 110 again provide an input to the first training objective 3161, wherein the initial set of subsequent parameters 3560are taken as the inputs for the ML predictors 106, 108 while the ML prediction predictor 110 provides the same input as per the initial epoch 3040. Thus, the training objective 3161 still depends on the ML predictors 106, 108, 110 (i.e. the training objective 316i depends on a, / 3, p) as well as on the dequantization function 500 and the bitrate cost model 600. A first set of subsequent parameters 356i are obtained and outputted as inputs to the second training objective 3162of the second subsequent epoch 3042. In the course of this epoch 3042the initial parameters of the ML
[0481] filing version, rmpredictors 106, 108 are updated to the first set of subsequent parameters 356i while leaving the parameters of the ML predictor 110 unchanged.
[0482] The mutually shared training stage 302 proceeds further with a second subsequent epoch 3042 in which the ML predictors 108, 110 provide an input to the second training objective 3162while the ML predictor 106 does not provide an input to the second training objective 3162, wherein the first set of subsequent parameters 356i are taken as inputs for the ML predictor 108 while the ML prediction predictor 110 provides the same input as per the initial epoch 3060. Further, the dequantization function 500 and the bitrate cost model 600 do not provide an input to the second training objective 3162but instead of the latter, the rate-distortion optimized quantized, RDOQ, 700 provides an input. Thus, the training objective 3162depends on the ML predictors 108, 110 (i.e. the training objective 3162depends on / 3, p) as well as on the RDOQ 700. A second set of subsequent parameters 3562are obtained and outputted as inputs to the third training objective 3163of the third subsequent epoch 3043. In the course of the second epoch 3042, the first set of subsequent parameters 356i of the ML predictor 108 is updated to the second set of subsequent parameters 3562, while the parameters of the ML predictor 106 and of the ML predictor 110 are left unchanged.
[0483] The mutually shared training stage 302 proceeds further with a third subsequent epoch 3043in which all of the ML predictors 106, 108, 110 again provide an input to the third training objective 3163, wherein the second set of subsequent parameters 3562is taken as the input for the ML predictors 108, the first set of subsequent parameters 356i is taken as the input for the M L predictor while the M L prediction predictor 110 provides the same input as per the initial epoch 3040. Thus, the training objective 3163still depends on the ML predictors 106, 108, 110 (i.e. the training objective 3163depends on a, / 3, p) as well as on the dequantization function 500 and the bitrate cost model 600. A third set of subsequent parameters 3563are obtained and outputted as inputs to the fourth training objective 3164of the fourth subsequent epoch 3044. In the course of the third epoch 3043the first set of subsequent parameters 356i of the ML predictor and the second set of subsequent parameters 3562of the ML predictor 108 are updated to the third set of subsequent parameters 3563while leaving the parameters of the ML predictor 110 unchanged.
[0484] The mutually shared training stage 302 proceeds further with a fourth subsequent epoch 3042in which the ML predictors 108, 110 provide an input to the fourth training objective 3162while the ML predictor 106 does not provide an input to the fourth training objective 3162, wherein the third set of subsequent parameters 3563are taken as inputs for the ML predictor 108 while the ML prediction predictor 110 provides the same input as per the initial epoch 3060. Further, the dequantization function 500 and the bitrate cost model 600 do not provide an input to the second training objective 3162but instead of the latter, the rate-distortion optimized quantized,
[0485] filing version, rmRDOQ, 700 provides an input. Thus, the training objective 3164depends on the ML predictors 108, 110 (i.e. the training objective 3164depends on / 3, p) as well as on the RDOQ 700. A fourth set of subsequent parameters 3564are obtained and outputted as inputs to a next training objective of a next subsequent epoch 3043. In the course of the fourth epoch 3044, the third set of subsequent parameters 3563of the ML predictor 108 is updated to the fourth set of subsequent parameters 3562, while the parameters of the ML predictor 106 and of the ML predictor 110 are left unchanged.
[0486] Thus, in accordance with embodiments, as part of training, the training objective may switch (e.g. alternates) between:
[0487] • a training objective 3160, 316i, 3163depending on
[0488] o the machine learning transform predictor 106 (e.g. a) for block based encoding, o the machine learning transform predictor 108 (e.g. P) for block based decoding, o the machine learning prediction predictor 110 (e.g. p),
[0489] o the dequantization function 500 and / or the bitrate cost model 600
[0490] so as to update parameters of the corresponding machine learning predictors 106, 106, 108 in an epoch 3O4o, 304i, 3043of the mutually shared training stage 302; and
[0491] • a training objective 3162, 3164depending on
[0492] o the machine learning transform predictor 108 (e.g. ) for block based decoding, o the machine learning prediction predictor 110 (e.g. p), and
[0493] o a rate distortion optimized quantization (700)
[0494] so as to update parameters of the corresponding machine learning predictor 108 in a subsequent epoch 3042, 3044of the mutually shared training stage 302 while leaving the parameters of the machine learning transform predictor 106 (e.g. a) for block based encoding and the machine learning predictor 110 unchanged.
[0495] At the culmination of the mutually shared training stage 302 which may be reached or arrived at in accordance to a convergence criterion (e.g. of the training error), obtained subsequent parameters of such a final epoch may then be used to obtain the updated parameters 406 of the ML predictor 106 and the updated parameters 408 of the ML predictor 108. In other words, the ML predictor 106 may have the updated parameters 406 as its trained parameters (or optimal parameters) and the ML predictor 108 may have the updated parameters 408 as its trained parameters (or optimal parameters).
[0496] Further to the embodiments described by Figs. 11-13, a tuple of parameters based mutually shared training of one or more machine learning predictors is now described.
[0497] filing version, rmIn accordance with embodiments, the method of training a machine learning predictor may comprise providing a plurality of initial sets of parameters of one or more machine learning predictors as a plurality of initial tuples.
[0498] Herein each of the plurality of initial sets of parameters is an initial tuple, i.e. in other words a first initial tuple is a collection of different initial sets A, B, P denoted as (A°o, B°o, P°o), a second initial tuple is a collection of different initial sets A, B, P denoted as (A°i, B°i, P°i) and a Nth initial tuple is a collection of different initial sets of A, B, P denoted as (A°N-I, B°N-I, P°N-I), thus different initial tuples constituting a plurality of initial sets of parameters (A°o, B°o, P°o), (A°i, B°i, P°i),..., (A°N-I, B°N-I, P°N-I), wherein an upper index denotes that the parameters are configured to initialize the training stage of the one or more ML predictors and a lower index denotes a number of the plurality of initial sets of parameters, or a multiplicity thereof, configured to be used for training the one or more machine learning predictors a,p,p. The number here refers to a number, or a count, of tuples used for training and is denoted as N in harmony with the aforedescribed notation. The number or count can be understood as a parameter indicative of a dimension of enumeration / enlisting the different tuples. It should be noted that A corresponds to parameters of the ML predictor a, B corresponds to parameters of the ML predictor / 3,and P corresponds to parameters of the ML predictor p]
[0499] The method may further comprise subjecting the plurality of initial tuples to a mutually shared training stage of the one or more machine learning predictors for obtaining updated sets of parameters as an updated tuple (e.g. or equivalently an optimal tuple, which may be denoted (Aopt, Bopt, Popt)). The method may further comprise: within an epoch of the mutually shared training stage, computing a training error for each of the plurality of initial tuples according to a training objective. The training objective may depend on the dequantization function, and the bitrate cost model, and / or one or more of the one or more machine learning predictors.
[0500] Further, within the epoch of the mutually shared training stage, the method may comprise updating a selected tuple of the plurality of initial tuples so as to obtain a subsequent set of parameters as a subsequent tuple while disregarding the rest of the plurality of initial tuples from the mutually shared training stage. For example, the selected tuple (A°j, B°j, P°j) being the / -th tuple of the N tuples, and thus the subsequent tuple is (A1;, B1j, P1i), wherein the change in index denotes an update of the parameters for use as input in the subsequent epoch. The selected tuple, e.g. selected tuple (A°j, B°j, P°i), may be selected among the plurality of initial tuples on a basis of its corresponding training error being lowest in value. That is, i.e. in other words, the selected tuple (A°j, B°j, P°j) has a lowest training error compared to all other tuples of the plurality of initial tuples.
[0501] filing version, rmFurther, with, in the epoch of the mutually shared training stage, the method may comprise outputting the subsequent set of parameters as inputs for a subsequent epoch of the training stage.
[0502] The application proceeds further with a provision of embodiments and details in the form of examples relating to non-linear transforms and prediction methods for VVC intra coding. Such embodiments and examples may be organized in sections. At first, a brief summary of such organized sections is described.
[0503] Section 1.1 describes (e.g. considers) the problem of optimizing non-linear modifications of the analysis transform (e.g. as described in Figs. 4-6 & 9 and earlier in the disclosure by way of the further ML transform predictor 34, denoted by p, and the ML transform predictor 38, denoted by a). In Section 1.2, non-linear transform coding tools are described (e.g. developed) by using a novel method for considering the impact of the rate-distortion optimized quantization inside the training. The following Section 1.3 describes (e.g. goes on to investigate) a data-driven training approach for estimating the bitrate impact of the transform coefficients which is based on using a parametrized probability model (e.g. the bitrate cost model 600 described in Figs. 10, 12 & 13 and earlier in the disclosure). Using the approach, embodiments relating to implementations of an end-to-end optimization of the analysis and synthesis transform are described. SecHon 1.4 then provides embodiments based on (e.g. uses, or making use of) this approach for obtaining non-linear transforms (e.g. as implementable by ML transform predictors 38, 34 - a, respectively) in combination with a nonlinear intra prediction mode (e.g. as implementable by ML prediction predictor 36 - p) from a training experiment. Furthermore the integration of these transforms and prediction modes into the VTM are described and the resulting bitrate savings are presented. The results are discussed in SecHon 1.5 and conclusions are drawn in the final Section 1.6.
[0504] Some embodiments for Non-linear Transforms and prediction methods for VVC Intra Coding are presented. These embodiments may also be described as being related to Non-linear Transforms and Intra Prediction Modes for Video Compression.
[0505] 1.1 Introduction
[0506] Embodiments relating to developing a novel ansatz for jointly optimizing non-linear transforms and non-linear intra predictions for block-based video coding are presented herein. Recall that, in convenHonal video coding, the transform coefficients are obtained from the original signal s by applying an intra prediction pred(.) with the reference samples sras input and an analysis transform A thereafter. The quantizer with step size A is applied to the residual which yields
[0507] u = A(s − pred(ŝr))
[0508] =P (1.1.1)
[0509] q = Q(u, A)
[0510] (1.1.2)
[0511] filing version, rmIn practical video coding applications, the forward transform is usually determined by A = BTwhere B denotes the orthogonal synthesis transform matrix. The encoder considers the impact of the quantization indices on the bitrate and the distortion as follows
[0512]
[0513] where the choice of the step size depends on the Lagrange parameter A. The reconstructed signal is then given by applying the synthesis transform to the reconstructed coefficients and adding the intra prediction which yields
[0514] û = DQ(q, Δ), (1.1.4)
[0515] ŝ = Bû + p. (1.1.5)
[0516] A learning-based approach for obtaining a non-linear modification of the synthesis transform is now described. This modification is an offset which is computed from the reconstructed coefficients in (1.1.4) and possibly from the reference samples ®r. Then, it is added to the transform block before the synthesis transform is performed. However, for determining a similar non-linear modification of the analysis transform before the quantization in (1.1.2), the proposed training ansatz does not work. It is based on obtaining a set of quantization indices (for instance, by encoding a large collection of video sequences) and optimizing the transform offset f with respect to the reconstruction error
[0517] MSE(s, B(u + f) + p).
[0518] The reconstructed coefficients certainly reflect characteristics of the analysis transform, but when stochastic gradient descent is performed, all gradients with respect to parameters inside the quantization function necessarily become zero. Vice versa, when the quantized data itself is used as training data, only the synthesis part of the transform coding stage is enhanced by the aforementioned non-linear method. The encoder is implicitly adapted at the inference stage by performing (1.1.3) with regard to the modified reconstruction. With this approach, the problem of smoothly approximating the quantization at the training stage is circumvented. The quantization is proven to have a strong impact on the coding efficiency of the learned non-linear transforms and ideally, the training stage should reflect the influence of the quantization. However, when rate-constrained quantization with a given entropy coding like the coefficient coding stage of VVC is applied, including the impact of the quantizer as part of the training is not trivial.
[0519] In accordance with some embodiments disclosed in the present application, the linear forward transform in (1.1.1) is replaced by a non-linear analysis transform of the residual error
[0520] u = α(s − p; ψα),
[0521] where denotes the parameters to be trained. Correspondingly, a non-linear synthesis transform replaces (1.1.5) and maps the reconstructed coefficients u back into the sample domain as follows
[0522] ŝ = β(û; ψβ) + p.
[0523] filing version, rmAs an additional degree of freedom, not only the transforms a and but the intra prediction may also be non-linear and trainable. Here, the non-linear intra prediction is given by
[0524] p = p(ŝr; ψp).
[0525] The parameters
[0526]
[0527] shall be obtained from a training experiment by minimizing the expected rate-distortion cost as follows
[0528]
[0529] Here, denotes the training dataset and Z(q) is the bitrate of the quantized transform coefficients. Since both the prediction and the synthesis transform are applied outside the quantization, updates with respect to these parameters can be computed regardless of whether these methods are linear or non-linear. However, it is important to note that the training objective (1.1.6) may not be used in combination with stochastic gradient descent for obtaining the parameters
[0530]
[0531] of the analysis transform. Thus, it is preferred that the following two problems be solved.
[0532] First, for additionally optimizing the analysis transform, a differentiable approximation of the quantization process described by Q( •, A) is preferred. Otherwise, when back propagation is used for computing the gradients with respect to a, the derivative becomes zero due to the chain rule. Second, different from the synthesis transform, the analysis transform immediately affects the bitrate and therefore, for the optimization an approximation of the expected bitrate is preferred. While adding uniform noise is known to work well for optimizing non-linear transforms in learned image compression, the situation here is more complicated. The encoder typically performs rate-distortion optimization methods in (1.1.3) which are aligned with the entropy coding of the transform coefficients.
[0533] Second, different from the synthesis transform, the analysis transform immediately affects the bitrate and therefore, the optimization requires an approximation of the expected bitrate. The entropy coding in learned image compression is determined by a parametrized probability density which is part of the training stage. Thus, the impact of the analysis transform u = α(s −
[0534]
[0535] of the resulting quantization vector q = Q(u, A) will be considered in a similar manner.
[0536] 1.2 Non-linear Modifications of the Analysis and Synthesis Transform
[0537] Non-linear modifications of the analysis transform can be optimized in combination with the non-linear transform offset on the decoder side when the quantization at the training stage is adapted. First, the suitability of the quantization approximation will be presented while the design of the learned transforms is motivated by the findings from [Sch+24], However, different concepts for non-linear transforms will be presented throughout the following sections.
[0538] It has already been described that a non-linear transform offset can be applied to the reconstructed transform coefficients for reducing the distortion in the sample domain. In more
[0539] filing version, rmdetail, the offset f is added to the coefficients before the synthesis transform is performed which yields
[0540] s = B(u + f) + p. (1.2.1)
[0541] By optimizing with respect to the reconstruction quality, one obtains the parameters
[0542]
[0543] computing from a training experiment as follows
[0544]
[0545] f = V((DQ ∘ Q(u, Δ), p, QP, mode), ψf)
[0546] ' (1.2.2)
[0547] Here, the inputs of the neural network V(-,xpf ) are the reconstructed coefficients, the prediction signal, the QP and the intra mode index. The joint application of B and
[0548]
[0549] on the decoder side yields a non-linear synthesis transform. Therefore, it seems reasonable to construct a non-linear analysis transform on the encoder side in a similar manner. However, when using an additive offset, one needs to take into account that increasing the magnitude of the transform coefficients will have a negative impact on the expected bitrate. The reasoning behind this is similar to the one for designing the non-linear coefficient prediction before. The / 1-norm of the transform block correlates to some degree with its expected bitrate and thus, increasing the coefficients could have a negative impact on the bitrate. Vice versa, reducing the absolute values of the coefficients on average should reduce the expected bitrate of the trans- form block. The following non-linear modification of the analysis transform is used for generating smaller transform coefficients
[0550] v := u − sign(u) · abs(g), (1.2.3)
[0551] g = V((u, p, QP, mode), ψg). (1.2.4)
[0552] Here, the sign and abs functions are applied element-wise, and • denotes element-wise multiplication. After the parameters have been determined in the training stage, these modifications are included into the video codec as follows. When rate-constrained quantization is performed, the distortion in the transform domain of v is considered. Note that this is different from considering the sample distortion since the offset g has been applied to the transform coefficients u as described in (1.2.3). The sample distortion can only be computed correctly for entire quantization vectors, when the impact of g and f on all transform coefficients has been considered. Thus, the optimization of the quantization indices until, and including, the AC1 coefficient is given by
[0553]
[0554] At the final scan index, the true distortion impact can be computed as follows. The reconstructed coefficients are determined by adding the transform offset to the quantization vector which yields
[0555] û(q) := DQ(q, Δ) + V((DQ(q, Δ), p, QP, mode), ψf).
[0556] =f
[0557] filing version, rmThe distortion is only computed for a small set of candidates which are tested at the final scan index. Recall that VVC supports both uniform scalar quantization and trellis-coded quantization with four states. In the case of TCQ, there are eight possible input vectors to be tested:
[0558]
[0559] (1.2.6) The eight candidates are derived following a similar motivation (e.g. consideration for) provided by the non-linear transform offset application presented in Sections II and III of [Sch+24],
[0560] 1.2.2 Training Details
[0561] Updates with respect to ψgcannot be computed when v is quantized. In learned image compression, this problem is usually avoided by either using a smooth approximation of the quantizer or adding uniform noise. However, adding uniform noise is designed to approximate uniform scalar quantization whereas the encoder in block- based video coding typically employs rate-constrained quantization. Thus, a modified quantizer is applied at the training stage for capturing the statistical properties of the quantization vectors, which are determined by the encoder:
[0562] given by Q(v, A),
[0563]
[0564] (1 27) The function dq( •, A) performs the aforementioned rate-distortion optimized quantization but sets each coefficient either to zero or leaves it at its original value if its quantization index was non-zero. Thus, it approximates the quantization and the reconstruction process which is given by DQ oQ( •, A). When stochastic gradient descent is performed, the impact of the analysis transform on the larger coefficients of the transform block is taken into account for computing the gradients. Expectedly, these are exactly the coefficients with the highest bitrate impact inside the transform block since zero coefficients are efficiently coded by a single flag. The training is performed by alternating between
[0565] min Ear [MSE(s, B(DQ o Q(v, A) + f) + p)]
[0566] ’
[0567] f - ¥((DQo Q(v, A),p)QP,mode), ’ / )
[0568] , (1-2.8)
[0569] and
[0570]
[0571] g = ¥((u, p, QP, mode), ψg).(129)
[0572] filing version, rmThe first training epoch employs (1.2.8) while initializing g = 0 in (1.2.3). The training is performed epoch-wise as usual but switches between updating only the parameters ψfin one epoch and only in the following epoch. The rationale behind alternating between both loss functions is that the training converges more stably. When (1.2.9) is used, the weights of f have been optimized to the reconstructed transform blocks as impacted by the current, fixed value of ψg. Furthermore, since the training objective in both instances is the reconstruction error, the magnitude of g is implicitly restricted to small values compared to u. Finally, the training data x is obtained by encoding the BVI-DVC database [MZB22] with a reference software implementation of VVC:
[0573] := {(s, ŝr, p, u = BT(s − p), mode, QP)i|i = 0, 1, 2,...}
[0574] As before, distinct training sets are used for each block shape and different network parameters are learned per shape, per transform option and per intra mode.
[0575] 1.2.3 Integration into the VTM-14.2
[0576] Different networks for computing the transform modifications f and g have been trained and implemented in the VTM-14.2 with the following main features:
[0577] • The supported block shapes and transform options are stated in Table 1.1 This yields a total of 182 distinct networks for computing the offset f and another 182 ones for computing the update g.
[0578] • Blocks of shape 32 x 32 and 64 x 64 use the weights optimized for 16 x 16.
[0579] • The supported transform options are DCT-II and DST-VII. Consequently, the use of f and g is disabled when an LFNST transform is applied. The network weights for the DST-VII as primary transform have been obtained in separate training runs from the weights for the DCT-II.
[0580] • The supported intra modes are PLANAR, DC, and the 65 angular modes. This excludes the use of f and g when the intra mode is one of the MIP modes. Given a fixed primary transform, say the DST-VII, the network weights for the different intra modes have been trained jointly.
[0581] Here, the network architecture of the transform modifications on the encoder and the decode side are the same. Let
[0582]
[0583] the corresponding set of parameters.
[0584] Table 1.1: Overview of the network architecture for computing the non-linear offset f and the update g on the encoder side per block shape W x H. Each network uses four hidden ReLU-layers and employs the prediction signal p as input. The columns refer to the last valid position L of the input, the hidden layer size K and number of multiplications per sample.
[0585]
[0586] filing version, rm
[0587]
[0588] For assessing the impact of the update on the encoder side, a second set of net- work weights f == { 'J’0'J’181} has been trained by fixing g = 0 while using the training objective (1.2.8). Note that using (1.2.9) for optimizing the parameters of f yields no additional benefit since all trainable parameters are applied after the quantization has been performed. In particular, it is reasonable to optimize f with respect to the distribution of DQ ° Q(u, A) rather than dq(u, A).
[0589] 1.2.4 Coding Results
[0590] The coding results are generated by encoding the CTC test set in All-Intra configuration and comparing against the VTM 14.2 anchor. The test codec applies the offsets f and g as described in the previous section when the block shape, the transform option and the intra mode are supported. The QP values are set to QP e {22, 27, 32, 37, 42}and BD-rates are computed athigh rates and low rates as before by using four successive values. The resulting bitrate savings are summarized in Table 1.2.
[0591] The coding gain of 1 % at high rates and 1.2 % at low rates in (a) has been achieved by applying the update g on the encoder side and the offset f on the decoder side. In contrast to this, omitting the update g in (b) while using distinctly optimized network weights for f yields a significant drop to 0.8 % at high rates and to 1.1 % at low rates. Note that the decoder complexity in (a) and (b) is similar since the network architecture of the non-linear offset which is added before the synthesis transform has not been altered. These results suggest that the function in (1.2.7) can be used in a reasonable manner for jointly optimizing the analysis and synthesis transform.
[0592] Table 1.2: The luma (Y) BD-rates of the learned, non-linear modifications of the analysis and synthesis transforms against VTM-14.2 in Al configuration. The headers state which offsets and which network weights have been used.
[0593]
[0594] filing version, rm
[0595]
[0596] Finally, the coding efficiency increases when the network weights in
[0597]
[0598] and ' g are allowed in combination with the LFNST transforms and the MIP intra modes. In the case of the LFNST, the network weights optimized for the DCT-II are re-used and g is combined with the DCT-II before applying the LFNST forward transform, i.e.
[0599]
[0600] ii) • abs(g))
[0601] The corresponding filter is applied at the decoder side in the domain of the DCT-I I, after the inverse LFNST has been performed. As for MIP, the network weights which have been optimized for the PLANAR intra mode are re-used, similar to how the MIP modes are handled by other coding tools in VVC, as well. The coding results due to this modification are summarized in Table 1.3. Note that no additional network weights have been trained and the results in (a) and (b) are accomplished using the same set of weights. Applying f and g to LFNST and MIP blocks i...
Claims
CLAIMS1. Decoder (50) for block based decoding of a predetermined block of a picture, configured to:predict (14) a predicted sample filling of the predetermined block;decode a synthesis transform block (32) of a prediction residual for the predetermined block from the data stream (40);reconstruct the predetermined block bysubjecting the synthesis transform block (32) to a machine learning transform predictor (34), which is configured to transform, or perform a transform transition of, the synthesis transform block (32) from a transform domain to a sample domain, to obtain a sample domain prediction residual; correcting the predicted sample filling using the sample domain prediction residual.
2. Decoder (50) of claim 1, whereinthe synthesis transform block (32) has been obtained by an encoder (10) using a further machine learning transform predictor (38), which is configured to transform, or perform a transform transition of, the prediction residual from the sample domain to the transform domain.
3. Decoder (50) of any one of the previous claims 1 or 2, wherein the machine learning transform predictor (34) is non-linear with respect to the synthesis transform block.
4. Decoder (50) of any one of the previous claims 1 to 3, wherein the machine learning transform predictor (34) is configured to additionally take as input one or more previously decoded samples neighbouring the predetermined block.
5. Decoder (50) of any one of the previous claims 1 to 4, wherein the machine learning transform predictor (34) is non-linear with respect to the one or more previously decoded samples neighbouring the predetermined block.
6. Decoder (50) of any one of the previous claims 1 to 5, wherein the machine learning transform predictor (34) is configured to additionally take as input the predicted sample filling.
7. Decoder (50) of any one of the previous claims 1 to 6, wherein the machine learning transform predictor (34) is non-linear with respect to the predicted sample filling.filing version, rm8. Decoder (50) of any one of the previous claims 1 to 7, wherein the transform transition of the synthesis transform block (32) is configured tosequentially transform fromthe transform domain to an intermediate transform domain, and / or the intermediate transform domain to the sample domain.
9. Decoder (50) of any one of the previous claims 1 to 8, whereinthe transform domain and / or the intermediate transform domain are configured to be at least one ofa DCT domain, a DST domain, a LFNST transform domain, a linear transform domain and an orthogonal transform domain.
10. Decoder (50) of any one of the previous claims 1 to 9, wherein transformations relating the transform domain to the intermediate transform domain, and / orthe intermediate transform domain to the sample domain,are configured to be a linear transform or an orthogonal transform.
11. Decoder (50) of any one of the previous claims 1 to 10, configured to predict the predicted sample filling of the predetermined block by inter prediction or intra prediction.
12. Decoder (50) of previous claim 11, configured topredict the predicted sample filling of the predetermined block byselecting a selected intra-prediction mode out of different intra-prediction modes signalled in the data stream, andpredict the predicted sample filling using the selected intra-prediction mode.
13. Decoder (50) of previous claim 12, wherein the different intra-prediction modes includes one or more ofa planar mode,a DC mode,a plurality of angular modes,one or more matrix-based intra prediction modes, according to each of which the predicted sample filling is obtained by a multiplication between a matrix associated with the respective matrix-based intra prediction mode and a vectorfiling version, rmobtained from sample values in a neighbourhood of the predetermined block, andone or more intra-subpartition modes, according to each of which the predicted sample filling is obtained by subdividing the predetermined block into subpartitions and intra-predicting the subpartitions sequentially with correcting a subpartition using the sample domain prediction residual before intra- predicting a next subpartition.
14. Decoder (50) of any one of the previous claims 1 to 13, configured to support different block sizes for the predetermined block and to correct the predicted sample filling using the sample domain prediction residual by varying one or more ofa number of parameters defining the machine learning transform predictor (34), the parameters defining the machine learning transform predictor (34) depending on a block size of the predetermined block.
15. Decoder (50) of any one of the previous claims 1 to 14, configured to support different intra-prediction modes for the predetermined block signalled in the data stream and to correct the predicted sample filling using the sample domain prediction residual by varying one or more ofa number of parameters defining the machine learning transform predictor (34), the parameters defining the machine learning transform predictor (34) depending on an intra prediction mode of the predetermined block.
16. Decoder (50) of any one of the previous claim 1 to 15, configured to correct the predicted sample filling using the sample domain prediction residual by varying one or more ofa number of parameters defining the machine learning transform predictor (34), the parameters defining the machine learning transform predictor (34) depending on one or more previously decoded samples neighbouring the predetermined block.
17. Decoder (50) of any one of the previous claims 1 to 16, wherein the machine learning transform predictor (34) is implemented byan initial synthesis transform configured tofiling version, rmreceive a set of transform coefficients of the synthesis transform block (32), andobtain an initial sample domain prediction residual based on the set of transform coefficients;a neural network comprising an initial non-linear fully-connected layer and one or more succeeding layers, the neural network configured toreceive the set of transform coefficients as inputs of the initial layer, receive an output of the initial layer as an input of the one or more succeeding layers; anda combiner unit configured to combine the initial sample domain prediction residual and an output of a final layer among the one or more succeeding layers to obtain the sample domain prediction residual.
18. Decoder (50) of previous claim 17, wherein the final layer of the one or more succeeding layers is a linear layer.
19. Decoder (50) of any one of the previous claims 17 or 18, wherein the neural network is configured to additionally take at least one of:the predicted sample filling,the one or more previously decoded samples neighbouring the predetermined block,a quantization parameter of the predetermined block, andan index of a selected intra-prediction mode selected from a plurality of intraprediction modes,as inputs of the initial layer of the neural network.
20. Decoder (50) of any one of the previous claims 17 to 19, wherein the neural network is configured to receive the set of transform coefficients of the synthesis transform block (32) concatenated with the predicted sample filling as inputs of the initial layer of the neural network.
21. Decoder (50) of previous claim 19, wherein the neural network is configured to additionally receive at least one of:the one or more previously decoded samples neighbouring the predetermined block,filing version, rmthe quantization parameter of the predetermined block, andthe index of a selected intra-prediction mode selected from a plurality of intraprediction modes,concatenated with the set of transform coefficients and the predicted sample filling as inputs of the initial layer of the neural network.
22. Decoder (50) of any one of the previous claims 17 to 21, wherein the initial synthesis transform is an orthogonal transform.
23. Decoder (50) of any one of the previous claims 17 to 22, wherein the initial synthesis transform is at least one of DCT, DST and a linear transform.
24. Decoder of any one of the previous claims 17 to 23, wherein the decoder is configured tosupport one or more of a number of different initial synthesis transforms of the machine learning transform predictor (34), wherein each initial synthesis transform is related to a neural network implementing the machine learning transform predictor (34), and correct the predicted sample filling using the sample domain prediction residual obtained by selecting and using a corresponding neural network related to an initial synthesis transform among the different initial synthesis transforms of the machine learning transform predictor (34).
25. Decoder of any one of the previous claims 17 to 24, wherein the decoder is configured tosupport one or more of a number of different block sizes of the predetermined block, wherein each block size is related to a neural network implementing the machine learning transform predictor (34), andcorrect the predicted sample filling using the sample domain prediction residual obtained by selecting and using a corresponding neural network related to a block size among the different block sizes of the predetermined block.
26. Decoder of any one of the previous claims 17 to 25, wherein the decoder is configured tosupport one or more of a number of different block sizes of the predetermined block, wherein each block size is related to a neural network implementing the machine learning transform predictor (34), andfiling version, rmcorrect the predicted sample filling using the sample domain prediction residual obtained by selecting and using a corresponding neural network related to a block size among the different block sizes of the predetermined block, wherein, in the case of the block size exceeding a predetermined block size value, the corresponding neural network takes, as an input, a sub-portion of the synthesis transform block (32).
27. Decoder of any one of the previous claims 17 to 26, wherein the decoder is configured tosupport one or more of a number of different block sizes of the predetermined block, wherein each block size is related to a neural network implementing the machine learning transform predictor (34), andwherein in the case of a block size equalling a mutually transposed block size among the different block sizes of the predetermined block, the decoder is configured to correct the predicted sample filling using the sample domain prediction residual by transposing the input of the initial layer and the output of the final layer of a corresponding neural network related to the mutually transposed block size of the predetermined block, andusing the parameters of the neural network related to the mutually transposed block size of the predetermined block.
28. Decoder of any one of the previous claims 17 to 27, whereineach of the plurality of intra-prediction modes or each group of intra-prediction modes into which the plurality of intra-prediction modes are grouped is related to a neural network implementing the machine learning transform predictor (34), andthe decoder is configured to correct the predicted sample filling using the sample domain prediction residual obtained by selecting and using a corresponding neural network related to an intra-prediction mode or a group of intra-prediction modes.
29. Decoder (50) of any one of the previous claims 1 to 28, wherein the number of transform coefficients in the set of transform coefficients is equal to a number of samples of the synthesis transform block (32).
30. Decoder (50) of any one of the previous claims 1 to 29, configured to decode the set of transform coefficients from the data stream (40) based on a scan order.
31. Decoder (50) of any one of the previous claims 1 to 30, configured to:filing version, rmpredict (14) the predicted sample filling of the predetermined block based on a machine learning prediction predictor (36) by inputting into the machine learning prediction predictor (36) one or more previously decoded samples neighbouring the predetermined block as inputs, and obtaining the predicted sample filling based on an output of the machine learning prediction predictor (36).
32. Decoder (50) of previous claim 31, wherein the machine learning prediction predictor (36) is non-linear with respect to the values of one or more previously decoded samples neighbouring the predetermined block.
33. Decoder (50) of any one of the previous claim 31 or 32, configured to support different block sizes for the predetermined block and to obtain the predicted sample filling by varying one or more ofa number of parameters defining the machine learning prediction predictor (36), the parameters defining the machine learning prediction predictor (36) depending on a block size of the predetermined block.
34. Decoder (50) of any one of the previous claims 31 to 33, configured to obtain the predicted sample filling by varying one or more ofa number of parameters defining the machine learning prediction predictor (36), the parameters defining the machine learning prediction predictor (36) depending on one or more previously decoded samples neighbouring the predetermined block.
35. Decoder (50) of any one of the previous claims 31 to 34, wherein the machine learning prediction predictor (36) is implemented byan initial predictor (63a) configured to receive the values of one or more previously decoded samples (63b) neighbouring the predetermined block and predict an initial predicted sample filling of the predetermined block using the values of one or more previously decoded samples (63b) neighbouring the predetermined block;a neural network (61) comprising an initial non-linear fully-connected layer (63c) and one or more succeeding layers (63d), the neural network (61) configured toreceive the values of one or more previously decoded samples (63b) neighbouring the predetermined block as inputs of the initial layer (63c),filing version, rmreceive an output of the initial layer (63c) as an input of the one or more succeeding layers (63d); anda predictor combiner unit (63f) configured to combine the initial predicted sample filling and an output of a final layer (63e) among the one or more succeeding layers (63d) to obtain the predicted sample filling.
36. Decoder (50) of any one of the previous claims 31 to 35, wherein the neural network (61) is configured toreceive the values of one or more previously decoded samples neighbouring the predetermined block concatenated with the initial predicted sample filling as the inputs of the initial layer.
37. Decoder of any one of the previous claims 31 to 36, wherein the decoder is configured to support one or more of a number of different intra-prediction modes for the initial predictor of the machine learning prediction predictor (36), wherein each intraprediction mode is related to a neural network implementing the machine learning prediction predictor (36), andobtain the predicted sample filling by selecting and using a corresponding neural network related to an intra-prediction mode among the different intra-prediction mode for the initial predictor of the machine learning prediction predictor (36).
38. Decoder of previous claim 37, wherein the decoder is configured tosupport one or more groups of intra-prediction modes into which the different intraprediction modes are grouped for the initial predictor of the machine learning prediction predictor (36), wherein each group of intra-prediction mode is related to a neural network implementing the machine learning prediction predictor (36), andthe decoder is configured to obtain the predicted sample filling by selecting and using a corresponding neural network related to a group of intra-prediction modes.
39. Decoder of any one of the previous claims 31 to 38, wherein the decoder is configured tosupport one or more of a number of different block sizes of the predetermined block (13), wherein each block size is related to a neural network implementing the machine learning prediction predictor (36), andfiling version, rmobtain the predicted sample filling by selecting and using a corresponding neural network related to a block size among the different block sizes of the predetermined block.
40. Decoder of any one of the previous claims 31 to 39, wherein the decoder is configured tosupport one or more of a number of different block sizes of the predetermined block, wherein each block size is related to a neural network implementing the machine learning prediction predictor (36), andobtain the predicted sample filling by selecting and using a corresponding neural network related to a block size among the different block sizes of the predetermined block, wherein, in the case of the block size exceeding a predetermined block size value, the corresponding neural network takes, as an input, a subset of the one or more previously decoded samples neighbouring the predetermined block.
41. Decoder of any one of the previous claims 31 to 40, wherein the decoder is configured tosupport one or more of a number of different block sizes of the predetermined block, wherein each block size is related to a neural network implementing the machine learning prediction predictor (36), andwherein in the case of a block size equalling a mutually transposed block size among the different block sizes of the predetermined block, the decoder is configured to obtain the predicted sample filling bytransposing the input of the initial layer and the output of the final layer of a corresponding neural network related to the mutually transposed block size of the predetermined block, andusing the parameters of the neural network related to the mutually transposed block size of the predetermined block.
42. Decoder (50) of any one of the previous claims 31 to 41, wherein the final layer of the one or more succeeding layers is a linear layer.43 Decoder (50) of any one of the previous claims 31 to 42, wherein the different intraprediction modes for the initial predictor includes one or more ofa planar mode,a DC mode,filing version, rma plurality of angular modes,one or more matrix-based intra prediction modes, according to each of which the predicted sample filling is obtained by a multiplication between a matrix associated with the respective matrix-based intra prediction mode and a vector obtained from sample values in a neighbourhood of the predetermined block, andone or more intra-subpartition modes, according to each of which the predicted sample filling is obtained by subdividing the predetermined block into subpartitions and intra-predicting the subpartitions sequentially with correcting a subpartition using the sample domain prediction residual before intra- predicting a next subpartition.
44. Encoder (10) for block based encoding of a predetermined block of a picture, configured to:predict (14) a predicted sample filling of the predetermined block;encode an analysis transform block (20) of a prediction residual for the predetermined block into the data stream, wherein the analysis transform block (20) is for reconstructing the predetermined block bysubjecting a synthesis transform block (32) to a machine learning transform predictor (34) which is configured to transform, or perform a transform transition of, the synthesis transform block (32) from a synthesis transform domain to a sample domain to obtain a sample domain prediction residual;correcting the predicted sample filling using the sample domain prediction residual.
45. Encoder (10) of previous claim 44, configured to:subject the prediction residual for the predetermined block to a further machine learning transform predictor (38), which is configured to transform, or perform a transform transition of, the prediction residual from the sample domain to an analysis transform domain, to obtain the analysis transform block (20) representing a transform domain prediction residual.
46. Encoder (10) of any one of the previous claims 44 or 45, whereinthe machine learning transform predictor (34) is non-linear with respect to the synthesis transform block (32); and / orfiling version, rmthe further machine learning transform predictor (38) is non-linear with respect to the analysis transform block (20).
47. Encoder (10) of any one of the previous claims 44 to 46, wherein the machine learning transform predictor (34) is configured to additionally take as input one or more previously decoded samples neighbouring the predetermined block; and / or the further machine learning transform predictor (38) is configured to additionally take as input one or more previously encoded samples neighbouring the predetermined block.
48. Encoder (10) of any one of the previous claims 44 to 47, wherein the machine learning transform predictor (34) is non-linear with respect to the one or more previously decoded samples neighbouring the predetermined block; and / orthe further machine learning transform predictor (38) is non-linear with respect to the one or more previously encoded samples neighbouring the predetermined block.
49. Encoder (10) of any one of the previous claims 44 to 48, whereinthe machine learning transform predictor (34) is configured to additionally take as input the predicted sample filling; and / orthe further machine learning transform predictor (38) is configured to additionally take as input the predicted sample filling.
50. Encoder (10) of any one of the previous claims 44 to 49, whereinthe machine learning transform predictor (34) is non-linear with respect to the predicted sample filling; and / orthe further machine learning transform predictor (38) is non-linear with respect to the predicted sample filling.
51. Encoder (10) of any one of the previous claims 44 to 50, wherein the transform transition of the synthesis transform block (32) is configured tosequentially transform fromthe synthesis transform domain to an intermediate synthesis transform domain, and / orthe intermediate synthesis transform domain to the sample domain.
52. Encoder (10) of any one of the previous claims 44 to 51, wherein the transform transition of the analysis transform block (32) is configured tofiling version, rmsequentially transform fromthe sample domain the sample domain for obtaining the sample domain prediction residual to an intermediate analysis transform domain, and / or the intermediate analysis transform domain to the analysis transform domain.
53. Encoder (10) of any one of the previous claims 44 to 52, wherein the analysis transform domain and / or the synthesis transform domain and / or the intermediate analysis transform domain and / or the intermediate synthesis transform domain are configured to be at least one of a DCT domain, a DST domain, a LFNST transform domain, a linear transform domain and an orthogonal transform domain.
54. Encoder (10) of any one of the previous claims 44 to 53, wherein transformations relatingthe synthesis transform domain to an intermediate synthesis transform domain, and / orthe intermediate synthesis transform domain to the sample domain, and / or the sample domain the sample domain for obtaining the sample domain prediction residual to an intermediate analysis transform domain, and / or the intermediate analysis transform domain to the analysis transform domain. are configured to be a linear transform or an orthogonal transform.
55. Encoder (10) of any one of the previous claims 44 to 54, configured to predict the predicted sample filling of the predetermined block by inter prediction or intra prediction.
56. Encoder (10) of previous claim 55, configured topredict the predicted sample filling of the predetermined block byselecting a selected intra-prediction mode out of different intra-prediction modes signalled in the data stream, andpredict the predicted sample filling using the selected intra-prediction mode.
57. Encoder (10) of any one of the previous claims 51 to 56, wherein the different intraprediction modes includes one or more ofa planar mode,a DC mode,a plurality of angular modes,filing version, rmone or more matrix-based intra prediction modes, according to each of which the predicted sample filling is obtained by a multiplication between a matrix associated with the respective matrix-based intra prediction mode and a vector obtained from sample values in a neighbourhood of the predetermined block, andone or more intra-subpartition modes, according to each of which the predicted sample filling is obtained by subdividing the predetermined block into subpartitions and intra-predicting the subpartitions sequentially with correcting a subpartition using the sample domain prediction residual before intra- predicting a next subpartition.
58. Encoder (10) of any one of the previous claims 44 to 57, configured to support different block sizes for the predetermined block and to correct the predicted sample filling using the sample domain prediction residual by varying one or more ofa number of parameters defining the further machine learning transform predictor (38),the parameters defining the further machine learning transform predictor (38) depending on a block size of the predetermined block.
59. Encoder (10) of any one of the previous claims 44 to 58, configured to support different intra-prediction modes for the predetermined block signalled in the data stream and to obtain the analysis transform block (20) representing the transform domain prediction residual by varying one or more ofa number of parameters defining the further machine learning transform predictor (38),the parameters defining the further machine learning transform predictor (38) depending on an intra prediction mode of the predetermined block.
60. Encoder (10) of any one of the previous claims 44 to 59, configured to obtain the analysis transform block (20) representing the transform domain prediction residual by varying one or more ofa number of parameters defining the further machine learning transform predictor (38),the parameters defining the further machine learning transform predictor (38)filing version, rmdepending on one or more previously encoded samples neighbouring the predetermined block.
61. Encoder (10) of any one of the previous claims 44 to 60, wherein the further machine learning transform predictor (38) is implemented byan initial analysis transform (43a) configured toreceive a sample domain prediction residual (43b) for the predetermined block, andobtain an initial analysis transform domain prediction residual based on the sample domain prediction residual for the predetermined block; a neural network (41) comprising an initial non-linear fully-connected layer (43c) and one or more succeeding layers (43d), the neural network (41) configured toreceive the sample domain prediction residual as inputs of the initial layer (43c),receive an output of the initial layer (43c) as an input of the one or more succeeding layers (43d); anda combiner unit (43f) configured to combine the initial analysis transform domain prediction residual and an output of a final layer (43e) among the one or more succeeding layers (43d) to obtain the analysis transform block (20) representing the transform domain prediction residual.
62. Encoder (10) of previous claim 61, wherein the final layer of the one or more succeeding layers is a linear layer.
63. Encoder (10) of any one of the previous claims 61 or 62, wherein the neural network is configured to additionally take at least one of:the predicted sample filling,one or more previously encoded samples neighbouring the predetermined block,a quantization parameter of the predetermined block, andan index of a selected intra-prediction mode selected from a plurality of intraprediction modes,as inputs of the initial layer of the neural network.filing version, rm64. Encoder (10) of any one of the previous claims 61 to 63, wherein the neural network is configured to receive the sample domain prediction residual concatenated with the predicted sample filling as inputs of the initial layer of the neural network.
65. Encoder (10) of any one of the previous claims 61 to 64, wherein the neural network is configured to additionally receive at least one of:the one or more previously encoded samples neighbouring the predetermined block,the quantization parameter of the predetermined block, andthe index of a selected intra-prediction mode selected from a plurality of intraprediction modes,concatenated with the sample domain prediction residual and the predicted sample filling as inputs of the initial layer of the neural network.
66. Encoder (10) of any one of the previous claims 61 to 65, wherein the initial analysis transform is an orthogonal transform.
67. Encoder (10) of any one of the previous claims 61 to 66, wherein the initial analysis transform is at least one of DCT, DST, a linear transform and an identity transform.
68. Encoder of any one of the previous claims 61 to 67, wherein the encoder is configured tosupport one or more of a number of different initial analysis transforms of the further machine learning transform predictor (38)., wherein each initial synthesis transform is related to a neural network implementing the further machine learning transform predictor (38), andobtain the analysis transform block (20) representing the transform domain prediction residual by selecting and using a corresponding neural network related to an initial synthesis transform among the different initial synthesis transforms of the further machine learning transform predictor (38).
69. Encoder of any one of the previous claims 61 to 68, wherein the encoder is configured tosupport one or more of a number of different block sizes of the predetermined block, wherein each block size is related to a neural network implementing the further machine learning transform predictor (38), andfiling version, rmobtain the analysis transform block (20) representing the transform domain prediction residual by selecting and using a corresponding neural network related to a block size among the different block sizes of the predetermined block.
70. Encoder of any one of the previous claims 61 to 69, wherein the encoder is configured tosupport one or more of a number of different block sizes of the predetermined block, wherein each block size is related to a neural network implementing the further machine learning transform predictor (38), andobtain the analysis transform block (20) representing the transform domain prediction residual by selecting and using a corresponding neural network related to a block size among the different block sizes of the predetermined block, wherein, in the case of the block size exceeding a predetermined block size value, the corresponding neural network takes, as an input, a sub-portion of the prediction residual.
71. Encoder (10) of any one of the previous claims 61 to 70, wherein the encoder is configured tosupport one or more of a number of different block sizes of the predetermined block, wherein each block size is related to a neural network implementing the further machine learning transform predictor (38), andwherein in the case of a block size equalling a mutually transposed block size among the different block sizes of the predetermined block, the encoder is configured to obtain the analysis transform block (20) representing the transform domain prediction residual bytransposing the input of the initial layer and the output of the final layer of a corresponding neural network related to the mutually transposed block size of the predetermined block, andusing the parameters of the neural network related to the mutually transposed block size of the predetermined block.
72. Encoder (10) of any one of the previous claims 61 to 71, whereineach of the plurality of intra-prediction modes or each group of intra-prediction modes into which the plurality of intra-prediction modes are grouped is related to a neural network implementing the further machine learning transform predictor (38), andfiling version, rmthe encoder is configured to obtain the analysis transform block (20) representing the transform domain prediction residual by selecting and using a corresponding neural network related to an intra-prediction mode or a group of intra-prediction modes.
73. Encoder (10) of any one of the previous claims 44 to 72, configured to support different block sizes for the predetermined block and to correct the predicted sample filling using the sample domain prediction residual by varying one or more ofa number of parameters defining the machine learning transform predictor (34), the parameters defining the machine learning transform predictor (34) depending on a block size of the predetermined block.
74. Encoder (10) of any one of the previous claims 44 to 73, configured to support different intra-prediction modes for the predetermined block signalled in the data stream and to correct the predicted sample filling using the sample domain prediction residual by varying one or more ofa number of parameters defining the machine learning transform predictor (34), the parameters defining the machine learning transform predictor (34) depending on an intra prediction mode of the predetermined block.
75. Encoder (10) of any one of the previous claims 44 to 74, configured to correct the predicted sample filling using the sample domain prediction residual by varying one or more of a number of parameters defining the machine learning transform predictor (34), the parameters defining the machine learning transform predictor (34) depending on one or more previously reconstructed samples neighbouring the predetermined block.
76. Encoder (10) of any one of the previous claims 44 to 75, wherein the machine learning transform predictor (34) is implemented byan initial synthesis transform configured toreceive a set of transform coefficients of the synthesis transform block (32), andobtain an initial sample domain prediction residual based on the set of transform coefficients;filing version, rma neural network comprising an initial non-linear fully-connected layer and one or more succeeding layers, the neural network configured toreceive the set of transform coefficients as inputs of the initial layer, receive an output of the initial layer as an input of the one or more succeeding layers; anda combiner unit configured to combine the initial sample domain prediction residual and an output of a final layer among the one or more succeeding layers to obtain the sample domain prediction residual.
77. Encoder (10) of previous claim 76, wherein the final layer of the one or more succeeding layers is a linear layer.
78. Encoder (10) of any one of the previous claims 76 to 77, wherein the neural network is configured to additionally take at least one of:the one or more previously reconstructed samples neighbouring the predetermined block,a quantization parameter of the predetermined block, andan index of a selected intra-prediction mode selected from a plurality of intraprediction modes,as inputs of the initial layer of the neural network.
79. Encoder (10) of any one of the previous claims 76 to 78, wherein the neural network is configured to receive the set of transform coefficients of the synthesis transform block (32) concatenated with the predicted sample filling as inputs of the initial layer of the neural network.
80. Encoder (50) of any one of the previous claims 76 to 79, wherein the neural network is configured to additionally receive at least one of:the one or more previously reconstructed samples neighbouring the predetermined block,the quantization parameter of the predetermined block, andthe index of a selected intra-prediction mode selected from a plurality of intraprediction modes,concatenated with the set of transform coefficients and the predicted sample filling as inputs of the initial layer of the neural network.filing version, rm81. Encoder (10) of any one of the previous claims 76 to 80, wherein the initial synthesis transform is one of a plurality of parameters defining the machine learning transform predictor (34).
82. Encoder (10) of any one of the previous claims 76 to 81, wherein the initial synthesis transform is an orthogonal transform.
83. Encoder (10) of any one of the previous claims 76 to 82, wherein the initial synthesis transform is at least one of DCT, DST and a linear transform.
84. Encoder of any one of the previous claims 76 to 83, wherein the encoder is configured to support one or more of a number of different initial synthesis transforms of the machine learning transform predictor (34), wherein each initial synthesis transform is related to a neural network implementing the machine learning transform predictor (34), and correct the predicted sample filling using the sample domain prediction residual obtained by selecting and using a corresponding neural network related to an initial synthesis transform among the different initial synthesis transforms of the machine learning transform predictor (34).
85. Encoder of any one of the previous claims 76 to 84, wherein the encoder is configured to support one or more of a number of different block sizes of the predetermined block, wherein each block size is related to a neural network implementing the machine learning transform predictor (34), andcorrect the predicted sample filling using the sample domain prediction residual obtained by selecting and using a corresponding neural network related to a block size among the different block sizes of the predetermined block.
86. Encoder of any one of the previous claims 76 to 85, wherein the encoder is configured to support one or more of a number of different block sizes of the predetermined block, wherein each block size is related to a neural network implementing the machine learning transform predictor (34), andcorrect the predicted sample filling using the sample domain prediction residual obtained by selecting and using a corresponding neural network related to a block size among the different block sizes of the predetermined block, wherein, in the case of the block size exceeding a predetermined block size value, the corresponding neural network takes, as an input, a sub-portion of the synthesis transform block (32).
87. Encoder of any one of the previous claims 76 to 86, wherein the decoder is configured tofiling version, rmsupport one or more of a number of different block sizes of the predetermined block, wherein each block size is related to a neural network implementing the machine learning transform predictor (34), andwherein in the case of a block size equalling a mutually transposed block size among the different block sizes of the predetermined block, the decoder is configured to correct the predicted sample filling using the sample domain prediction residual by transposing the input of the initial layer and the output of the final layer of a corresponding neural network related to the mutually transposed block size of the predetermined block, andusing the parameters of the neural network related to the mutually transposed block size of the predetermined block.
88. Encoder of any one of the previous claims 76 to 87, whereineach of the plurality of intra-prediction modes or each group of intra-prediction modes into which the plurality of intra-prediction modes are grouped is related to a neural network implementing the machine learning transform predictor (34), andthe encoder is configured to correct the predicted sample filling using the sample domain prediction residual obtained by selecting and using a corresponding neural network related to an intra-prediction mode or a group of intra-prediction modes.
89. Encoder (10) of any one of the previous claims 44 to 88, wherein the number of transform coefficients in the set of transform coefficients of the synthesis transform block (32) is equal to the number of samples of the synthesis transform block (32).
90. Encoder (10) of any one of the previous claims 44 to 89, configured to encode the set of transform coefficients of the synthesis transform block (32) into the data stream based on a scan order.
91. Encoder (10) of any one of the previous claims 44 to 90, configured to:predict (14) the predicted sample filling of the predetermined block based on a machine learning prediction predictor (36) by inputting into the machine learning prediction predictor (36) one or more previously encoded samples neighbouring the predetermined block as inputs, and obtaining the predicted sample filling based on an output of the machine learning prediction predictor (36).
92. Encoder (10) of previous claim 91, wherein the machine learning prediction predictor (36) is non-linear with respect to the values of one or more previously encoded samples neighbouring the predetermined block.filing version, rm93. Encoder (10) of any one of the previous claims 91 or 92, configured to support different block sizes for the predetermined block and to obtain the predicted sample filling by varying one or more ofa number of parameters defining the machine learning prediction predictor (36), the parameters defining the machine learning prediction predictor (36) depending on a block size of the predetermined block.
94. Encoder (10) of any one of the previous claims 91 to 93, configured to obtain the predicted sample filling by varying one or more ofa number of parameters defining the machine learning prediction predictor (36), the parameters defining the machine learning prediction predictor (36) depending on one or more previously encoded samples neighbouring the predetermined block.
95. Encoder (10) of any one of the previous claims 91 to 94, wherein the machine learning prediction predictor (36) is implemented byan initial predictor configured to receive the values of one or more previously encoded samples neighbouring the predetermined block and predict an initial predicted sample filling of the predetermined block using the values of one or more previously encoded samples neighbouring the predetermined block; a neural network comprising an initial non-linear fully-connected layer and one or more succeeding layers, the neural network configured toreceive the values of one or more previously encoded samples neighbouring the predetermined block as inputs of the initial layer, receive an output of the initial layer as an input of the one or more succeeding layers; anda predictor combiner unit configured to combine the initial predicted sample filling and an output of a final layer among the one or more succeeding layers to obtain the predicted sample filling.
96. Encoder (10) of any one of the previous claims 91 to 95, wherein the neural network is configured tofiling version, rmreceive the values of one or more previously encoded samples neighbouring the predetermined block concatenated with the initial predicted sample filling as the inputs of the initial layer.
97. Encoder (10) of any one of the previous claims 91 to 96, wherein the encoder is configured tosupport one or more of a number of different intra-prediction modes for the initial predictor of the machine learning prediction predictor (36), wherein each intraprediction mode is related to a neural network implementing the machine learning prediction predictor (36), andobtain the predicted sample filling by selecting and using a corresponding neural network related to an intra-prediction mode among the different intra-prediction mode for the initial predictor of the machine learning prediction predictor (36).
98. Encoder of any one of the previous claims 91 to 97, whereineach group of intra-prediction modes into which the different intra-prediction modes are grouped is related to a neural network implementing the machine learning transform predictor (34), andthe decoder is configured to obtain the predicted sample filling by selecting and using a corresponding neural network related to a group of intra-prediction modes.
99. Encoder of any one of the previous claims 91 to 98, wherein the encoder is configured tosupport one or more of a number of different block sizes of the predetermined block, wherein each block size is related to a neural network implementing the machine learning prediction predictor (36), andobtain the predicted sample filling by selecting and using a corresponding neural network related to a block size among the different block sizes of the predetermined block.
100. Encoder of any one of the previous claims 91 to 99, wherein the encoder is configured tosupport one or more of a number of different block sizes of the predetermined block, wherein each block size is related to a neural network implementing the machine learning prediction predictor (36), andfiling version, rmobtain the predicted sample filling by selecting and using a corresponding neural network related to a block size among the different block sizes of the predetermined block, wherein, in the case of the block size exceeding a predetermined block size value, the corresponding neural network takes, as an input, a sub-portion of the synthesis transform block (32).
101. Encoder of any one of the previous claims 91 to 100, wherein the encoder is configured tosupport one or more of a number of different block sizes of the predetermined block, wherein each block size is related to a neural network implementing the machine learning prediction predictor (36), andwherein in the case of a block size equalling a mutually transposed block size among the different block sizes of the predetermined block, the decoder is configured to obtain the predicted sample filling bytransposing the input of the initial layer and the output of the final layer of a corresponding neural network related to the mutually transposed block size of the predetermined block, andusing the parameters of the neural network related to the mutually transposed block size of the predetermined block.
102. Encoder (10) of any one of the previous claims 91 to 101, wherein the final layer of the one or more succeeding layers is a linear layer.
103. Encoder (10) of any one of the previous claims 91 to 102, wherein the initial predictor is one of a plurality of parameters defining the machine learning predictor (36).
104. Encoder (10) of any one of the previous claims 91 to 103, wherein the different intraprediction modes for the initial predictor includes one or more ofa planar mode,a DC mode,a plurality of angular modes,one or more matrix-based intra prediction modes, according to each of which the predicted sample filling is obtained by a multiplication between a matrix associated with the respective matrix-based intra prediction mode and a vector obtained from sample values in a neighbourhood of the predetermined block, andfiling version, rmone or more intra-subpartition modes, according to each of which the predicted sample filling is obtained by subdividing the predetermined block into subpartitions and intra-predicting the subpartitions sequentially with correcting a subpartition using the sample domain prediction residual before intra- predicting a next subpartition.
105. Encoder (90) for block based encoding of a predetermined block of a picture, configured to:predict (14) a predicted sample filling of the predetermined block;encode an analysis transform block (20) of a prediction residual for the predetermined block into the data stream (40) byencoding transform coefficients of the analysis transform block (20) of the prediction residual bysubjecting the analysis transform block (20), accompanied with the predicted sample filling, to a machine learning reductive offset predictor (21) so as to obtain a reductive offset block;correcting each transform coefficient of the analysis transform block (20) based on a corresponding absolute value of the reductive offset block and the sign of the transform coefficient to obtain a corresponding corrected transform coefficient of a modified transform block (22) lower in magnitude, andencoding the modified transform block (22) into the data stream (40), as part of a R / D optimizer and / or a prediction loop of the encoder:subjecting the modified transform block (22), accompanied with the predicted sample filling, to a machine learning improvement predictor (30) so as to obtain an additive offset block;forming a sum of the modified transform block (22) and the additive offset block (30) to obtain an improved transform block (23);subjecting the improved transform block to a synthesis transform block (32); and correcting the predicted sample filling using the prediction residual.
106. Encoder (10) of previous claim 105, configured to predict the predicted sample filling of the predetermined block byfiling version, rmselecting a selected intra-prediction mode out of a plurality of intra-prediction modes, andpredicting the predicted sample filling using the selected intra-prediction mode, wherein the machine learning reductive offset predictor and / or the machine learning improvement predictor additionally take as inputan index of the selected intra-prediction mode.
107. Encoder of any one of the previous claims 105 or 106, wherein the plurality of intraprediction modes includes one or more ofa DC mode,a planar mode,a plurality of angular modes,one or more matrix-based intra prediction modes, according to which the predicted sample filling is obtained by a multiplication between a matrix associated with the respective matrix-based intra prediction mode and a vector obtained from sample values in a neighbourhood of the predetermined block, andone or more intra sub-partition modes, according to each of which the predicted sample filling is obtained by subdividing the predetermined block into sub-partitions and intrapredicting the sub-partitions sequentially with correcting a sub-partition using the prediction residual before intra-predicting a next sub-partition.
108. Encoder of any one of the previous claims 105 to 107, wherein the machine learning reductive offset predictor and / or the machine learning improvement predictor additionally take as input a quantization parameter of the predetermined block.
109. Encoder of any one of the previous claims 105 to 108, wherein the machine learning reductive offset predictor is non-linear with respect tothe analysis transform block (20),the predicted sample filling,the quantization parameter of the predetermined block, andthe index of the selected intra-prediction mode.
110. Encoder of any one of the previous claims 105 to 109, wherein the machine learning improvement predictor is non-linear with respect tofiling version, rmthe analysis transform block (20) and / or the modified transform block (22), the predicted sample filling,the quantization parameter of the predetermined block, andthe index of the selected intra-prediction mode.
111. Encoder of any one of the previous claims 105 to 110, wherein the machine learning reductive offset predictor is implemented bya neural network comprising an initial non-linear fully-connected layer and one or more succeeding layers, the neural network configured toreceive the analysis transform block (20), the predicted sample filling, the quantization parameter and the index of the selected intra-prediction mode as inputs of the initial layer,receive an output of the initial layer as an input of the one or more succeeding layers.
112. Encoder of any one of the previous claims 105 to 111, wherein a final layer among the one or more succeeding layers is a linear layer.
113. Encoder of any one of the previous claims 105 to 112, wherein the encoder is configured tosupport one or more of a number of different block sizes of the predetermined block, wherein each block size of the different block is related to a neural network implementing the machine learning reductive offset predictor, andcompute the reductive offset block by selecting and using a corresponding neural network related to a block size of the predetermined block.
114. Encoder of any one of the previous claims 105 to 113, wherein the encoder is configured tosupport one or more of a number of different transform types for the analysis transform block (20), wherein each transform type of the different types is related to a neural network implementing the machine learning reductive offset predictor, and compute the reductive offset block by selecting and using a corresponding neural network related to a transform type of the predetermined block.filing version, rm115. Encoder of any one of the previous claims 105 to 114, wherein the machine learning reductive offset predictor is implemented by a neural network and wherein the encoder is configured tosupport one or more of a number of different block sizes of the predetermined block, wherein each block size of the different block sizes is related to a neural network implementing the machine learning reductive offset predictor, andcompute the reductive offset block by selecting and using a corresponding neural network related to a block size of the predetermined block, wherein, in the case of the block size exceeding a predetermined block size value, the corresponding neural network takes, as an input, a sub-portion of the analysis transform block (20).
116. Encoder of any one of the previous claims 105 to 115, wherein the machine learning reductive offset predictor is implemented by a neural network and wherein the encoder is configured tosupport one or more of a number of different block sizes of the predetermined block, wherein each block size of the different block sizes is related to a neural network implementing the machine learning reductive offset predictor, andwherein in the case of a block size equalling a mutually transposed block size among different block sizes of the predetermined block, the encoder is configured to compute the reductive offset block bytransposing the input of the initial layer and the output of the final layer of a corresponding neural network related to the mutually transposed block size of the predetermined block, andusing the parameters of the neural network related to the mutually transposed block size of the predetermined block.
117. Encoder of any one of the previous claims 105 to 116, wherein the machine learning improvement predictor is implemented bya neural network comprising an initial non-linear fully-connected layer and one or more succeeding layers, the neural network configured toreceive the analysis transform block (20) and / or the modified transform block (22), the predicted sample filling, the quantization parameter and the index of the selected intra-prediction mode as inputs of the initial layer, receive an output of the initial layer as an input of the one or more succeeding layers.filing version, rm118. Encoder of previous claim 117, wherein a final layer among the one or more succeeding layers is a linear layer.
119. Encoder of any one of the previous claims 117 or 118, wherein the encoder is configured tosupport one or more of a number of different block sizes of the predetermined block, wherein each block size of the different block sizes is related to a neural network implementing the machine learning improvement predictor, andcompute the additive offset block by selecting and using a corresponding neural network related to a block size of the predetermined block.
120. Encoder of any one of the previous claims 117 to 119, wherein the machine learning improvement predictor is implemented by a neural network and wherein the encoder is configured tosupport one or more of a number of different transform types for the analysis transform block (20), wherein each transform type of the different transform types is related to a neural network implementing the machine learning improvement predictor, and compute the additive offset block by selecting and using a corresponding neural network related to a transform type of the predetermined block.
121. Encoder of any one of the previous claims 117 to 120, wherein the machine learning improvement predictor is implemented by a neural network and wherein the encoder is configured tosupport one or more of a number of different block sizes of the predetermined block, wherein each block size of the different block sizes is related to a neural network implementing the machine learning improvement predictor, andcompute the additive offset block by selecting and using a corresponding neural network related to a block size of the predetermined block, wherein, in the case of the block size exceeding a predetermined block size value, the corresponding neural network takes, as an input, a sub-portion of the analysis transform block (20) and / or the modified transform block (22).
122. Encoder of any one of the previous claims 117 to 121, wherein the machine learning improvement predictor is implemented by a neural network and wherein the encoder is configured tofiling version, rmsupport one or more of a number of different block sizes of the predetermined block, wherein each block size of the different block sizes is related to a neural network implementing the machine learning improvement predictor, andwherein in the case of a block size equalling a mutually transposed block size among different block sizes of the predetermined block, the encoder is configured to compute the additive offset block bytransposing the input of the initial layer and the output of the final layer of a corresponding neural network related to the mutually transposed block size of the predetermined block, andusing the parameters of the corresponding neural network associated with mutually transposed block size of the predetermined block.
123. Encoder of any one of the previous claims 105 to 122, configured to encode the index of the selected intra-prediction mode into the data stream.
124. Encoder of any one of the previous claims 105 to 123, wherein the analysis transform block (20) is a DCT or a DST or a linear transform.
125. Encoder of any one of the previous claims 105 to 124, configured to encode the analysis transform block (20) and / or the modified transform block (22) of the prediction residual into the data stream using entropy coding.
126. Method of training a machine learning predictor (100) configured to be used for block based coding of a predetermined block of a picture, the method comprising: providing an initial set of parameters (200) of the machine learning predictor (100); subjecting the initial set of parameters (200) to a training stage (300) for obtaining an updated set of parameters (400) based on:a dequantization function (500), wherein the dequantization function (500) is configured to,for each transform coefficient of a set of transform coefficients of a transform block of a prediction residual for the predetermined block, each transform coefficient having an original value,set a dequantized value of the transform coefficient to the original value of the transform coefficient if a quantized value of the transform coefficient is non-zero, andfiling version, rmset a dequantized value of the transform coefficient to zero if a quantized value of the transform coefficient is zero, wherein the quantized value of the transform coefficient is derived by a quantizer based on a quantization parameter of the predetermined block;and / ora bitrate cost model (600), wherein the bitrate cost model (600) is configured to,for each transform coefficient of a set of transform coefficients of a transform block of a prediction residual for the predetermined block, obtain a bitrate cost associated with the transform block using a density function, wherein the density function is configured to output a probability value associated with a quantization index of the transform coefficient, wherein attributes of the density function comprisea scan index of the transform coefficient, a last scan index associated with the transform block, which is indicative of a last non-zero- quantized transform coefficient; anda quantization parameter of the predetermined block.
127. Method of training a machine learning predictor (100) of previous claim 126, wherein the machine learning predictor (100) as a machine learning transform predictor is configured totake, as input,a transform block of a prediction residual for the predetermined block, and / ora predicted sample filling of the predetermined block, and provide, as output, a sample domain prediction residual;and / orfiling version, rmwherein the machine learning predictor (100) as a machine learning transform predictor is configured totake, as input,a sample domain prediction residual for the predetermined block, and / ora predicted sample filling of the predetermined block, and provide, as output, a transform block representing a transform domain prediction residual;wherein the machine learning predictor (100) as a machine learning prediction predictor is configured totake, as input, values of one or more previously coded samples neighbouring the predetermined block, andprovide, as output, a predicted sample filling of the predetermined block; and / orwherein the machine learning predictor (100) as a machine learning reductive offset predictor is configured totake, as input, an analysis transform block (20) of a prediction residual for the predetermined block and a predicted sample filling of the predetermined block, andprovide, as output, a reductive offset block, wherein the reductive offset block is configured tocorrect each transform coefficient of the analysis transform block (20) based on corresponding absolute value of the reductive offset block and the sign of the transform coefficient for obtaining a corresponding corrected transform coefficient of a modified transform block lower in magnitude;and / orwherein the machine learning predictor (100) as a machine learning improvement predictor is configured tofiling version, rmtake, as input, an analysis transform block (20) of a prediction residual for the predetermined block and a predicted sample filling of the predetermined block, andprovide, as output, an additive offset block, wherein the additive offset block is configured tocorrect each transform coefficient of the transform block based on a corresponding value of the additive offset block for obtaining a corresponding improved transform coefficient of an improved transform block greater in magnitude.
128. Method of training a machine learning predictor (100) of any one of the previous claims 126 or 127, wherein the attributes of the density function additionally comprisea number of non-zero-quantized transform coefficients.
129. Method of training a machine learning predictor (100) of any one of the previous claims 126 to 128, whereineach group of a plurality of quantization parameters associated with the predetermined block into which the plurality of quantization parameters is grouped is related to a density function outputting a corresponding probability value depending on the corresponding group of quantization parameters.
130. Method of training a machine learning predictor (100) of any one of the previous claims 126 to 129, whereineach group of a plurality of scan indices associated with the transform block into which the transform block is grouped is related to a density function outputting a corresponding probability value depending on the corresponding group of scan indices.
131. Method of training a machine learning predictor (100) of any one of the previous claims 126 to 130, whereineach group of a plurality of last scan indices associated with the transform block into which the transform block is grouped is related to a density function outputting a corresponding probability value depending on the corresponding group of last scan indices.
132. Method of training a machine learning predictor (100) of any one of the previous claims 126 to 131, wherein,filing version, rmfor each transform coefficient of the set of transform coefficients of the transform block of a prediction residual for the predetermined block,the density function is implemented bya neural network comprising an initial non-linear fully connected layer and one or more succeeding layers, the neural network configured toreceive the transform coefficient, the scan index of the transform coefficient, the last scan index associated with the transform block and the quantization parameter of the predetermined as inputs of the initial layer,receive an output of the initial layer as an input of the one or more succeeding layers, andprovide an output of a final layer among the one or more succeeding layers to obtain the probability value.
133. Method of training a machine learning predictor (100) of any one of the previous claims 126 to 132, further comprising:as an epoch (3040-2) of the training stage (300),computing a training error (3280-2) according to a training objective (3080-2), wherein the training objective (3080-2) depends onthe output of the machine learning predictor (100), and / orthe dequantization function (500), and / orthe bitrate cost model (600),updating the initial set of parameters (200) so as to obtain a subsequent set of parameters (3480-2) related to the corresponding training error (3280-2) being lower in value while leaving parameters of the bitrate cost model (600) unchanged; andoutputting the subsequent set of parameters (3480-2) as inputs for a subsequent epoch (3040-2) of the training stage (300).
134. Method of training a machine learning predictor (100) of any one of the previous claims 126 to 133, further comprising:traininga machine learning improvement predictor (102), andfiling version, rma machine learning reductive offset predictor (104)together so as to have a mutually shared training stage (302);switching between:a training objective (3121, 3123) depending on the additive offset block so as to update parameters of the machine learning improvement predictor (102) in an epoch (3041, 3043) of the mutually shared training stage (302) while leaving parameters of the machine learning reductive offset predictor (104) unchanged, anda training objective (3122, 3124) depending on both the additive offset block and the reductive offset block so as to update the parameters of the machine learning reductive offset predictor (104) in a subsequent epoch (3042, 3044) of the mutually shared training stage (302) while leaving parameters of the machine learning improvement predictor (102) unchanged.
135. Method of training a machine learning predictor of any one of the previous claims 126 to 134, further comprising:traininga machine learning transform predictor (106) for block based encoding, a machine learning transform predictor (108) for block based decoding, and a machine learning prediction predictor (110)together so as to have a mutually shared training stage (302);switching between:a training objective (3141, 3143) depending onthe machine learning transform predictor (106) for block based encoding,the machine learning transform predictor (108) for block based decoding,the machine learning prediction predictor (110), the dequantization function (500) and / or the bitrate cost model (600),filing version, rmso as to update parameters of the corresponding machine learning predictors (106, 108, 110) in an epoch (3041, 3043) of the mutually shared training stage (302); anda training objective (3142, 3144) depending onthe machine learning transform predictor (108) for block based decoding, andthe machine learning prediction predictor (110), so as to update parameters of the corresponding machine learning predictors (108, 110) in a subsequent epoch (3042, 3044) of the mutually shared training stage (302) while leaving the parameters of the machine learning transform predictor (106) for block based encoding unchanged.
136. Method of training a machine learning predictor of any one of the previous claims 126 to 135, further comprising:traininga machine learning transform predictor (106) for block based encoding, and a machine learning transform predictor (108) for block based decoding, together so as to have a mutually shared training stage (302);switching between:a training objective (3160, 3161, 3163) depending onthe machine learning transform predictor (106) for the block based encoding,the machine learning transform predictor (108) for the block based decoding, andthe machine learning prediction predictor (110), the dequantization function (500) and the bitrate cost model (600), so as to update parameters of the corresponding machine learning predictors (106, 108) in an epoch (3040, 3041, 3043) of the mutually shared training stage (302); anda training objective (3162, 3164) depending onfiling version, rma rate-distortion-optimized quantization (700),the machine learning prediction predictor (110), and the machine learning transform predictor (108) for the block based decoding,so as to update parameters of the machine learning transform predictor (108) for the block based decoding in a subsequent epoch (3042, 3044) of the mutually shared training stage (302) while leaving the parameters of the machine learning transform predictor (106) for block based encoding and the machine learning prediction predictor (110) unchanged.
137. Method of training a machine learning predictor of any one of the previous claims 126 to 136, further comprising:providing a plurality of initial sets of parameters of one or more machine learning predictors as a plurality of initial tuples;subjecting the plurality of initial tuples to a mutually shared training stage of the one or more machine learning predictors for obtaining updated sets of parameters as an updated tuple,with, in an epoch of the mutually shared training stage,computing a training error for each of the plurality of initial tuples according to a training objective depending on:the dequantization function, andthe bitrate cost model, and / orone or more of the one or more machine learning predictors, updating a selected tuple of the plurality of initial tuples so as to obtain a subsequent set of parameters as a subsequent tuple while disregarding the rest of the plurality of initial tuples from the mutually shared training stage,wherein the selected tuple is selected among the plurality of initial tuples on a basis of its corresponding training error being lowest in value.outputting the subsequent set of parameters as inputs for a subsequent epoch of the training stage.filing version, rm138. Method of training a machine learning predictor of any previous claims 126 to 137, further comprising:subjecting the transform block to a parametrization based on a skew symmetric matrix and the matrix exponentiation operation for obtaining orthogonality of the transform block.
139. Method of training a machine learning predictor of any previous claims 126 to 138, further comprising:initializing the transform block with an initial transform, wherein the initial transform is a DCT or DST a LFNST transform or a NSPT transform or an identity transform.
140. Method of training a machine learning predictor of any previous claims 126 to 139, wherein the density function is obtained bycomputing a bitrate error function based on a difference between a cross entropy of the density function associated with the transform block, anda true bitrate cost depending on a code length of the last scan index.
141. Method of training a machine learning predictor of any previous claims 126 to 140, further comprising:initializing the predicted sample filling with an initial predicted sample filling using a PLANAR intra-prediction mode.
142. Method for block based decoding of a predetermined block of a picture, wherein the method comprises:predicting (14) a predicted sample filling of the predetermined block;decoding a synthesis transform block (32) of a prediction residual for the predetermined block from the data stream (40);reconstructing the predetermined block bysubjecting the synthesis transform block (32) to a machine learning transform predictor (34), which is configured to transform, or perform a transform transition of, the synthesis transform block (32) from a transform domain to a sample domain, to obtain a sample domain prediction residual;filing version, rmcorrecting the predicted sample filling using the sample domain prediction residual.
143. Method for block based encoding of a predetermined block of a picture, wherein the method comprises:predicting (14) a predicted sample filling of the predetermined block;encoding an analysis transform block (20) of a prediction residual for the predetermined block into the data stream, wherein the analysis transform block (20) is for reconstructing the predetermined block bysubjecting a synthesis transform block (32) to a machine learning transform predictor (34) which is configured to transform, or perform a transform transition of, the synthesis transform block (32) from a synthesis transform domain to a sample domain to obtain a sample domain prediction residual;correcting the predicted sample filling using the sample domain prediction residual.
144. Method for block based encoding of a predetermined block of a picture, wherein the method comprises:predicting (14) a predicted sample filling of the predetermined block;encoding an analysis transform block (20) of a prediction residual for the predetermined block into the data stream (40) byencoding transform coefficients of the analysis transform block (20) of the prediction residual bysubjecting the analysis transform block (20), accompanied with the predicted sample filling, to a machine learning reductive offset predictor (21) so as to obtain a reductive offset block;correcting each transform coefficient of the analysis transform block (20) based on a corresponding absolute value of the reductive offset block and the sign of the transform coefficient to obtain a corresponding corrected transform coefficient of a modified transform block (22) lower in magnitude, andencoding the modified transform block (22) into the data stream (40), as part of a R / D optimizer and / or a prediction loop of the encoder:filing version, rmsubjecting the modified transform block (22), accompanied with the predicted sample filling, to a machine learning improvement predictor (30) so as to obtain an additive offset block;forming a sum of the modified transform block (22) and the additive offset block (30) to obtain an improved transform block (23);subjecting the improved transform block to a synthesis transform block (32); and correcting the predicted sample filling using the prediction residual.
145. A data stream signalling a picture (14), wherein the picture is encoded into the data stream using the method of any one of claims 143 or 144.
146. A computer program for implementing the method according to any one of claims 126, 142, 143 or 144.
147. An apparatus comprising a machine learning predictor, an encoder and / or a decoder, wherein the machine learning predictor has been trained using the methods according to any one of the previous claims 126 to 141.filing version, rm
