Autoencoder and method for adaptive learned image compression with configurable encoder and decoder for variable bitrate applications

Abstract

An autoencoder and a method for adaptive learned image compression with configurable encoder and decoder for variable bitrate applications are described, wherein the encoder comprises: - a pre-trained transformer-based architecture operable to convert input image data into quantized latent representations (9); - one or more low-rank adapter modules (W a,W b) integrated within said encoder (g a), each adapter module (W a,W b) being configured to adjust the encoding process for different target bitrates while maintaining pre-trained model parameters; - wherein said one or more low-rank adapter modules (W a,W b) are incorporated into fully connected layers (120,121) of the encoder's multi-layer perception modules (MLP), enabling said encoder (g a) to achieve efficient rate-distortion performance across a range of bitrates by fine-tuning only parameters of said adapter modules (W a,W b); - a mechanism adapted to merge said adapter modules (W a,W b) with pre-trained model weights following adaptation, thereby restoring the encoder's complexity to its original level.

Description

[0001] AUTOENCODER AND METHOD FOR ADAPTIVE LEARNED IMAGE COMPRESSION WITH CONFIGURABLE ENCODER AND DECODER FOR VARIABLE BITRATE APPLICATIONS

[0002] DESCRIPTION

[0003] The present invention relates to an autoencoder and a method for adaptive learnable image compression. More in particular, the present invention relates to an autoencoder and a method for adaptive learnable image compression with configurable encoder and decoder for variable bitrate applications.

[0004] Learned Image Compression (LIC) has recently sparked a lot of interest due to its competitive performance compared to traditional approaches.

[0005] LIC models are typically organized as variational autoencoders (see D. P. Kingma and M. Welling, "Auto-encoding variational bayes", arXiv preprint arXiv:1312.6114, 2013, reference [1]; S. Zhang, L. Wang, X. Mao, F. Yang, and S. Wan, "Rate controllable learned image compression based on rfl model", 2022 IEEE International Conference on Visual Communications and Image Processing (VCIP), 2022, reference [2]; X. Pan, D. Ding, L. Wang, X. Xu, and S. Liu, "Low-complexity transform network architecture for jpeg ai image codec", 2023 IEEE International Conference on Visual Communications and Image Processing (VCIP), 2023, reference [3]), where an image x is projected onto a latent space y by an encoder; then, this is quantized, entropy-coded, and decompressed by a decoder recovering a more or less faithful reconstruction of the original image. Such models are trained end-to-end, backpropagating the gradient of some rate-distorsion (RD) cost function that is proportional to the image distortion and the rate.

[0006] LIC models outperformed standardized codecs like JPEG (see G. K. Wallace, "The jpeg still picture compression standard", Communications of the ACM, vol. 34, no. 4, 1991, reference [4]), JPEG2000 (see M. Rabbani and R. Joshi, "An overview of the jpeg 2000 still image compression standard", Signal processing: Image communication, vol. 17, no. 1, 2002, reference [5]), and BPG (see F. Bellard, "Bpg image format", http: / / bellard.org / bpg / ", 2017, reference [6]) in terms of RD performance (see Z. Cheng, H. Sun, M. Takeuchi, and J. Katto, "Learned image compression with discretized gaussian mixture likelihoods and attention modules", 2020 IEEE / CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, Computer Vision Foundation / IEEE, 2020, reference [7]; and Y. Qian, M. Lin, X. Sun, Z. Tan, and R. Jin, "Entroformer: A transformer-based entropy model for learned image compression", arXiv preprint arXiv:2202.05492, 2022, reference [8]).

[0007] Despite remarkable performance in terms of RD trade-off, LIC models show significant shortcomings in adapting the coding bitrate. Indeed, meeting multiple bitrates requires storing multiple encoders and decoders: one per different bitrate. The associated complexity is a major drawback compared to traditional standardized codecs, where changing the quality point introduces no memory overhead. This motivates recent interest by the community in smart variable-rate LIC models.

[0008] Some seminal works (see G. Toderici, S. M. O'Malley, S. J. Hwang, D. Vincent, D. Minnen, S. Baluja, M. Covell, and R. Sukthankar, "Variable rate image compression with recurrent neural networks", arXiv preprint arXiv:1511.06085, 2015, reference [9]; G. Toderici, D. Vincent, N. Johnston, S. Jin Hwang, D. Minnen, J. Shor, and M. Covell, "Full resolution image compression with recurrent neural networks", Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2017, reference

[0010] ) pioneered variable-rate LIC using convolutional LSTM (Long Short-Term Memory) networks, allowing progressive bit transmission where image quality improves with increased bits.

[0009] Y. Choi, M. El-Khamy, and J. Lee, "Variable rate deep image compression with a conditional autoencoder", Proceedings of the IEEE / CVF International Conference on Computer Vision, 2019, reference

[0011] , integrated both fully connected networks into convolution layers and mixed-bin size to control quantization loss across bitrates.

[0010] Similarly, F. Yang, L. Herranz, J. Van De Weijer, J. A. I. Guitian, A. M. Lopez, and M. G. Mozerov, "Variable rate deep image compression with modulated autoencoder", IEEE Signal Processing Letters, vol. 27, 2020, reference

[0012] , introduced a modulated autoencoder for rate adaptation at discrete points along the RD curve, while J. Lee, S. Jeong, and M. Kim, "Selective compression learning of latent representations for variable-rate image compression", Advances in Neural Information Processing Systems, vol. 35, 2022, reference

[0013] , proposed a method that adapts the quantization step and introduced a learnable masking system to mask the latent representation, achieving different rates. However, all of these works increased remarkably both memory and computational requirements.

[0011] Following a different approach, T. Chen and Z. Ma, "Variable bitrate image compression with quality scaling factors", IEEE International Conference on Acoustics, Speech and Signal Processing, 2020, reference

[0014] , introduced some scaling factors directly applied to the latent representation, while Z. Cui, J. Wang, S. Gao, T. Guo, Y. Feng, and B. Bai, "Asymmetric gained deep image compression with continuous rate adaptation", Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2021, reference

[0015] , added two learnable modules before and after quantization, called gain units, to adapt the latent representation for different bitrates. One big advantage of this approach is that the entire architecture is trained end-to-end with only one step, yet with an increasing training complexity and lowering the quality of results.

[0012] Recently, Parameter-Efficient Training (PET) methods have been developed to help large pre-trained models adapt to new tasks with minimal additional parameters. Adapters, i.e. small modules inserted into the pre-trained model (see E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, "Lora: Low-rank adaptation of large language models", International Conference on Learning Representations, 2021, reference

[0016] ; M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim, "Visual prompt tuning", European Conference on Computer Vision, Springer, 2022, reference

[0017] and J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, and G. Neubig, "Towards a unified view of parameter-efficient transfer learning", 2021, reference

[0018] ) are extremely popular both in Natural Language Processing (see N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly, "Parameter-efficient transfer learning for nip", International conference on machine learning. PMLR, 2019, reference

[0019] , and N. Mundra, S. Doddapaneni, R. Dabre, A. Kunchukuttan, R. Puduppully, and M. M. Khapra, "A comprehensive analysis of adapter efficiency", Proceedings of the 7th Joint International Conference on Data Science & Management of Data (11th ACM IKDD CODS and 29th COMAD), 2024, reference

[0020] ) and Computer Vision (see P. Gao, S. Geng, R. Zhang, T. Ma, R. Fang, Y. Zhang, H. Li, and Y. Qiao, "Clip-adapter: Better vision-language models with feature adapters", International Journal of Computer Vision, vol. 132, no. 2, 2024, reference

[0021] and I. E. Marouf, E. Tartaglione, and S. Lathuiliere, "Mini but mighty: Finetuning vits with mini adapters", Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision, 2024, reference

[0022] ) for their efficiency in adjusting data representations to new tasks, offering performance comparable to full fine-tuning but with far fewer trainable parameters.

[0013] Adapters typically consist of two fully connected layers, a non-linear activation function between them (see J. He, C. Zhou, X. Ma, T. Berg-Kirkpatrick, and G. Neubig, "Towards a unified view of parameter-efficient transfer learning", International Conference on Learning Representations, 2022, reference

[0023] ) and a multiplicative scaling factor for its output.

[0014] The inclusion of the non-linearity in adapter modules for PET methods is currently an object of discussion by the community (see V. Fomenko, H. Yu, J. Lee, S. Hsieh, and W. Chen, "A note on lora," arXiv preprint arXiv:2404.05086, 2024, reference

[0024] ).

[0015] BACKGROUND ON LIC

[0016] Figure 1a illustrates a LIC autoencoder architecture 1 by R. Zou, C. Song, and Z. Zhang, "The devil is in the details: Window-based attention for image compression", Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2022, pp. 17492-17501, reference

[0025] , that will be used as a reference in the rest of the description.

[0017] An image x is input into a learnable encoder gathat produces a (low-dimensional) latent representation y, further quantized as y. Each element ŷiis modeled as a single Gaussian distribution with standard deviation σᵢ and mean μᵢ estimated by the hyperprior (see J. Balle, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston, "Variational image compression with a scale hyperprior", arXiv preprint arXiv:1802.01436, 2018, reference

[0029] ), and the channelwise entropy model (see D. Minnen and S. Singh, "Channel-wise autoregressive entropy models for learned image compression", 2020 IEEE International Conference on Image Processing (ICIP), IEEE, 2020, reference

[0030] ). The entire entropy parameter estimation system is indicated with the reference number 10 in Figure 1a.

[0018] In foundational works like J. Balle, V. Laparra, and E. P. Simoncelli, "End-to-end optimized image compression", arXiv preprint arXiv:1611.01704, 2016, reference

[0031] , the latent representation is modeled as a fully factorized distribution, where the rate is either computed analytically (see A. Presta, A. Fiandrotti, E. Tartaglione, and M. Grangetto, "A differentiable entropy model for learned image compression", International Conference on Image Analysis and Processing, Springer, 2023, reference

[0032] ), or through an auxiliary neural network. To capture spatial correlation within the image, reference

[0029] introduced a scale hyperprior autoencoder built upon the main architecture, aiming to extract the entropy parameters. Since the latent representation, in this case, is modeled using Gaussian priors, the hyperprior autoencoder extracts the entropy parameters as standard deviations σᵢ and means μᵢ.

[0019] Accurate rate estimation has been the goal of recent research including the use of a local context model based on masked convolution for improved (see D. Minnen, J. Balle, and G. D. Toderici, "Joint autoregressive and hierarchical priors for learned image compression", Advances in neural information processing systems, vol. 31, 2018, reference

[0033] ), and channel-wise contexts to reduce computational time (see reference

[0030] ). Recent works have employed both non-local and local window-based attention mechanisms (see reference [7] and reference

[0025] ), as well as hybrid models combining CNNs and transformers (see reference [8] and J. Liu, H. Sun, and J. Katto, "Learned image compression with mixed transformer-cnn architectures", Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2023, reference

[0034] ).

[0020] The quantized representation y is finally entropy-coded as a compressed bitstream that is sent to the receiver where it is entropy-decoded and a learnable decoder gsrecovers a proxy x of the original image. The encoding-decoding process can be formulated as:

[0021]

[0022] Such LIC models are trained to minimize the cost function:

[0023]

[0024] where R is an estimator of the entropy of the compressed bitstream and D is some end-to-end pixel distortion metric such as MSE (Mean Squared Error) or SSIM (Structural Similarity Index).

[0025] The hyperparameter X in formula (2) controls the tradeoff between rate and distortion, and to attain a different tradeoff the whole model must be retrained, i.e., a different model needs to be trained for each target bitrate.

[0026] The present invention aims at solving these and other problems by providing an autoencoder and a method for adaptive learned image compression with configurable encoder and decoder for variable bitrate applications that adapt a single pre-trained model to different bitrates, eliminating the need to store multiple models.

[0027] It is a further scope of the present invention to provide an autoencoder and a method for adaptive learned image compression with configurable encoder and decoder for variable bitrate applications that adapt a single pre-trained model to different bitrates that reduce both the training time and computational resources required, lowering overall costs.

[0028] It is a further scope of the present invention to provide an autoencoder and a method for adaptive learned image compression with configurable encoder and decoder for variable bitrate applications that maintains comparable rate-distortion performance to state-of-the-art models while achieving variable bitrate adaptation, thus balancing quality and efficiency.

[0029] It is a further scope of the present invention to provide an autoencoder and a method for adaptive learned image compression with configurable encoder and decoder for variable bitrate applications that allow the merging of adapters with the main model, restoring complexity and inference time to levels comparable to fixed-rate models.

[0030] It is a further scope of the present invention to provide an autoencoder and a method for adaptive learned image compression with configurable encoder and decoder for variable bitrate applications that address the limitations of existing LIC methods in terms of memory usage, computational cost, and performance consistency.

[0031] In a nutshell, the present invention exploits the idea of efficiently adapting one model trained at a specific quality point and adapting it to more bitrates by introducing lightweight adapters for transformer-based LIC models.

[0032] Instead of training separate models for each specific bitrate, adapters can be built upon a pre-trained module and adaptively fine-tuned for the required bit rate, achieving comparable RD performance to state-of-the-art models.

[0033] In the framework according to the present invention, adapters are linear operators acting in parallel to each fully connected layer of the Multi-Layer Perception (MLP) module of the Symmetrical Transformer Framework (STF) (see reference

[0025] ). After tuning the adapter for a new rate, it is applied a merging system that integrates it with the main linear module, reducing the inference time and making the variable rate models according to the invention as fast as fixed-rate ones.

[0034] The main contributions of the present invention may be summarized as follows: • although adapters have been already used in LIC (see K. Tsubota, H. Akutsu, and K. Aizawa, "Universal deep image compression via content-adaptive optimization with adapters", Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision, 2023, reference

[0026] ; A. Presta, G. Spadaro, E. Tartaglione, A. Fiandrotti, and M. Grangetto, "Domain adaptation for learned image compression with supervised adapters", 2024 Data Compression Conference (DCC), IEEE, 2024, reference

[0027] ; J. Liu, X. Jin, R. Feng, Z. Chen, and W. Zeng, "Composable image coding for machine via task-oriented internal adaptor and external prior", 2023 IEEE International Conference on Visual Communications and Image Processing (VCIP), 2023, reference

[0028] ), to the best of the Applicants' knowledge, performing rate adaptation from a unique LIC model through adapters is innovative. This approach enables massive memory savings;

[0035] • it is shown how to merge the obtained adapters with the original model weights, restoring the complexity to the same level as the original model without any loss in RD performance;

[0036] • the empirical evaluations in standard image compression setups show that model adaptation is very satisfactory in terms of quality-memory trade-off.

[0037] According to a first aspect of the invention, it is described an encoder for a learned image compression model configured to adapt to variable bitrates, comprising: a pre-trained transformer-based architecture operable to convert input image data into quantized latent representations; one or more low-rank adapter modules integrated within the encoder, each adapter module being configured to adjust the encoding process for different target bitrates while maintaining pre-trained model parameters; wherein the one or more low-rank adapter modules are incorporated into fully connected layers of the encoder's multi-layer perception modules, enabling the encoder to achieve efficient rate-distortion performance across a range of bitrates by fine-tuning only parameters of the adapter modules; a mechanism adapted to merge the adapter modules with pre-trained model weights following adaptation, thereby restoring the encoder' s complexity to its original level.

[0038] According to an embodiment of the encoder according to the invention, the transformer-based architecture includes a multi-head self-attention module configured to capture dependencies across the input image data, thereby enhancing the representation quality of the quantized latent representations.

[0039] According to a further embodiment of the encoder according to the invention, each one or more low-rank adapter modules comprises a pair of rank decomposition matrices and a scaling factor, the scaling factor being adjustable to control the magnitude of changes introduced by the adapter modules during the encoding process.

[0040] According to a further embodiment of the encoder according to the invention, the encoder further comprises one or more layer normalization units to stabilize the output and ensure consistent scaling of data prior to quantization.

[0041] According to a further embodiment of the encoder according to the invention, the mechanism for merging the adapter modules with the pre-trained model weights is configured to include a process for reducing inference time and computational complexity of the encoder post-adaptation, thereby optimizing the encoder's performance efficiency.

[0042] According to a second aspect of the invention, it is described a decoder for a learned image compression model configured to adapt to variable bitrates, comprising: a pre-trained transformer-based architecture operable to process quantized latent representations of image data and reconstruct the image; one or more low-rank adapter modules integrated within the decoder, each adapter module being configured to adjust the decoding process for different target bitrates while maintaining pre-trained model parameters; wherein the one or more low-rank adapter modules are incorporated into fully connected layers of the decoder's multilayer perception modules, enabling the decoder to sustain efficient rate-distortion performance across a range of bitrates by fine-tuning only parameters of the adapter modules; a mechanism adapted to merge the adapter modules with pre-trained model weights following adaptation, thereby restoring the decoder's complexity to its original level.

[0043] According to an embodiment of the decoder according to the invention, the transformer-based architecture includes a multi-head self-attention module configured to capture dependencies across the quantized latent representations, thereby enhancing the accuracy of a reconstructed image.

[0044] According to a further embodiment of the decoder according to the invention, each one or more low-rank adapter modules comprises a pair of rank decomposition matrices and a scaling factor, the scaling factor being adjustable to control the magnitude of changes introduced by the adapter modules during the decoding process.

[0045] According to a further embodiment of the decoder according to the invention, the decoder further comprises one or more layer normalization units to stabilize the output and ensure consistent scaling of data prior to image reconstruction.

[0046] According to a further embodiment of the decoder according to the invention, the mechanism for merging the adapter modules with the pre-trained model weights is configured to include a process for reducing inference time and computational complexity of the decoder post-adaptation, thereby optimizing the decoder's performance efficiency.

[0047] According to a further aspect of the invention, it is described an autoencoder for learnable image compression comprising an encoder according to the invention and a decoder according to the invention.

[0048] According to a further aspect of the invention, it is described a method for adapting an encoder in a learned image compression model to variable bitrates, comprising the steps of: providing a pre-trained transformer-based architecture configured to convert input image data into quantized latent representations; integrating one or more low-rank adapter modules within the encoder, each adapter module being operable to adjust the encoding process for different target bitrates while keeping pre-trained model parameters fixed; incorporating the adapter modules into fully connected layers of encoder's multi-layer perception modules; fine-tuning only the parameters of the adapter modules to achieve efficient rate-distortion performance across a range of bitrates; merging the adapter modules with pre-trained model weights after adaptation to restore the encoder's complexity to its original level. According to a further aspect of the invention, it is described a method for adapting a decoder in a learned image compression model to variable bitrates, comprising the steps of: providing a pre-trained transformer-based architecture configured to process quantized latent representations of image data and reconstruct the image; integrating one or more low-rank adapter modules within the decoder, each adapter module being operable to adjust the decoding process for different target bitrates while maintaining pre-trained model parameters fixed; incorporating the adapter modules into fully connected layers of decoder's multi-layer perception modules; fine-tuning only the parameters of the adapter modules to achieve efficient ratedistortion performance across a range of bitrates; merging the adapter modules with pre-trained model weights after adaptation to restore the decoder's complexity to its original level.

[0049] According to a further aspect of the invention, it is described a method for adapting a learned image compression model to variable bitrates, comprising: utilizing a pretrained transformer-based learned image compression model; incorporating low-rank adapter modules into the model, wherein the adapter modules are configured to adjust the model for different target bitrates without requiring retraining of the entire model; maintaining pre-trained model weights as fixed and only training the adapter modules to achieve the desired bitrate adaptation.

[0050] The invention will be described in detail hereinafter through non-limiting embodiments with reference to the attached Figures, wherein:

[0051] Figure la shows a general architecture of a prior art autoencoder for learned image compression (LIC) utilizing a hyperprior and a channel-wise autoregressive context model;

[0052] Figure lb shows an architecture of a transformer block of the architecture of Figure la;

[0053] Figure 1c shows a Multi-Layer Perception module of Figure lb incorporating low-rank adapters (LoRA) according to the invention;

[0054] Figure 2 represents a bar graph showing the memory consumption of the ensemble of Symmetrical Transformer Framework models to cover the entire RD curve versus Symmetrical Transformer Framework adopting the method according to the present invention;

[0055] Figures 3a and 3b show rate distortion plots for the Kodak and the CLIC dataset, respectively, wherein an anchor point represents the result of an anchor model for a Symmetrical Transformer Framework adopting the method according to the present invention;

[0056] Figure 4 represents a table showing average per image encoding and decoding times, memory usage, and FLOPs on different devices, wherein the results are in reference to a single model;

[0057] Figure 5 shows a rate distortion performance comparison between the method according to the present invention versus naive fine-tuning of the fully connected layers in the Multi-Layer Perceptions.

[0058] Low-Rank Adapter

[0059] It is now described ALICE, an autoencoder for variable bitrate LIC according to the invention, and the related training procedure. ALICE stands for "Adapt your Learnable Image Compression modEl for variable bitrates".

[0060] The present invention moves from the observation that both encoder gaand decoder gsin Figure la are implemented as a sequence of Transformer blocks.

[0061] Each Transformer block is in turn implemented as a Multi-head Self- Attention module (MSA) and an MLP (Multi-Layer Perception) module, where each MLP is implemented as two fully connected layers, as represented in Figure lb.

[0062] The transformer-based topology of Figure la is leveraged to add variable bitrate coding to a single trained model by fine-tuning the auxiliary adapter modules of Figure 1c to meet different bitrates.

[0063] To adapt the pre-trained model for a different bit rate, adapter modules are plugged in both the encoder gaand the decoder gs. Specifically, the pre-trained model weights are frozen and trainable rank decomposition matrices are plugged into each fully connected layer of each MLP (see reference

[0016] ).

[0064] More formally, let h = W z be the output of the original fully connected layer where W0 ∈ Rfin×fout(and fin,fout are the fan-in and the fan-out of the fully connected layer, respectively) linearly transform the input z: the modified forward pass yields h = W0z + α(ΔWz) = W0z + α(WbWaz), where the parameters in Wa∈ Rfin×rare initialized to zero, and those belonging to Wb∈ Rr×foutare initialized sampling from N (0, 1). This evidently results in a AW equal to zero at the beginning of the adaptation phase. The value r is a hyperparameter used to specify the rank of the low-rank matrices, while a is a scaling factor used to determine the magnitude of the changes introduced by the adapter layer.

[0065] It is important to highlight that, as long as r <

[0066]

[0067] Waand Wb are learnable matrices having lower rank than the original weight matrix Wo.

[0068] For this reason, fine-tuning Waand instead of Wo directly, enables a more parameter-efficient adaptation, controlled by the hyper-parameter r.

[0069] More in detail, the transformer block 100, as depicted in Figure lb and Figure 1c, comprises the following blocks:

[0070] a first Layer Normalization (LN) unit 102 receiving input data 101;

[0071] a Window-based Multi-head Self-Attention Module (W-MSA) 103: this module is responsible for capturing dependencies across different parts of the input data by attending to multiple positions simultaneously. The multi-head mechanism allows the model to focus on various aspects of the input data, thereby improving the representation quality;

[0072] a first sum node 104 that sums the output of the W-MSA module 103 with the input data 101;

[0073] a second Layer Normalization (LN) unit 105 receiving the output of the first sum node 104;

[0074] a first Multi-Layer Perception (MLP) Module 106;

[0075] a second sum node 107 that sums the output of the first MLP module 106 with the output of the first sum node 104;

[0076] a third Layer Normalization (LN) unit 108 receiving the output of the second sum node 107:

[0077] a Shifted Window-based Multi-head Self-Attention Module (SW-MSA) 109: the shifted window approach is an extension of the window-based method, in which the windows are shifted with respect to the input tensor. This technique allows the model to capture cross-window dependencies and enhances the ability to model global relationships over multiple layers, improving the model's expressiveness and performance.

[0078] a third sum node 110 that sums the output of the SW-MSA module 109 with the output of the second sum node 107; a fourth Layer Normalization (LN) unit 111 receiving the output of the third sum node 110;

[0079] a second Multi-Layer Perception (MLP) Module 112;

[0080] a fourth sum node 113 that sums the output of the second MLP Module 112 with the output of the third sum node 110.

[0081] The first 102, second 105, third 108 and fourth 111 Layer Normalization units are used to stabilize the output of the respective units to which they are connected and ensure consistent scaling of data prior to quantization.

[0082] The first and the second Multi-Layer Perception (MLP) Module 106,112 comprises each two fully connected layers 120,121 with an intermediate non-linear activation function 122. They process the output from the W-MSA module 103 and the SW-MSA module 109, respectively, transforming it into a form suitable for subsequent layers.

[0083] A Non-Linear Activation Function (GeLU) 122 is positioned between the two fully connected layers 120,121 of the MLP module 106,112. The Gaussian Error Linear Unit (GeLU) activation function introduces non-linearity to the model, enabling it to learn more complex patterns and features from the data.

[0084] Fine-tuning Process

[0085] Following S. Chen, C. Ge, Z. Tong, J. Wang, Y. Song, J. Wang, and P. Luo, "AdaptFormer: Adapting vision transformers for scalable visual recognition", NeurlPS, 2022, reference

[0035] , adapters Wa, Wbare plugged to each fully connected layer 120,121 of the MLPs 106,112 in both the encoder gaand decoder gs. Starting from the pre-trained STF model at a high bit rate that will be referred to as anchor in the following, only the adapter parameters are fine-tuned, keeping the anchor model parameters frozen. Here the goal is to modify the latent space to filter-out some information to compress at a lower bit-rate. For this reason, starting from the anchor the adapters Wa, Wbfor a lower λ, i.e. for a rate lower than that the anchor model was trained for, are plugged and fine-tuned. The procedure is repeated for each target rate, i.e., five times in the Applicants' experiments below for a total of six different RD tradeoffs, including the anchor, which represents the starting point.

[0086] As can be seen in Figure 2, this procedure saves a lot of memory space. In fact, to cover the whole rate-distortion curve of the experiments below, it is only needed to save the anchor model and the fine-tuned adapters, instead of saving six whole models (i.e., one for each RD tradeoff).

[0087] Merging ALICE: MALICE

[0088] As described above, the method according to the present invention allows to effectively control the bit rate of the compressed images by simply plugging the set of adapters corresponding to a specific bit rate. However, these extra modules slightly increase the complexity of the model, impacting the inference time.

[0089] Instead of plugging and training extra modules, one could directly fine-tune the fully-connected layers, without thus changing the complexity of the model and obtaining the same results (i.e., plug the fine-tuned fully-connected layers to change the rate). However, as previously described, this would require training and storing more parameters in memory.

[0090] For this reason, since there is no non-linearity in the adapters used in the method according to the present invention, it is allowed to merge the adapters with the fully connected layer after the fine-tuning achieving the same compression results. From formula (3) it can be simply rewritten

[0091]

[0092] where

[0093]

[0094] By using directly, it is recovered the same (space and time) complexity as the starting model, but having trained fewer parameters than fine-tuning the fully-connected layers.

[0095] Experimental Setup

[0096] As far as training is concerned, the method according to the invention is implemented by leveraging the original repository of STF (see reference

[0025] ) and the CompressAI platform (see J. Begaint, F. Racape, S. Feltman, and A. Pushparaja, "Compressai: a pytorch library and evaluation platform for end-to-end compression research", arXiv preprint arXiv:2011.03029, 2020, reference

[0036] ).

[0097] It was selected the highest bit-rate pre-trained model from the STF repository as the anchor, which was trained on X = 0.0483.

[0098] It was then trained the adapter module for 280k iterations, using a batch size of 16 (on X = [0.0018, 0.0035, 0.0067, 0.0130, 0.025]), instead of the 1.8M iterations required to train the anchor.

[0099] It is employed a cosine annealing learning rate schedule, starting from 0.001.

[0100] The models according to the invention are optimized using the MSE quality metric, adhering to settings similar to those in STF (see reference

[0025] ).

[0101] The method according to the invention is evaluated by calculating the average RD performance using PSNR to gauge the distortion on the commonly used Kodak image set (see E. Kodak, "Kodak lossless true color image suite (photocd pcd0992)", 1993, http: / / r0k.us / graphics / kodak / , reference

[0037] ) and the CLIC validation dataset (see G. Toderici, W. Shi, R. Timofte, L. Theis, J. Balle, E. Agustsson, N. Johnston, and F. Mentzer, "Workshop and challenge on learned image compression (clic2020)", CVPR, 2020, reference

[0038] ).

[0102] The results obtained through the method according to the invention are compared with the original STF framework, other reference LIC methods (see references [7],

[0029] and

[0033] ) and VCC-VTM 23.3 (see B. Bross et al., "Overview of the versatile video coding (WC) standard and its applications", IEEE TCSVT, 2021, reference

[0039] ).

[0103] Rate-Distortion Performance

[0104] Figure 3a and Figure 3b illustrate RD performance for reference methods and for the method according to the present invention on the Kodak and CLIC datasets. It is recalled that for the method according to the present invention the highest rate point on the RD curve corresponds to the pre-trained anchor model, while the other points to the model adapted at lower bitrates.

[0105] For both datasets, the method according to the present invention demonstrates similar trends, successfully varying the bit-rate. The slight performance degradation at lower bit rates can be attributed to the masking ability of self-attention modules in the STF framework. Self-attention blocks in LIC are crucial for filtering out information at lower bitrates, as noted in reference

[0025] .

[0106] Since the adapters according to the invention operate only within the MLP modules of STF, this performance drop is expected. However, the adaptability of the method according to the present invention offers a significant tradeoff compared to the slight drop in performance making it a highly practical solution for variable rate.

[0107] Ablation Studies

[0108] Adapters vs Fully-Connected Layers Finetuning. Next, it is compared the adapterbased method according to the invention against a naive adapter-less baseline. Namely, the Fully-Connected layers are fine-tuned in the Multi-Layer Perceptions of Symmetrical Transformer Framework of all the transformer blocks to obtain different RD trade-offs.

[0109] Figure 4 shows comparable RD performance albeit fine-tuning the adapters involves training about 350k parameters only, whereas fine-tuning the Fully-Connected layers amounts to training over 8.5M parameters, representing a ~ 24.29x reduction.

[0110] Memory and Time Complexity. One major advantage of the method according to the present invention is that, rather than storing multiple models trained for different bit rates, only a single highest bit rate model needs to be stored on the edge device. For other bit rates, only the adapters need to be stored. Alternatively, if a single unified model is preferred, it is possible to adopt MALICE where the adapter is merged with the FC layer of the STF model.

[0111] The time complexity of the method according to the invention, both with MALICE and without ALICE merged Fully-Connected layers, compared with the baseline STF, is provided in Figure 4.

[0112] For both GPU and CPU cases, the addition of the adapter module slightly increases the encoding and decoding time compared to the baseline, as well as the memory occupancy and FLOPs. However, when merged with the FC of the STF module, the time complexity falls back to that of the baseline model.

[0113] To sum up, in the present invention it is proposed to obtain multiple quality points given one anchor only, by employing lightweight adapters. By leveraging PET methods, it is shown that it is possible to achieve rate adaptation without training separate models for each bitrate. The method according to the present invention involves plugging small adapter modules into the transformer-based LIC models, fine-tuned to meet the required bitrate. These adapters, thanks to the lack of nonlinearities, can be effectively merged with the model, leading as well to consistent memory savings.

[0114] Finally, the Applicants' experiments show that performance comparable with state-of-the-art fixed rate LIC models, yet at a fraction of the training and deployment cost.

[0115] The present description has tackled some of the possible variants, but it will be apparent to the man skilled in the art that other embodiments may also be implemented, wherein some elements may be replaced with other technically equivalent elements. The present invention is not therefore limited to the explanatory examples described herein, but may be subject to many modifications, improvements or replacements of equivalents parts and elements without departing from the basic inventive idea, as set out in the following claims.

Claims

1. CLAIMS1. Encoder (ga) for a learned image compression model configured to adapt to variable bitrates, comprising:3.a pre-trained transformer-based architecture operable to convert input image data into quantized latent representations (y);4.one or more low-rank adapter modules (Wa,Wb) integrated within said encoder (ga), each adapter module (Wa,Wb) being configured to adjust the encoding process for different target bitrates while maintaining pre-trained model parameters; wherein said one or more low-rank adapter modules (Wa,Wb) are incorporated into fully connected layers (120,121) of the encoder's multi-layer perception modules (MLP), enabling said encoder (ga) to achieve efficient rate-distortion performance across a range of bitrates by fine-tuning only parameters of said adapter modules (Wa,Wb); a mechanism adapted to merge said adapter modules (Wa,Wb) with pre-trained model weights following adaptation, thereby restoring the encoder's complexity to its original level.

2. Encoder (ga) according to claim 1, wherein said transformer-based architecture includes a multi-head self-attention module (103,109) configured to capture dependencies across the input image data, thereby enhancing the representation quality of the quantized latent representations (y).

3. Encoder (ga) according to claim 1, wherein each one or more low-rank adapter modules (144,144) comprises a pair of rank decomposition matrices and a scaling factor, the scaling factor being adjustable to control the magnitude of changes introduced by the adapter modules (144,144) during the encoding process.

4. Encoder (ga) according to claim 1, further comprising one or more layer normalization units (102,105,108,111) to stabilize the output and ensure consistent scaling of data prior to quantization.

5. Encoder (ga) according to claim 1, wherein the mechanism for merging the adapter modules (144,144) with said pre-trained model weights is configured to include a process for reducing inference time and computational complexity of theencoder post-adaptation, thereby optimizing the encoder's performance efficiency.

6. A decoder (gs) for a learned image compression model configured to adapt to variable bitrates, comprising:10.a pre-trained transformer-based architecture operable to process quantized latent representations (y) of image data and reconstruct the image (x);11.one or more low-rank adapter modules (Wa, Wb) integrated within said decoder (gs), each adapter module (Wa, Wb) being configured to adjust the decoding process for different target bitrates while maintaining pre-trained model parameters; wherein said one or more low-rank adapter modules (Wa, Wb) are incorporated into fully connected layers (120,121) of the decoder's multi-layer perception modules (MLP), enabling said decoder (gs) to sustain efficient rate-distortion performance across a range of bitrates by fine-tuning only parameters of said adapter modules (Wa, Wb);12.a mechanism adapted to merge said adapter modules (Wa, Wb) with pre-trained model weights following adaptation, thereby restoring the decoder's complexity to its original level.

7. A decoder (gs) according to claim 6, wherein said transformer-based architecture includes a multi-head self-attention module (103,109) configured to capture dependencies across the quantized latent representations (y), thereby enhancing the accuracy of a reconstructed image (x).

8. A decoder (gs) according to claim 6, wherein each one or more low-rank adapter modules (Wa, Wb) comprises a pair of rank decomposition matrices and a scaling factor, the scaling factor being adjustable to control the magnitude of changes introduced by the adapter modules (Wa, Wb) during the decoding process.

9. A decoder (gs) according to claim 6, further comprising one or more layer normalization units (102,105,108,111) to stabilize the output and ensure consistent scaling of data prior to image reconstruction.

10. A decoder (gs) according to claim 6, wherein the mechanism for merging said adapter modules (Wa, Wb) with said pre-trained model weights is configured to include a process for reducing inference time and computational complexity of thedecoder post-adaptation, thereby optimizing the decoder's performance efficiency.

11. An autoencoder (1) for learnable image compression comprising an encoder (go) according to any of the claims 1 to 5 and a decoder (gs) according to any of the claims 6 to 10.

12. A method for adapting an encoder (gs) in a learned image compression model to variable bitrates, comprising the steps of:19.providing a pre-trained transformer-based architecture configured to convert input image data into quantized latent representations (y);20.integrating one or more low-rank adapter modules (Wa, VVb) within the encoder, each adapter module (144,144) being operable to adjust the encoding process for different target bitrates while keeping pre-trained model parameters fixed;21.incorporating the adapter modules (144,144) into fully connected layers (120, 121) of encoder's multi-layer perception modules (MLP);22.fine-tuning only the parameters of the adapter modules (144,144) to achieve efficient rate-distortion performance across a range of bitrates;23.merging the adapter modules (144,144) with pre-trained model weights after adaptation to restore the encoder's complexity to its original level.

13. A method for adapting a decoder (ga) in a learned image compression model to variable bitrates, comprising the steps of:25.providing a pre-trained transformer-based architecture configured to process quantized latent representations (y) of image data and reconstruct the image (x); integrating one or more low-rank adapter modules (144,144) within the decoder (ga), each adapter module (144,144) being operable to adjust the decoding process for different target bitrates while maintaining pre-trained model parameters fixed;26.incorporating said adapter modules (144,144) into fully connected layers (120, 121) of decoder's multi-layer perception modules (MLP);27.fine-tuning only the parameters of said adapter modules (144,144) to achieve efficient rate-distortion performance across a range of bitrates;28.merging said adapter modules (144,144) with pre-trained model weights after adaptation to restore the decoder's complexity to its original level.

14. A method for adapting a learned image compression model to variable bitrates, comprising:30.- utilizing a pre-trained transformer-based learned image compression model; - incorporating low-rank adapter modules (Wa / Wb) into the model, wherein the adapter modules (Wa, Wb) are configured to adjust the model for different target bitrates without requiring retraining of the entire model;31.- maintaining pre-trained model weights as fixed and only training the adapter modules (Wa, Wb) to achieve the desired bitrate adaptation.

Images