Method and data processing system for lossy image or video encoding, transmission and decoding

Abstract

A method for lossy image or video encoding, transmission and decoding, the method comprising receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation; range asymetric numeral system (rANS) encoding the latent representation to produce a bitstream and transmitting the bitstream to a second computer system; at the second computer system, rANS decoding the bitstream to produce the latent representation; and decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image.

Description

[0001] Improved rANS / WO

[0002] METHOD AND DATA PROCESSING SYSTEM FOR LOSSY IMAGE OR VIDEO ENCODING, TRANSMISSION AND DECODING

[0003] CROSS REFERENCE TO RELATED APPLICATIONS

[0004] This application claims priority to United Kingdom Patent Application No. GB 2500183.5, filed January 8, 2025, which is incorporated herein by reference in its entirety. BACKGROUND

[0005] This invention relates to a method and system for lossy image or video encoding, transmission and decoding, a method, apparatus, computer program and computer readable storage medium for lossy image or video encoding and transmission, and a method, apparatus, computer program and computer readable storage medium for lossy image or video receipt and decoding.

[0006] There is increasing demand from users of communications networks for images and video content. Demand is increasing not just for the number of images viewed, and for the playing time of video; demand is also increasing for higher resolution content. This places increasing demand on communications networks and increases their energy use because of the larger amount of data being transmitted.

[0007] To reduce the impact of these issues, image and video content is compressed for transmission across the network. The compression of image and video content can be lossless or lossy compression. In lossless compression, the image or video is compressed such that all of the original information in the content can be recovered on decompression. However, when using lossless compression there is a limit to the reduction in data quantity that can be achieved. In lossy compression, some information is lost from the image or video during the compression process. Known compression techniques attempt to minimise the apparent loss of information by the removal of information that results in changes to the decompressed image or video that is not particularly noticeable to the human visual system. JPEG, JPEG2000, AVC, HEVC and AVI are examples of compression processes for image and / or video files.

[0008] In general terms, known lossy image compression techniques use the spatial correlations between pixels in images to remove redundant information during compression. For example, in an image of a blue sky, if a given pixel is blue, there is a high likelihood that the neighbouring pixels, and their neighbouring pixels, and so on, are also blue. There is accordingly no need to retain all the raw pixel data. Instead, we can retain only a subset of the pixels which take upImproved rANS / WO

[0009] fewer bits and infer the pixel values of the other pixels using information derived from spatial correlations.

[0010] A similar approach is applied in known lossy video compression techniques. That is, spatial correlations between pixels allow the removal of redundant information during compression. However, in video compression, there is further information redundancy in the form of temporal correlations. For example, in a video of an aircraft flying across a blue-sky background, most of the pixels of the blue sky do not change at all between frames of the video. The most of the blue sky pixel data for the frame at position t = 0 in the video is identical to that at position t = 10. Storing this identical, temporally correlated, information is inefficient. Instead, only the blue sky pixel data for a subset of the frames is stored and the rest are inferred from information derived from temporal correlations.

[0011] In the realm of lossy video compression in particular, the removal of redundant temporally correlated information in a video sequence is known as inter-frame redundancy.

[0012] One technique using inter-frame redundancy that is widely used in standard video compression algorithms involves the categorization of video frames into three types: I-frames, P-frames, and B-frames. Each frame type carries distinct properties concerning their encoding and decoding process, playing different roles in achieving high compression ratios while maintaining acceptable visual quality.

[0013] 1-frames, or intra-coded frames, serve as the foundation of the video sequence. These frames are self-contained, each one encoding a complete image without reference to any other frame. In terms of compression, I-frames are least compressed among all frame types, thus carrying the most data. However, their independence provides several benefits, including being the starting point for decompression and enabling random access, crucial for functionalities like fast-forwarding or rewinding the video.

[0014] P-frames, or predictive frames, utilize temporal redundancy in video sequences to achieve greater compression. Instead of encoding an entire image like an I-frame, a P-frame represents the difference between itself and the closest preceding I- or P-frame. The process, known as motion compensation, identifies and encodes only the changes that have occurred, thereby significantly reducing the amount of data transmitted. Nonetheless, P-frames are dependent on previous frames for decoding. Consequently, any error during the encoding or transmission process may propagate to subsequent frames, impacting the overall video quality.

[0015] B-frames, or bidirectionally predictive frames, represent the highest level of compression. Unlike P-frames, B-frames use both the preceding and following frames as references in their encoding process. By predicting motion both forwards and backwards inImproved rANS / WO

[0016] time, B-frames encode only the differences that cannot be accurately anticipated from the previous and next frames, leading to substantial data reduction. Although this bidirectional prediction makes B-frames more complex to generate and decode, it does not propagate decoding errors since they are not used as references for other frames. Artificial intelligence (AI) based compression techniques achieve compression and decompression of images and videos through the use of trained neural networks in the compression and decompression process. Typically, during training of the neutral networks, the difference between the original image and video and the compressed and decompressed image and video is analyzed and the parameters of the neural networks are modified to reduce this difference while minimizing the data required to transmit the content. However, Al based compression methods may achieve poor compression results in terms of the appearance of the compressed image or video or the amount of information required to be transmitted.

[0017] An example of an Al based image compression process comprising a hyper-network is described in Balle, Johannes, et al. " Variational image compression with a scale hyperprior." arXiv preprint arXiv: 1802.01436 (2018), which is incorporated herein by reference in its entirety.

[0018] An example of an Al based video compression approach is shown in Agustsson, E., Minnen, D., Johnston, N., Balle, J., Hwang, S. J., and Toderici, G. (2020), Scale-space flow for end-to-end optimized video compression. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 8503-8512), which is incorporated herein by reference in its entirety.

[0019] A further example of an Al based video compression approach is shown in Mentzer, F., Agustsson, E., Balle, J., Minnen, D., Johnston, N., and Toderici, G. (2022, November). Neural video compression using gans for detail synthesis and propagation. In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXVI (pp. 562-578), which is incorporated herein by reference in its entirety.

[0020] SUMMARY

[0021] According to a first aspect, there is provided a method for lossy image or video encoding, transmission and decoding, the method comprising: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation; range asymetric numeral system (rANS) encoding the latent representation to produce a bitstream and transmitting the bitstream to a second computer system; at the second computer system, rANS decoding the bitstream to produce the latentImproved rANS / WO

[0022] representation; and decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image.

[0023] Optionally, rANS encoding comprises producing a final rANS state, and splitting the final rANS state into a plurality of data objects, wherein at least one of the data objects is stored using fewer bits than the other data objects.

[0024] Optionally, producing the bitstream comprises writing the plurality of data objects to the bitstream.

[0025] Optionally, splitting the final rANS state comprises performing one or more bitwise XOR operations on the final rANS state to produce the plurality of data objects, and wherein the at least one of the data objects is stored using fewer bits comprises a remainder of the one or more bitwise XOR operations.

[0026] Optionally, the least one data object stored using fewer bits is stored using 16 bits or 8 bits.

[0027] Optionally, the other of the plurality of data objects are stored using 64 bits or 32 bits. Optionally, the method comprises, at the first computer system, modifying the latent representation by identifying values of the latent representation outside of a range, and substituting identified values with a flag comprising a value inside of the range.

[0028] Optionally, the method comprises, at the first computer system, while rANS encoding, modifying the latent representation by identifying values of the latent representation outside of a range, removing the identified values from the latent representation, and writing one or more position flags into the bitstream indicative of positions of removed identified values.

[0029] Optionally, the method comprises, at the second computer system, before the decoding with the second trained neural network, modifying the latent representation by substituting the flags with the identified values.

[0030] Optionally, the method comprises, at the second computer system, while rANS decoding, inserting the identified values into the latent representation at the positions of the removed identified values.

[0031] Optionally, the method comprises, at the first computer system, losslessly encoding the identified values into the bitstream, and, at the second computer system, losslessly decoding the bitstream to produce the identified values.

[0032] Optionally, the losslessly encoding and decoding of the identified values comprises Golomb encoding and decoding.

[0033] Optionally, the method comprises writing metadata into the bitstream indicative of respective lengths of all but one of the plurality of data objects.Improved rANS / WO

[0034] Optionally, rANS decoding comprises joining the plurality of data objects to produce the final rANS state.

[0035] According to an aspect, there is provided method for lossy image or video encoding and transmission, the method comprising: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation; rANS encoding the latent representation to produce a bitstream and transmitting the bitstream to a second computer system; wherein rANS encoding comprises: producing a final rANS state, and splitting the final rANS state into a plurality of data objects, wherein at least one of the data objects is stored using fewer bits than the other data objects.

[0036] According to an aspect, there is provided a method for lossy image or video receipt and decoding, the method comprising: at a second computer system, rANS decoding a bitstream to produce a latent representation, the bitstream produced by encoding an input image at a first computer system using a first trained neural network to produce a latent representation, and rANS encoding the latent representation to produce the bitstream, wherein rANS encoding comprises: producing a final rANS state, and splitting the final rANS state into a plurality of data objects, wherein at least one of the data objects is stored using fewer bits than the other data objects; at the second computer system, rANS decoding the bitstream to produce the latent representation; and decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image.

[0037] Optionally, rANS encoding comprises producing a final rANS state, and splitting the final rANS state into a plurality of data objects, wherein at least one of the data objects is stored using fewer bits than the other data objects.

[0038] Optionally, producing the bitstream comprises writing the plurality of data objects to the bitstream.

[0039] Optionally, splitting the final rANS state comprises performing one or more bitwise XOR operations on the final rANS state to produce the plurality of data objects, and wherein the at least one of the data objects is stored using fewer bits comprises a remainder of the one or more bitwise XOR operations.

[0040] Optionally, the least one data object stored using fewer bits is stored using 16 bits or 8 bits.

[0041] Optionally, the other of the plurality of data objects are stored using 64 bits or 32 bits. Optionally, the method comprises, at the first computer system, modifying the latent representation by identifying values of the latent representation outside of a range, and substituting identified values with a flag comprising a value inside of the range.Improved rANS / WO

[0042] Optionally, the method comprises, at the first computer system, while rANS encoding, modifying the latent representation by identifying values of the latent representation outside of a range, removing the identified values from the latent representation, and writing one or more position flags into the bitstream indicative of positions of removed identified values.

[0043] Optionally, the method comprises, at the second computer system, before the decoding with the second trained neural network, modifying the latent representation by substituting the flags with the identified values.

[0044] Optionally, the method comprises, at the second computer system, while rANS decoding, inserting the identified values into the latent representation at the positions of the removed identified values.

[0045] Optionally, the method comprises, at the first computer system, losslessly encoding the identified values into the bitstream, and, at the second computer system, losslessly decoding the bitstream to produce the identified values.

[0046] Optionally, the losslessly encoding and decoding of the identified values comprises Golomb encoding and decoding.

[0047] Optionally, the method comprises writing metadata into the bitstream indicative of respective lengths of all but one of the plurality of data objects.

[0048] Optionally, rANS decoding comprises joining the plurality of data objects to produce the final rANS state.

[0049] According to an aspect, there is provided a method for lossy image or video encoding, transmission and decoding, the method comprising: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation; rANS encoding the latent representation to produce a bitstream and transmitting the bitstream to a second computer system; at the second computer system, rANS decoding the bitstream to produce the latent representation; decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image; and wherein rANS encoding comprises producing a final rANS state, and splitting the final rANS state into a plurality of data object; and wherein producing the bitstream comprises writing metadata into the bitstream indicative of respective lengths of all but one of the plurality of data objects.

[0050] Optionally, rANS decoding comprises joining the plurality of data objects to produce the final rANS state.

[0051] Optionally, producing the bitstream comprises writing the plurality of data objects to the bitstream.Improved rANS / WO

[0052] Optionally, splitting the final rANS state comprises performing one or more bitwise XOR operations on the final rANS state to produce the plurality of data objects, and wherein the at least one of the data objects is stored using fewer bits comprises a remainder of the one or more bitwise XOR operations.

[0053] Optionally, the least one data object stored using fewer bits is stored using 16 bits or 8 bits.

[0054] Optionally, the other of the plurality of data objects are stored using 64 bits or 32 bits. Optionally, the method comprises, at the first computer system, modifying the latent representation by identifying values of the latent representation outside of a range, and substituting identified values with a flag comprising a value inside of the range.

[0055] Optionally, the method comprises at the first computer system, while rANS encoding, modifying the latent representation by identifying values of the latent representation outside of a range, removing the identified values from the latent representation, and writing one or more position flags into the bitstream indicative of positions of removed identified values.

[0056] Optionally, the method comprises, at the second computer system, before the decoding with the second trained neural network, modifying the latent representation by substituting the flags with the identified values.

[0057] Optionally, the method comprises, at the second computer system, while rANS decoding, inserting the identified values into the latent representation at the positions of the removed identified values.

[0058] Optionally, the method comprises, at the first computer system, losslessly encoding the identified values into the bitstream, and, at the second computer system, losslessly decoding the bitstream to produce the identified values.

[0059] Optionally, the losslessly encoding and decoding of the identified values comprises Golomb encoding and decoding.

[0060] According to an aspect, there is provided method for lossy image or video encoding and transmission, the method comprising: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation; rANS encoding the latent representation to produce a bitstream and transmitting the bitstream to a second computer system; wherein rANS encoding comprises producing a final rANS state, and splitting the final rANS state into a plurality of data object; and wherein the method comprises writing metadata into the bitstream indicative of respective lengths of all but one of the plurality of data objects.Improved rANS / WO

[0061] According to an aspect, there is provided a method for lossy image or video receipt and decoding, the method comprising: at a second computer system, rANS decoding a bitstream to produce a latent representation, the bitstream produced by encoding an input image at a first computer system using a first trained neural network to produce a latent representation, and rANS encoding the latent representation to produce the bitstream, wherein rANS encoding comprises producing a final rANS state, and splitting the final rANS state into a plurality of data objects; and the producing the bitstream comprises writing metadata into the bitstream indicative of respective lengths of all but one of the plurality of data objects; at the second computer system, rANS decoding the bitstream to produce the latent representation; and decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image.

[0062] Optionally, rANS decoding comprises joining the plurality of data objects to produce the final rANS state.

[0063] Optionally, producing the bitstream comprises writing the plurality of data objects to the bitstream.

[0064] Optionally, splitting the final rANS state comprises performing one or more bitwise XOR operations on the final rANS state to produce the plurality of data objects, and wherein the at least one of the data objects is stored using fewer bits comprises a remainder of the one or more bitwise XOR operations.

[0065] Optionally, the least one data object stored using fewer bits is stored using 16 bits or 8 bits.

[0066] Optionally, the other of the plurality of data objects are stored using 64 bits or 32 bits. Optionally, the method comprises, at the first computer system, modifying the latent representation by identifying values of the latent representation outside of a range, and substituting identified values with a flag comprising a value inside of the range.

[0067] Optionally, the method comprises at the first computer system, while rANS encoding, modifying the latent representation by identifying values of the latent representation outside of a range, removing the identified values from the latent representation, and writing one or more position flags into the bitstream indicative of positions of removed identified values.

[0068] Optionally, the method comprises, at the second computer system, before the decoding with the second trained neural network, modifying the latent representation by substituting the flags with the identified values.Improved rANS / WO

[0069] Optionally, the method comprises, at the second computer system, while rANS decoding, inserting the identified values into the latent representation at the positions of the removed identified values.

[0070] Optionally, the method comprises, at the first computer system, losslessly encoding the identified values into the bitstream, and, at the second computer system, losslessly decoding the bitstream to produce the identified values.

[0071] Optionally, the losslessly encoding and decoding of the identified values comprises Golomb encoding and decoding.

[0072] According to an aspect, there is provided a method for lossy image or video encoding, transmission and decoding, the method comprising: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation; rANS encoding the latent representation to produce a bitstream, the rANS encoding comprising modifying the latent representation based on a range of values of the latent representation; transmitting the bitstream to a second computer system; at the second computer system, rANS decoding the bitstream to produce the latent representation, the rANS decoding comprising modifying the latent representation based on the range of values of the latent representation; decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image.

[0073] Optionally, the modifying the latent representation comprises identifying values of the latent representation outside of the range, and substituting identified values with a flag comprising a value inside of the range.

[0074] Optionally, the method comprises, at the first computer system, losslessly encoding the identified values into the bitstream, and, at the second computer system, losslessly decoding the bitstream to produce the identified values.

[0075] Optionally, the method comprises, at the second computer system, before the decoding with the second trained neural network, modifying the latent representation by substituting the flags with the identified values.

[0076] Optionally, the losslessly encoding and decoding of the identified values comprises Golomb encoding and decoding.

[0077] Optionally, the method comprises at the first computer system modifying the latent representation by identifying values of the latent representation outside of the range, removing the identified values from the latent representation, and writing one or more position flags into the bitstream indicative of positions of removed identified values.Improved rANS / WO

[0078] Optionally, the method comprises at the second computer system inserting the identified values into the latent representation at the positions of the removed identified values.

[0079] According to an aspect, there is provided a method for lossy image or video encoding and transmission, the method comprising: receiving an input image at a first computer system; and encoding the input image using a first trained neural network to produce a latent representation; rANS encoding the latent representation to produce a bitstream, the rANS encoding comprising modifying the latent representation based on a range of the values of the latent representation; and transmitting the bitstream to a second computer system.

[0080] According to an aspect, there is provided a method for lossy image or video receipt and decoding, the method comprising: at a second computer system, rANS decoding a bitstream to produce a latent representation, the bitstream produced by encoding an input image at a first computer system using a first trained neural network to produce a latent representation, and rANS encoding the latent representation to produce the bitstream, the rANS encoding comprising modifying the latent representation based on a range of values of the latent representation; wherein the rANS decoding comprises modifying the latent representation based on the range of values of the latent representation; and decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image.

[0081] Optionally, the modifying the latent representation comprises identifying values of the latent representation outside of the range, and substituting identified values with a flag comprising a value inside of the range.

[0082] Optionally, the method comprises, at the first computer system, losslessly encoding the identified values into the bitstream, and, at the second computer system, losslessly decoding the bitstream to produce the identified values.

[0083] Optionally, the method comprises, at the second computer system, before the decoding with the second trained neural network, modifying the latent representation by substituting the flags with the identified values.

[0084] Optionally, the losslessly encoding and decoding of the identified values comprises Golomb encoding and decoding.

[0085] Optionally, the method comprises at the first computer system modifying the latent representation by identifying values of the latent representation outside of the range, removing the identified values from the latent representation, and writing one or more position flags into the bitstream indicative of positions of removed identified values.Improved rANS / WO

[0086] Optionally, the method comprises at the second computer system inserting the identified values into the latent representation at the positions of the removed identified values.

[0087] According to an aspect, there is provided a method for lossy image or video encoding, transmission and decoding, the method comprising: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation; rANS encoding a first portion of the latent representation and entropy encoding a second portion of the latent representation using a second entropy encoding method to produce a bitstream and transmitting the bitstream to a second computer system; at the second computer system, rANS decoding and entropy decoding, using a second entropy decoding method the bitstream to produce the first and second portions of the latent representation, and combining the first and second portions to produce the latent representation; and decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image.

[0088] Optionally, the second entropy encoding and decoding methods comprise non-rANS entropy encoding and decoding methods.

[0089] Optionally, the non-rANS entropy encoding and decoding methods comprise Golomb encoding and decoding.

[0090] According to an aspect there is provided, a method for lossy image or video encoding and transmission, the method comprising: receiving an input image at a first computer system; encoding the input image using a first trained neural network to produce a latent representation; rANS encoding a first portion of the latent representation and entropy encoding a second portion of the latent representation using a second entropy encoding method to produce a bitstream and transmitting the bitstream to a second computer system.

[0091] According to an aspect, there is provided a method for lossy image or video receipt and decoding, the method comprising: receiving a bitstream at a second computer system, the bitstream produced by rANS encoding a first portion of a latent representation and entropy encoding a second portion of the latent representation using a second entropy encoding method to produce the bitstream, the latent representation produced by encoding an input image using a first trained neural network; at the second computer system, rANS decoding and entropy decoding, using a second entropy decoding method, the bitstream to produce the first and second portions of a latent representation, and combining the first and second portions to produce the latent representation; and decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image.Improved rANS / WO

[0092] Optionally, the second entropy encoding and decoding methods comprise non-rANS entropy encoding and decoding methods.

[0093] Optionally, the non-rANS entropy encoding and decoding methods comprise Golomb encoding and decoding.

[0094] According to an aspect, there is provided a data processing apparatus configured to perform any of the above methods.

[0095] According to an aspect, there is provided a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out any of the above methods.

[0096] According to an aspect, there is provided a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out any of the above methods.

[0097] BRIEF DESCRIPTION OF THE DRAWINGS

[0098] Aspects of the invention will now be described by way of examples, with reference to the following figures in which:

[0099] Figure 1 illustrates an example of an image or video compression, transmission and decompression pipeline;

[0100] Figure 2 illustrates a further example of an image or video compression, transmission and decompression pipeline including a hyper-network;

[0101] Figure 3 illustrates an example of a video compression, transmission and decompression pipeline;

[0102] Figure 4 illustrates an example of a video compression, transmission and decompression system; and

[0103] Figure 5 illustrates a constructing a bitstream using methods according to the present disclosure.

[0104] DETAILED DESCRIPTION OF THE DRAWINGS

[0105] Compression processes may be applied to any form of information to reduce the amount of data, or file size, required to store that information. Image and video information is an example of information that may be compressed. The file size required to store the information, particularly during a compression process when referring to the compressed file, may be referred to as the rate. In general, compression can be lossless or lossy. In both forms of compression, the file size is reduced. However, in lossless compression, no information is lostImproved rANS / WO

[0106] when the information is compressed and subsequently decompressed. This means that the original file storing the information is fully reconstructed during the decompression process. In contrast to this, in lossy compression information may be lost in the compression and decompression process and the reconstructed file may differ from the original file. Image and video files containing image and video data are common targets for compression.

[0107] In a compression process involving an image, the input image may be represented as x. The data representing the image may be stored in a tensor of dimensions H × W × C, where H represents the height of the image, W represents the width of the image and C represents the number of channels of the image. Each H × W data point of the image represents a pixel value of the image at the corresponding location. Each channel C of the image represents a different component of the image for each pixel which are combined when the image file is displayed by a device. For example, an image file may have 3 channels with the channels representing the red, green and blue component of the image respectively. In this case, the image information is stored in the RGB colour space, which may also be referred to as a model or a format. Other examples of colour spaces or formats include the CMKY and the YCbCr colour models. However, the channels of an image file are not limited to storing colour information and other information may be represented in the channels. As a video may be considered a series of images in sequence, any compression process that may be applied to an image may also be applied to a video. Each image making up a video may be referred to as a frame of the video.

[0108] The output image may differ from the input image and may be represented by x. The difference between the input image and the output image may be referred to as distortion or a difference in image quality. The distortion can be measured using any distortion function which receives the input image and the output image and provides an output which represents the difference between input image and the output image in a numerical way. An example of such a method is using the mean square error (MSE) between the pixels of the input image and the output image, but there are many other ways of measuring distortion, as will be known to the person skilled in the art. The distortion function may comprise a trained neural network.

[0109] Typically, the rate and distortion of a lossy compression process are related. An increase in the rate may result in a decrease in the distortion, and a decrease in the rate may result in an increase in the distortion. Changes to the distortion may affect the rate in a corresponding manner. A relation between these quantities for a given compression technique may be defined by a rate-distortion equation.

[0110] Al based compression processes may involve the use of neural networks. A neural network is an operation that can be performed on an input to produce an output. A neuralImproved rANS / WO

[0111] network may be made up of a plurality of layers. The first layer of the network receives the input. One or more operations may be performed on the input by the layer to produce an output of the first layer. The output of the first layer is then passed to the next layer of the network which may perform one or more operations in a similar way. The output of the final layer is the output of the neural network.

[0112] Each layer of the neural network may be divided into nodes. Each node may receive at least part of the input from the previous layer and provide an output to one or more nodes in a subsequent layer. Each node of a layer may perform the one or more operations of the layer on at least part of the input to the layer. For example, a node may receive an input from one or more nodes of the previous layer. The one or more operations may include a convolution, a weight, a bias and an activation function. Convolution operations are used in convolutional neural networks. When a convolution operation is present, the convolution may be performed across the entire input to a layer. Alternatively, the convolution may be performed on at least part of the input to the layer.

[0113] Each of the one or more operations is defined by one or more parameters that are associated with each operation. For example, the weight operation may be defined by a weight matrix defining the weight to be applied to each input from each node in the previous layer to each node in the present layer. In this example, each of the values in the weight matrix is a parameter of the neural network. The convolution may be defined by a convolution matrix, also known as a kernel. In this example, one or more of the values in the convolution matrix may be a parameter of the neural network. The activation function may also be defined by values which may be parameters of the neural network. The parameters of the network may be varied during training of the network.

[0114] Other features of the neural network may be predetermined and therefore not varied during training of the network. For example, the number of layers of the network, the number of nodes of the network, the one or more operations performed in each layer and the connections between the layers may be predetermined and therefore fixed before the training process takes place. These features that are predetermined may be referred to as the hyperparameters of the network. These features are sometimes referred to as the architecture of the network.

[0115] To train the neural network, a training set of inputs may be used for which the expected output, sometimes referred to as the ground truth, is known. The initial parameters of the neural network are randomized and the first training input is provided to the network. The output of the network is compared to the expected output, and based on a difference between the output and the expected output the parameters of the network are varied such that the differenceImproved rANS / WO

[0116] between the output of the network and the expected output is reduced. This process is then repeated for a plurality of training inputs to train the network. The difference between the output of the network and the expected output may be defined by a loss function. The result of the loss function may be calculated using the difference between the output of the network and the expected output to determine the gradient of the loss function. Back-propagation of the gradient descent of the loss function may be used to update the parameters of the neural network using the gradients dL / dy of the loss function. A plurality of neural networks in a system may be trained simultaneously through back-propagation of the gradient of the loss function to each network.

[0117] In the context of image or video compression, this type of system, where simultaneous training with back-propagation through each element or the whole network architecture may be referred to as end-to-end, learned image or video compression. Unlike in traditional compression algorithms that use primarily handcrafted, manually constructed steps, an end-to-end learned system learns itself during training what combination of parameters best achieves the goal of minimising the loss function. This approach is advantageous compared to systems that are not end-to-end learned because an end-to-end system has a greater flexibility to learn weights and parameters that might be counter-intuitive to someone handcrafting features.

[0118] It will be appreciated that the term "training" or "learning" as used herein means the process of optimizing an artificial intelligence or machine learning model, based on a given set of data. This involves iteratively adjusting the parameters of the model to minimize the discrepancy between the model's predictions and the actual data, represented by the abovedescribed rate-distortion loss function.

[0119] The training process may comprise multiple epochs. An epoch refers to one complete pass of the entire training dataset through the machine learning algorithm. During an epoch, the model's parameters are updated in an effort to minimize the loss function. It is envisaged that multiple epochs may be used to train a model, with the exact number depending on various factors including the complexity of the model and the diversity of the training data.

[0120] Within each epoch, the training data may be divided into smaller subsets known as batches. The size of a batch, referred to as the batch size, may influence the training process. A smaller batch size can lead to more frequent updates to the model's parameters, potentially leading to faster convergence to the optimal solution, but at the cost of increased computational resources. Conversely, a larger batch size involves fewer updates, which can be more computationally efficient but might converge slower or even fail to converge to the optimal solution.Improved rANS / WO

[0121] The learnable parameters are updated by a specified amount each time, determined by the learning rate. The learning rate is a hyperparameter that decides how much the parameters are adjusted during the training process. A smaller learning rate implies smaller steps in the parameter space and a potentially more accurate solution, but it may require more epochs to reach that solution. On the other hand, a larger learning rate can expedite the training process but may risk overshooting the optimal solution or causing the training process to diverge.

[0122] The training described herein may involve use of a validation set, which is a portion of the data not used in the initial training, which is used to evaluate the model's performance and to prevent overfitting. Overfitting occurs when a model learns the training data too well, to the point that it fails to generalize to unseen data. Regularization techniques, such as dropout or L1 / L2 regularization, can also be used to mitigate overfitting.

[0123] It will be appreciated that training a machine learning model is an iterative process that may comprise selection and tuning of various parameters and hyperparameters. As will be appreciated, the specific details, such as hyper parameters and so on, of the training process may vary and it is envisaged that producing a trained model in this way may achieved in a number of different ways with different epochs, batch sizes, learning rates, regularisations, and so on, the details of which are not essential to enabling the advantages and effects of the present disclosure, except where stated otherwise. The point at which an "untrained" neural network is considered be "trained" is envisaged to be case specific and depend on, for example, on a number of epochs, a plateauing of any further learning, or some other metric and is not considered to be essential in achieving the advantages described herein.

[0124] More details of an end-to-end, learned compression process will now be described. It will be appreciated that in some cases, end-to-end, learned compression processes may be combined with one or more components that are handcrafted or trained separately.

[0125] In the case of Al based image or video compression, the loss function may be defined by the rate distortion equation. The rate distortion equation may be represented by Loss = D + A * R, where D is the distortion function, is a weighting factor, and R is the rate loss. may be referred to as a lagrange multiplier. The langrange multiplier provides as weight for a particular term of the loss function in relation to each other term and can be used to control which terms of the loss function are favoured when training the network.

[0126] In the case of Al based image or video compression, a training set of input images may be used. An example training set of input images is the KODAK image set (for example at www.cs.albany.edu / xypan / research / snr / Kodak.html). An example training set of input images is the IMAX image set. An example training set of input images is the Imagenet dataset (forImproved rANS / WO

[0127] example at www.image-net.org / download). An example training set of input images is the CLIC Training Dataset P ("professional") and M ("mobile") (for example at http: / / ch al 1 enge. compressi on. cc / tasks / ).

[0128] An example of an Al based compression, transmission and decompression process 100 is shown in Figure 1. As a first step in the Al based compression process, an input image 5 is provided. The input image 5 is provided to a trained neural network 110 characterized by a function fg acting as an encoder. The encoder neural network 110 produces an output based on the input image. This output is referred to as a latent representation of the input image 5. In a second step, the latent representation is quantised in a quantisation process 140 characterised by the operation Q, resulting in a quantized latent. The quantisation process transforms the continuous latent representation into a discrete quantized latent. An example of a quantization process is a rounding function.

[0129] In a third step, the quantized latent is entropy encoded in an entropy encoding process 150 to produce a bitstream 130. The entropy encoding process may be for example, range or arithmetic encoding. In a fourth step, the bitstream 130 may be transmitted across a communication network.

[0130] In a fifth step, the bitstream is entropy decoded in an entropy decoding process 160. The quantized latent is provided to another trained neural network 120 characterized by a function ge acting as a decoder, which decodes the quantized latent. The trained neural network 120 produces an output based on the quantized latent. The output may be the output image of the Al based compression process 100. The encoder-decoder system may be referred to as an autoencoder.

[0131] Entropy encoding processes such as range or arithmetic encoding are typically able to losslessly compress given input data up to close to the fundamental entropy limit of that data, as determined by the total entropy of the distribution of that data. Accordingly, one way in which end-to-end, learned compression can minimise the rate loss term of the rate-distortion loss function and thereby increase compression effectiveness is to learn autoencoder parameter values that produce low entropy latent representation distributions. Producing latent representations distributed with as low an entropy as possible allows entropy encoding to compress the latent distributions as close to or to the fundamental entropy limit for that distribution. The lower the entropy of the distribution, the more entropy encoding can losslessly compress it and the lower the amount of data in the corresponding bitstream. In some cases where the latent representation is distributed according to a gaussian or Laplacian distribution, this learning may comprise learning optimal location and scale parameters of the gaussian orImproved rANS / WO

[0132] Laplacian distributions, in other cases, it allows the learning of more flexible latent representation distributions which can further help to achieve the minimising of the ratedistortion loss function in ways that are not intuitive or possible to do with handcrafted features. Examples of these and other advantages are described in W02021 / 220008A1, which is incorporated herein by reference in its entirety.

[0133] Something which is closely linked to the entropy encoding of the latent distribution and which accordingly also has an effect on the effectiveness of compression of end-to-end learned approaches is the quantisation step. During inference, a rounding function may be used to quantise a latent representation distribution into bins of given sizes, a rounding function is not differentiable everywhere. Rather, a rounding function is effectively one or more step functions whose gradient is either zero (at the top of the steps) or infinity (at the boundary between steps). Back propagating a gradient of a loss function through a rounding function is challenging. Instead, during training, quantisation by rounding function is replaced by one or more other approaches. For example, the functions of a noise quantisation model are differentiable everywhere and accordingly do allow backpropagation of the gradient of the loss function through the quantisation parts of the end-to-end, learned system. Alternatively, a straight-through estimator (STE) quantisation model or one other quantisation models may be used. It is also envisaged that different quantisation models may be used for during evaluation of different term of the loss function. For example, noise quantisation may be used to evaluate the rate or entropy loss term of the rate-distortion loss function while STE quantisation may be used to evaluate the distortion term.

[0134] In a similar manner to how learning parameters top produce certain distributions of the latent representation facilitates achieving better rate loss term minimisation, end-to-end learning of the quantisation process achieves a similar effect. That is, learnable quantisation parameters provide the architecture with a further degree of freedom to achieve the goal of minimising the loss function. For example, parameters corresponding to quantisation bin sizes may be learned which is likely to result in an improved rate-distortion loss outcome compared to approaches using hand-crafted quantisation bin sizes.

[0135] Further, as the rate-distortion loss function constantly has to balance a rate loss term against a distortion loss term, it has been found that the more degrees of freedom the system has during training, the better the architecture is at achieving optimal rate and distortion trade off.

[0136] The system described above may be distributed across multiple locations and / or devices. For example, the encoder 110 may be located on a device such as a laptop computer,Improved rANS / WO

[0137] desktop computer, smart phone or server. The decoder 120 may be located on a separate device which may be referred to as a recipient device. The system used to encode, transmit and decode the input image 5 to obtain the output image 6 may be referred to as a compression pipeline.

[0138] The Al based compression process may further comprise a hyper-network 105 for the transmission of meta-information that improves the compression process. The hyper-network 105 comprises a trained neural network 115 acting as a hyper-encoder fg and a trained neural network 125 acting as a hyper-decoder gg. An example of such a system is shown in Figure 2. Components of the system not further discussed may be assumed to be the same as discussed above. The neural network 115 acting as a hyper-encoder receives the latent that is the output of the encoder 110. The hyper-encoder 115 produces an output based on the latent representation that may be referred to as a hyper-latent representation. The hyper-latent is then quantized in a quantization process 145 characterised by Qhto produce a quantized hyper-latent. The quantization process 145 characterised by Qhmay be the same as the quantisation process 140 characterised by Q discussed above.

[0139] In a similar manner as discussed above for the quantized latent, the quantized hyper-latent is then entropy encoded in an entropy encoding process 155 to produce a bitstream 135. The bitstream 135 may be entropy decoded in an entropy decoding process 165 to retrieve the quantized hyper-latent. The quantized hyper-latent is then used as an input to the trained neural network 125 acting as a hyper-decoder. However, in contrast to the compression pipeline 100, the output of the hyper-decoder may not be an approximation of the input to the hyper-encoder 115. Instead, the output of the hyper-decoder is used to provide parameters for use in the entropy encoding process 150 and entropy decoding process 160 in the main compression process 100. For example, the output of the hyper-decoder 125 can include one or more of the mean, standard deviation, variance or any other parameter used to describe a probability model for the entropy encoding process 150 and entropy decoding process 160 of the latent representation. In the example shown in Figure 2, only a single entropy decoding process 165 and hyper-decoder 125 is shown for simplicity. However, in practice, as the decompression process usually takes place on a separate device, duplicates of these processes will be present on the device used for encoding to provide the parameters to be used in the entropy encoding process 150.

[0140] Further transformations may be applied to at least one of the latent and the hyper-latent at any stage in the Al based compression process 100. For example, at least one of the latent and the hyper latent may be converted to a residual value before the entropy encoding process 150,155 is performed. The residual value may be determined by subtracting the mean value ofImproved rANS / WO

[0141] the distribution of latents or hyper-latents from each latent or hyper latent. The residual values may also be normalised.

[0142] To perform training of the Al based compression process described above, a training set of input images may be used as described above. During the training process, the parameters of both the encoder 110 and the decoder 120 may be simultaneously updated in each training step. If a hyper-network 105 is also present, the parameters of both the hyper-encoder 115 and the hyper-decoder 125 may additionally be simultaneously updated in each training step. The training process may further include a generative adversarial network (GAN). When applied to an Al based compression process, in addition to the compression pipeline described above, an additional neutral network acting as a discriminator is included in the system. The discriminator receives an input and outputs a score based on the input providing an indication of whether the discriminator considers the input to be ground truth or fake. For example, the indicator may be a score, with a high score associated with a ground truth input and a low score associated with a fake input. For training of a discriminator, a loss function is used that maximizes the difference in the output indication between an input ground truth and input fake.

[0143] When a GAN is incorporated into the training of the compression process, the output image 6 may be provided to the discriminator. The output of the discriminator may then be used in the loss function of the compression process as a measure of the distortion of the compression process. Alternatively, the discriminator may receive both the input image 5 and the output image 6 and the difference in output indication may then be used in the loss function of the compression process as a measure of the distortion of the compression process. Training of the neural network acting as a discriminator and the other neutral networks in the compression process may be performed simultaneously. During use of the trained compression pipeline for the compression and transmission of images or video, the discriminator neural network is removed from the system and the output of the compression pipeline is the output image 6.

[0144] Incorporation of a GAN into the training process may cause the decoder 120 to perform hallucination. Hallucination is the process of adding information in the output image 6 that was not present in the input image 5. In an example, hallucination may add fine detail to the output image 6 that was not present in the input image 5 or received by the decoder 120. The hallucination performed may be based on information in the quantized latent received by decoder 120.

[0145] Details of a video compression process will now be described. As discussed above, a video is made up of a series of images arranged in sequential order. Al based compression process 100 described above may be applied multiple times to perform compression,Improved rANS / WO

[0146] transmission and decompression of a video. For example, each frame of the video may be compressed, transmitted and decompressed individually. The received frames may then be grouped to obtain the original video.

[0147] The frames in a video may be labelled based on the information from other frames that is used to decode the frame in a video compression, transmission and decompression process. As described above, frames which are decoded using no information from other frames may be referred to as I-frames. Frames which are decoded using information from past frames may be referred to as P-frames. Frames which are decoded using information from past frames and future frames may be referred to as B-frames. Frames may not be encoded and / or decoded in the order that they appear in the video. For example, a frame at a later time step in the video may be decoded before a frame at an earlier time.

[0148] The images represented by each frame of a video may be related. For example, a number of frames in a video may show the same scene. In this case, a number of different parts of the scene may be shown in more than one of the frames. For example, objects or people in a scene may be shown in more than one of the frames. The background of the scene may also be shown in more than one of the frames. If an object or the perspective is in motion in the video, the position of the object or background in one frame may change relative to the position of the object or background in another frame. The transformation of a part of the image from a first position in a first frame to a second position in a second frame may be referred to as flow, warping or motion compensation. The flow may be represented by a vector. One or more flows that represent the transformation of at least part of one frame to another frame may be referred to as a flow map.

[0149] An example Al based video compression, transmission, and decompression process 200 is shown in Figure 3. The process 200 shown in Figure 3 is divided into an I-frame part 201 for decompressing I-frames, and a P-frame part 202 for decompressing P-frames. It will be understood that these divisions into different parts are arbitrary and the process 200 may also be considered as a single, end-to-end pipeline.

[0150] As described above, 1-frames do not rely on information from other frames so the I-frame part 201 corresponds to the compression, transmission, and decompression process illustrated in Figures 1 or 2. The specific details will not be repeated here but, in summary, an input image x0is passed into an encoder neural network 203 producing a latent representation which is quantised and entropy encoded into a bitstream 204. The subscript 0 in x0indicates the input image corresponds to a frame of a video stream at position t = 0. This may be the first frame of an entire video stream or the first frame of a chunk of a video stream made up of, forImproved rANS / WO

[0151] example, an I-frame and a plurality of subsequent P-frames and / or B-frames. The bitstream 204 is then entropy decoded and passed into a decoder neural network 205 to reproduce a reconstructed image x0which in this case is an I-frame. The decoding step may be performed both locally at the same location as where the input image compression occurs as well as at the location where the decompression occurs. This allows the reconstructed image x0to be available for later use by components of both the encoding and decoding sides of the pipeline.

[0152] In contrast to I-frames, P-frames (and B-frames) do rely on information from other frames. Accordingly, the P-frame part 202 at the encoding side of the pipeline takes as input not only the input image xtthat is to be compressed (corresponding to a frame of a video stream at position t), but also one or more previously reconstructed images xt-1from an earlier frame t-1. As described above, the previously reconstructed xt-1is available at both the encode and decode side of the pipeline and can accordingly be used for various purposes at both the encode and decode sides.

[0153] At the encode side, previously reconstructed images may be used for generating a flow map containing information indicative of inter-frame movement of pixels between frames. In the example of Figure 3, both the image being compressed xtand the previously reconstructed image from an earlier frame xt-1are passed into a flow module part 206 of the pipeline. The flow module part 206 comprises an autoencoder such as that of the autoencoder systems of Figures 1 and 2 but where the encoder neural network 207 has been trained to produce a latent representation of a flow map from inputs xt-1and xt, which is indicative of inter-frame movement of pixels or pixel groups between xt-1and xt. The latent representation of the flow map is quantised and entropy encoded to compress it and then transmitted as a bitstream 208. On the decode side, the bitstream is entropy decoded and passed to a decoder neural network 209 to produce a reconstructed flow map f.

[0154] The reconstructed flow map f is applied to the previously reconstructed image x̂t-1to generate a warped image x̂t-1,w. It is envisaged that any suitable warping technique may be used, for example bi-linear or tri-linear warping, as is described in Agustsson, E., Minnen, D., Johnston, N., Balle, J., Hwang, S. J., and Toderici, G. (2020), Scale-space flow for end-to-end optimized video compression. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 8503-8512), which is incorporated herein by reference in its entirety. It is further envisaged that a scale-space flow approach as described in the above paper may also optionally be used. The warped image x̂t-1,wis a prediction of how the previously reconstructed image xt-1might have changed between frame positions t-1 and t,Improved rANS / WO

[0155] based on the output flow map produced by the flow module part 206 autoencoder system from the inputs of xtand x̂t-1.

[0156] As with the I-frame, the reconstructed flow map f and corresponding warped image xt-l wmay be produced both on the encode side and the decode side of the pipeline so they are available for use by other components of the pipeline on both the encode and decode sides.

[0157] In the example of Figure 3, both the image being compressed xtand the

[0158]

[0159] arepassed into a residual module part 210 of the pipeline. The residual module part 210 comprises an autoencoder system such as that of the autoencoder systems of Figures 1 and 2 but where the encoder neural network 211 has been trained to produce a latent representation of a residual map indicative of differences between the input image xtand the warped image

[0160]

[0161] The latent representation of the residual map is then quantised and entropy encoded into a bitstream 212 and transmitted. The bitstream 212 is then entropy decoded and passed into a decoder neural network 213 which reconstructs a residual map r from the decoded latent representation.

[0162] Alternatively, a residual map may first be pre-calculated between xtand the xt-l wand the pre-calculated residual map may be passed into an autoencoder for compression only. This hand-crafted residual map approach is computationally simpler, but reduces the degrees of freedom with which the architecture may learn weights and parameters to achieve its goal during training of minimising the rate-distortion loss function.

[0163] Finally, on the decode side, the residual map r is applied (e.g., combined by addition, subtraction or a different operation) to the warped image to produce a reconstructed image xtwhich is a reconstruction of image xtand accordingly corresponds to a P-frame at position t in a sequence of frames of a video stream. It will be appreciated that the reconstructed image xtcan then be used to process the next frame. That is, it can be used to compress, transmit and decompress xt+1, and so on until an entire video stream or chunk of a video stream has been processed.

[0164] Alternatively, the residual autoencoder may be trained to reconstruct the frame xtdirectly from the entropy decoded bitstream by removing the connection between x̂t-1,wand the output of the residual block 210, thereby eliminating any direct combination step with the warped previously decoded image to speed up inference. In this case, the flow information is intuitively understood to be indirectly captured within the residual information, which the residual decoder is able to learn to use to directly reconstruct the output image xt.

[0165] Alternatively, the residual autoencoder may be trained to reconstruct the frame xtdirectly from the entropy decoded bitstream in combination with some representation of flowImproved rANS / WO

[0166] injected into one or more layers of the residual decoder. In this case, the flow information is intuitively understood to be indirectly captured within the injected information, which the residual decoder is able to learn to use while decoding the latent representation of flow information to directly reconstruct the output image xt.

[0167] Thus, for a block of video frames comprising an I-frame and n subsequent P-frames, the bitstream may contain (i) a quantised, entropy encoded latent representation of the 1-frame image, and (ii) a quantised, entropy encoded latent representation of a flow map and residual map of each P-frame image. For completeness, whilst not illustrated in Figure 3, any of the autoencoder systems of Figure 3 may comprise hyper and hyper-hyper networks such as those described in connection with Figure 2. Accordingly, the bitstream may also contain hyper and hyper-hyper parameters, their latent quantised, entropy encoded latent representations and so on, of those networks as applicable.

[0168] Finally, the above approach may generally also be extended to B-frames, for example as is described in Pourreza, R., and Cohen, T. (2021). Extending neural p-frame codecs for b-frame coding. In Proceedings of the IEEE / CVF International Conference on Computer Vision (pp. 6680-6689).

[0169] The above-described flow and residual based approach is highly effective at reducing the amount of data that needs to be transmitted because, as long as at least one reconstructed frame (e.g., I-frame x̂t-1) is available, the encode side only needs to compress and transmit a flow map and a residual map (and any hyper or hyper-hyper parameter information, as applicable) to reconstruct a subsequent frame.

[0170] Figure 4 shows an example of an Al image or video compression process such as that described above in connection with Figures 1-3 implemented in a video streaming system 400. The system 400 comprises a first device 401 and a second device 402. The first and second devices 401, 402 may be user devices such as smartphones, tablets, AR / VR headsets or other portable devices. In contrast to known systems which primarily perform inference on GPUs such as Nvidia A100, Geforce 3090, Gefore 4090 GPU cards, the system 400 of Figure 4 performs inference on a CPU of the first and second devices respectively. That is, compute for performing both encoding and decoding are performed by the respective CPUs of the first and second devices 401, 402. This places very different power usage, memory and runtime constraints on the implementation of the above methods than when implementing Al-based compression methods on GPUs. In one example, the CPU of first and second devices 401, 402 may comprise a Qualcomm Snapdragon CPU.Improved rANS / WO

[0171] The first device 401 comprises a media capture device 403, such as a camera, arranged to capture a plurality of images, referred to hereafter as a video stream 404, of a scene 405. The video stream 404 is passed to a pre-processing module 406 which splits the video stream into blocks of frames, various frames of which will be designated as I-frames, P-frames, and / or B-frames. The blocks of frames are then compressed by an Al-compression module 407 comprising the encode side of the Al-based video compression pipeline of Figure 3. The output of the Al-compression module is accordingly a bitstream 408a which is transmitted from the first device 401, for example via a communications channel, for example over one or more of a WiFi, 3G, 4G or 5G channel, which may comprise internet or cloud-based 409 communications.

[0172] The second device 402 receives the communicated bitstream 408b which is passed to an Al-decompression module 410 comprising the decode side of the Al-based video compression pipeline of Figure 3. The output of the Al-decompression module 410 is the reconstructed 1-frames, P-frames and / or B-frames which are passed to a post-processing module 411 where they can prepared, for example passed into a buffer, in preparation for streaming 412 to and rendering on a display device 413 of the second device 402.

[0173] It is envisaged that the system 400 of Figure 4 may be used for live video streaming at 30fps of a 1080p video stream, which means a cumulative latency of both the encode and decode side is below substantially 50ms, for example substantially 30ms or less. Achieving this level of runtime performance with only CPU compute on user devices presents challenges which are not addressed by known methods and systems or in the wider Al-compression literature.

[0174] For example, execution of different parts of the compression pipeline during inference may be optimized by adjusting the order in which operations are performed using one or more known CPU scheduling methods. Efficient scheduling can allow for operations to be performed in parallel, thereby reducing the total execution time. It is also envisaged that efficient management of memory resources may be implemented, including optimising caching methods such as storing frequently-accessed data in faster memory locations, and memory reuse, which minimizes memory allocation and deallocation operations.

[0175] A number of concepts related to the Al compression processes and / or their implementation in a hardware system discussed above will now be described. Although each concept is described separately, one or more of the concepts described below may be applied in an Al based compression process as described above.Improved rANS / WO

[0176] Improving rANS

[0177] As described above, one step of an Al-based compression pipeline is the lossless or entropy encoding step. A known lossless encoding approach is range asymetric numeral system (rANS), described in Duda, Jarek. " Asymmetric numeral systems: entropy coding combining speed of Huffman coding with compression rate of arithmetic coding." arXiv preprint arXiv: 1311.2540 (2013) and is incorporated herein by reference in its entirety. A brief explanation of rANS is provided below using a toy example.

[0178] Consider a list of symbols [0, 1,2,3], In the space of our toy example, we know how often each symbol typically appears, giving a corresponding list of symbol frequencies: [2, 3, 1, 4], That is, the symbol 3 occurs the most, followed by the symbol 1, followed by the symbol 0, followed by the symbol 2 (which occurs a quarter as often as the symbol 3). Tying this back to Al-based image compression, our list of symbols are all possible values of a latent representation of an input frame (optionally quantised, and / or normalised) and the respective frequencies are simply the counts of each of the symbols.

[0179] From the list of symbols and their frequencies, we can produce: (i) the total frequency (i.e. the total counts of all symbols combined) and (ii) the cumulative frequencies (treating the first frequency as zero). Finally, we also choose normalisation parameters comprising a base, some arbitrary rANS size threshold which we will use to periodically flush out bits of the rANS state into the bitstream (although this process, known as bit flushing, is optional), and a starting rANS state value (e.g., 1). This then gives us the following items of information for our toy example:

[0180] Symbols: [0, 1, 2, 3]

[0181] Frequencies: [2, 3, 1, 4], or more explicitly:

[0182] / (0) = 2

[0183] / (I) = 3

[0184] / (2) = 1

[0185] / (3) = 4

[0186] Total frequency: F = 2 + 3 + l + 4 = 10

[0187] Cumulative frequencies: [0, 2, 5, 6], or more explicitly

[0188] / c(0) = 0

[0189] / c(l) = 0 + 2 = 2

[0190] / c(2) = 2 + 3 = 5

[0191] / c(3) = 5 + 1 = 6Improved rANS / WO

[0192] Normalisation parameters:

[0193] base = 24

[0194] rANS state threshold = 160

[0195] starting rANS state x = 1

[0196] That is, when using bit flushing, whenever the rANS state exceeds the number 160, we move 4 bits out into the bitstream to reduce the size of the rANS state, allowing us to keep adding to it without exceeding the threshold. Alternatively, if there is no threshold, we can simply keep adding to the rANS state until all of the symbols in our message have been encoded. These values are of course illustrative and a toy example only. The actual values of the normalization parameters are use case dependent, as will be appreciated by the skilled person.

[0197] With this information, we are ready to rANS encode a message comprising one or more symbols taken from our symbol list. In our toy example, lets encode the message (0, 1, 2, 0, 1, 3). The rANS algorithm takes the starting rANS state x = 1 and then sequentially encodes the symbols in our message into the state by iteratively modifying the state using the information calculated above and the following formula:

[0198] x' = ⌊x / f(s)⌋ • F + fc(s) + (x mod f(s))

[0199] J )

[0200] where, [•] indicates the floor operation, where x is the current rANS state, x' is the updated rANS state, (s) is the frequency of the symbol being encoded, F is the total frequency, fc(s) is the cumulative frequency up to the symbol being encoded,

[0201] That is, starting with the first symbol of our message: "0", we know that (0) = 2 and / c(0) = 0, giving:

[0202] 1

[0203] x’ = [-] • 10 + 0 + (1 mod 2) = 0 - 10 + 0 + 1 = 1

[0204] Thus, after this step, our new rANS state x = 1 (i.e. unchanged) and we check if we need to bit flush by checking if the rANS state is greater than the threshold. That is, we check if 1 >= 160, which is false, so no flush of bits into the bitstream is needed.

[0205] Next, we encode the second symbol of our message: "1", we know that (I) = 3 and / c(l) = 2, giving:

[0206] 1

[0207] x’ = [-] • 10 + 2 + (1 mod 3) = 0 - 10 + 2 + 1 = 3

[0208] Thus, after this step, our new rANS state x = 3 and we check if we need to bit flush by checking if the rANS state is greater than the threshold. That is, we check if 3 >= 160, which is false, so no flush of bits into the bitstream is needed.Improved rANS / WO

[0209] Next, we encode the third symbol of our message: "2", we know that (2) = 1 and / c(2) = 5, giving:

[0210] 3

[0211] x' = ⌊3 / 1⌋ • 10 + 5 + (3 mod 1) = 3 • 10 + 5 + 0 = 35

[0212]

[0213] Thus, after this step, our new rANS state x = 35 and we check if we need to bit flush by checking if the rANS state is greater than the threshold. That is, we check if 35 >= 160, which is false, so no flush of bits into the bitstream is needed.

[0214] Next, we encode the fourth symbol of our message: "0", we know that (0) = 2 and / c(0) = 0, giving:

[0215] 35

[0216] x' = ⌊35 / 2⌋ • 10 + 0 + (35 mod 2) = 17 • 10 + 0 + 1 = 171 Thus, after this step, our new rANS state x = 171 and we check if we need to bit flush by checking if the rANS state is greater than the threshold. That is, we check if 171 >= 160, which is true, so we can now flush bits from our rANS state into the bitstream to ensure it doesn't go over the value 160 (and thus stays below our desired bitsize).

[0217] In order to flush bits using our base 24, we output the lower 4 bits of the rANS state x = 171. That is, 171 in binary is 10101011 and its lower 4 bits are 1011, which we append to the bitstream, the remaining bits 1010 give us our new state x = 10, which can also be estimated by dividing the current state by our base 24, that is 171 / 16 « 10.68... ~ 10.

[0218] With the bits flushed we can continue and encode our fifth symbol of our message: " 1", we know that (I) = 3 andc(l) = 2, giving:

[0219] 10

[0220] x' = ⌊10 / 3⌋ • 10 + 2 + (10 mod 3) = 3 • 10 + 2 + 1 = 33

[0221]

[0222] Thus, after this step, our new rANS state x = 33 and we check if we need to bit flush by checking if the rANS state is greater than the threshold. That is, we check if 33 >= 160, which is false, so no flush of bits into the bitstream is needed.

[0223] Next, we encode the sixth symbol of our message: "3", we know that (3) = 4 and / c(3) = 6, giving:

[0224] 33

[0225] x' = ⌊33 / 4⌋ • 10 + 6 + (33 mod 4) = 8 • 10 + 6 + 1 = 87 Thus, after this step, our new rANS state x = 87 and we check if we need to bit flush by checking if the rANS state is greater than the threshold. That is, we check if 87 >= 160, which is false, so no flush of bits into the bitstream is needed.

[0226] With all symbols now encoded into the final rANS state x = 87, we can flush its bits into the bitstream (concatenating the new bits to the already existing bits 1011 from our earlierImproved rANS / WO

[0227] bit flushing). To do this, we can keep flushing in our base of 24until we get to a small value such as 1. That is:

[0228] Starting with x = 87, we check that 87 > 1, which is true, so we flush bits. The lower 4 bits of87 are 0111, so we write 0111 to the bitstream and get the remainder: x = [(87 / 16)] = 5. We again check that 5 > 1, which is true, so we flush bits again. That is, the lower 4 bits of 5 are 0101 so we write 0101 to the bitstream and check the remainder: x = [(5 / 16)] = 0. A final check confirms that 0 > 1 is false so we have fully encoded our message into the bitstream: 1011 0111 0101.

[0229] As described above, bit flushing is optional. In which case, the rANS state keeps getting larger until the whole message has been encoded into a single, final rANS state (i.e. some number) whose binary representation is the contents of the bitstream. Note also that it is advantageous for runtime to construct a lookup table to store the symbols, frequencies, total frequency, and cumulative frequency, rather than trying to calculate these on the fly. In the case of Al-based compression, these are all immediately available and / or estimatable from the latent representation, and / or some representation thereof, produced by the encoder neural network. Thus, rANS is particularly suited for use as the entropy encoder in an Al-based compression pipeline.

[0230] With our bitstream now created, we can transmit it to a receiving side of the pipeline. Upon receipt, we can do the reverse steps to decode the bitstream back into our original message by iteratively moving through the rANS states, popping out symbols from our message at each step. This is illustrated in the steps below.

[0231] That is, rANS decoding must reconstruct the original sequence and does so by recovering symbols in reverse order of how they were encoded. Since we encoded in the sequence (0, 1,2,0, 1,3) starting from an initial state, decoding will find the symbols in reverse: (3, 1,0,2, 1,0) as we unwind the transformations. After collecting them in reverse, the final set of symbols can be reversed to obtain the message in its original order. We first reconstruct the final rANS state x = 87 using the a priori information that we were flushing bits into the bitstream using base 24down to the value 1.

[0232] That is, given the bitstream 1011 0111 0101 and knowing that the flushing process outputs the lower 4 bits of x at each step and then reduces x as:

[0233] xnew= ⌊x / 16⌋

[0234] We reconstruct x by reversing this operation. We start from the last flushed 4 bits 0101 (which equals 5) and get:Improved rANS / WO

[0235] xnew= (xcurrent« 4) + b

[0236] Since xcurrent = 0, we get:

[0237] new = (0 « 4) + 5 = 5

[0238] The next 4 bits in our bitstream are 0111 (which equals 7), giving us:

[0239] xnew= (xcurrent« 4) + 7

[0240] Substituting xcurrent= 5, we get:

[0241] new = (5 « 4) + 7 = (5 • 16) + 7 = 80 + 7 = 87

[0242] Thus, we have reconstructed our final rANS state from the bitstream and have a set of unused bits (1011) that we can use later to inflate a rANS state (i.e. the inverse of the bit flushing).

[0243] Next, we sequentially pop a symbol out of the current rANS state by calculating:

[0244] r = x' mod F

[0245] and identifying which range of the total frequency r falls into, that is: fcs < r < fcs) + / (■$)• In this specific toy example with the symbols and frequencies identified above we get:

[0246] Symbol 0: [0,1]Symbol 1: [2,4]Symbol 2: [5]Symbol 3: [6,9]

[0247] That is, if r is between 0 and 1 we pop the symbol 0 into our message, if r is between 2 and 4 we pop the symbol 1 into our message, if r is 5 we pop the the symbol 2 into our message, and if r is between 6 and 9 we pop the symbol 3 into our message.

[0248] After popping a symbol from the state, we update the rANS state by applying the rANS decode formula:

[0249] x = f(s) • ⌊x' / F⌋ + (r - fc(s))

[0250] Applying this to our retrieved final rANS state x = 87 we get:

[0251] r = 87 mod 10 = 7

[0252] The value 7 is between 6 and 9, corresponding to the range for symbol 3, so we pop the symbol 3 into our message as the first decoded symbol. We then apply the rANS decode formula, knowing (3) = 4 and / c(3) = 6 to update the rANS state:

[0253] 87

[0254] x = 4 • ⌊87 / 10⌋ + (7 - 6) = 33

[0255]

[0256] Improved rANS / WO

[0257] Thus, our updated rANS state is x = 33 and we have the decoded message (3). Next, we check if we need to inflate the state with the remaining bits of the bitstream by checking if x =< F, that is is 33 =< 10, which is false so we can proceed to decoding the next symbol of our message:

[0258] r = 33 mod 10 = 3

[0259] The value 3 is between 2 and 4, corresponding to to the range for symbol 1, so we pop the symbol 1 into our message as the second decoded symbol, and apply the rANS decode formula, knowing f(1) = 3 and f_c(1) = 2 to update the rANS state:x = 3 · ⌊33 / 10⌋ + (3 - 2) = 10

[0260] Thus, our updated rANS state is x = 10 and we have the decoded message (3,1). Next we check if we need to inflate the state with the remaining bits of the bitstream by checking if x =< F, that is is 33 =< 10, which is true so we can add in the remaining bits 1011 before popping the next symbol. That is, we get:

[0261] x = x • base + (1011) = 10 • 16 + 11 = 160 + 11 = 171 So, the inflated state we will pop the next symbol from is x = 171. Note also we have used up all the remaining bits of the bitstream that we had periodically flushed into the bitstream during encoding. Thus, popping the next symbol out we get:

[0262] r = 171 mod 10 = 1

[0263] The value 1 is between 0 and 1, corresponding to the range for symbol 0, so we pop the symbol 0 into our message as the next decoded symbol. We then apply the rANS decode formula, knowing f(0) = 2 and f_c(0) = 0 to update the rANS state:x = 2 · ⌊17 / 10⌋ + (1 - 0) = 35

[0264]

[0265] Thus, our updated rANS state is x = 35 and we have the decoded message (3,1,0). We haven't got any bits left in the bitstream so we can proceed to the next symbol:

[0266] r = 35 mod 10 = 5

[0267] The value 5 is in the 5 range corresponding to the range for symbol 2, so we pop the symbol 2 into our message as the next decoded symbol. We then apply the rANS decode formula, knowing f(2) = 1 and f_c(2) = 5 to update the rANS state:x = 1 · ⌊35 / 10⌋ + (5 - 5) = 3

[0268] Thus, our updated rANS state is x = 3 and we have the decoded message (3, 1,0, 2). We haven't got any bits left in the bitstream so we can proceed to the next symbol:Improved rANS / WO

[0269] r = 3 mod 10 = 3

[0270] The value 3 is in the between 2 and 4 range corresponding to the range for symbol 1, so we pop the symbol 1 into our message as the next decoded symbol. We then apply the rANS decode formula, knowing f(1) = 3 and f_c(1) = 2 to update the rANS state:x = 3 · ⌊3 / 10⌋ + (3 - 2) = 1

[0271]

[0272] Thus, our updated rANS state is x = 1 and we have the decoded message (3, 1,0, 2,1). We haven't got any bits left in the bitstream so we can proceed to the next symbol:

[0273] r = 1 mod 10 = 1

[0274] The value 1 is in the between 0 and 1 range corresponding to the range for symbol 0, so we pop the symbol 0 into our message as the next decoded symbol. We then apply the rANS decode formula, knowing f(0) = 2 and f_c(0) = 0 to update the rANS state:x = 2 · ⌊1 / 10⌋ + (1 - 0) = 1

[0275]

[0276] Thus, our updated rANS state is x = 1 and we have the decoded message (3, 1,0, 2, 1,0). Note that x = 1 corresponds to the starting state rANS so we have now fully decoded the original message, albeit in reverse order. To finally retrieve the original message in the correct order we reverse the message to (0,1, 2, 0,1, 3) thereby completing the rANS decode (with bitflushing) operation.

[0277] Whilst the above toy example is based on an arbitrary toy message, the same principles apply to rANS encoding a latent representation produced by an encoder neural network of an Al-based compression pipeline. That is, the symbols are the range of values in a channel or channels of the latent representation and the frequencies are the counts of each value (i.e. the distribution of those values). The other metrics needed for rANS (i.e. the cumulative frequencies and total frequency) can be easily calculated from those counts or predicted (e.g., using an entropy model whose location, scale and / or range parameters are predicted using the hyper network, and so on). Thus, the toy example above can be lifted across and applied to losslessly encoding a latent representation of an input frame.

[0278] Note however in practice that encoding all of the values of a channel or channels of a latent representation as a single message is computationally slow. Instead, these values can first be divided into chunks, each treated as a standalone message, to be rANS encoded and decoded in parallel. Indeed, the final rANS states of all the parallel rANS encode operations can themselves be concatenated into a standalone message and rANS encoded into a higher level final rANS state and written into the bitstream. In which case, on decode, the part of theImproved rANS / WO

[0279] bitstream corresponding to the concatenated final rANS states is rANS decoded to reconstruct the lower level final rANS states, which can then be rANS decoded in parallel.

[0280] This parallel rANS approach allows rANS to be made arbitrarily parallelisable and fast while compute resources are available to create the desired number of parallel threads.

[0281] However, even when using a parallel rANS approach, there are still bit savings to be made.

[0282] One area where further savings can be made is in the number of bits used to store the final rANS state of the one or more rANS operations. That is, the final rANS state itself will take up some number of bits based on how large the number is that represents the state. For example, in a hypothetical example where a final rANS state is the number 2, we will need at least 2 bits (1 and 0) to store the state. Whereas if the final rANS state is the number 200, we will need at least 8 bits (11001000) to store the state, and so on. The size of rANS states has no limit so these can get very large for very long messages. As we do not know a priori how large a final rANS state is going to be, known rANS compression methods store final rANS states in a single data object using a predetermined number of bits chosen so as to be large enough to record the largest rANS states for whatever the use case is that rANS is being applied to. For example, this may comprise always storing the final rANS state as a single 32-, 48-, 64- bit (and so on) data object, even if the actual state being stored could be stored in far fewer bits. This naive approach may suffice in cases where a single message is being rANS encoded and / or where the resulting difference between the actual number of bits required and the naive 32-, 48-, or 64-bit (and so on) data object is negligible. However, in cases where a parallel rANS approach is being used, storing each final rANS state of the parallel rANS operations as a naive 32- or 64-bit object can result in very significant bit overheads.

[0283] To address this problem, the present disclosure introduces the concept of splitting the final rANS state into a plurality of chunks (that is, data objects) of a predetermine number of bits, leaving a final "remainder" chunk that can be stored with fewer bits than the predetermined number of bits, the difference between which represents a bit saving relative to storing the state as a single data object. Whilst this saving may be small if only applied to a single final rANS state, it very quickly adds up when applied to each final rANS state of many parallel rANS operations. This concept is illustrated below with a toy example.

[0284] Consider some final rANS state S. We do not know a priori how big the final rANS state will be before we have finished rANS encoding the input message but let's say we know empirically for this toy example scenario that S is always small enough to fit into a 64-bit data object. On the decode side, we equally naively assume the final rANS state will be whateverImproved rANS / WO

[0285] 64-bit data object is received. For example, if the final rANS state S = 0xl23456789ABCDEF0 (in hexadecimal), then our data object will be a 64-bit data object. If instead the final rANS state S = 0x12345678, then, even though it could be stored as a 32-bit object, the data object S is nevertheless padded e.g., with Os and stored as a 64-bit object. In this way, the data object storing the rANS state always takes up the predetermined number of bits, and this is assumed on both encode and decode side for consistency. As described above, this is not ideal because every final rANS state that could be stored with fewer bits has redundant padding bits. For example, in some Al-based compression pipelines, the final rANS states that arise from rANS encoding a latent representation of an input frame may typically be represented in 32-48 bits. Thus, naively storing these states as 64-bit objects wastes 16-32 bits. That is, whenever the state requires a data object that is not divisible by 8 for it to be represented (e.g., 33 bits, which is 1 more bit than 32 bits) then naively representing that number by e.g., a 64-bit (or two 32-bit objects) is a significant waste of bits.

[0286] Instead, the present disclosure stores S in some number n data objects, each represented by a number of bits smaller than that needed to fit the whole of S.

[0287] Returning to our toy example, if we expect our final rANS state to be representable by x bits or fewer, rather than always naively representing it as a x-bit data object (or multiple data smaller-than-x bit objects that cumulatively add up to x bits), we can instead split the final rANS state S into some number n of m-bit chunks and a remainder chunk that is representable in fewer than m-bits in order to reliase a bit saving. That is, m < x and x, n, m E Uk. A toy example of this is illustrated below.

[0288] Consider the following final rANS state S = 0x123456789ABC. We separate this rANS state into n 32-bit chunks, and a remainder 16-bit chunk. In this case we get a single 32-bit chunk S1, leaving a remainder 16-bit chunk S2

[0289] S = 0x12345678

[0290] S2= 0x9ABC

[0291] These chunks can be concatenated or otherwise joined in a bitstream (optionally using a simple flag of 1-2 bits to separate them). The size of S + S2= 48 bits (plus the bits of any optional separating flag). This is smaller than a single 64-bit object or two 32-bit objects, thus giving us a 12 bit saving relative to naively treating every final rANS state as a 64-bit data object or two 32-bit objects.

[0292] Alternatively, the final rANS state S in this toy example may be separated into three 16-bit chunks (leaving no remainder), or six 8-bit chunks (also leaving no remainder), and so on.Improved rANS / WO

[0293] The smaller the chunk sizes, the smaller the remainder (if any) can be and thus the greater the bit saving. However, a trade off is that the more chunks there are, the greater the number of any optional separating flags there may be. The remainder chunk, if present, may be any number of bits in size that are smaller than the size of the other chunks. For example, if the remainder chunk is represented by k bits, then k < m. The specific value of k may thus depend on the chosen value of m and on the typical size of the final rANS states of the use case the present disclosure is being applied to. For example, if it is expected from empirical testing that a collection of final rANS states are always representable by data objects of 32-48 bits, and m (i.e. the size of each chunk) is chosen to be 32 bits, then k may be chosen to be 16 bits, so that the one chunk of 32 bits plus the remainder chunk of 16 bits together allow states of 48 bits in size to be represented. More generally, k is chosen so that the n chunks of size m plus the remainder chunk of size k can together always represent the largest final rANS state in the collection of states created, which is use case dependent.

[0294] When the data objects are received on the decode side, the reverse operations of whatever splitting operation was performed to create the chunks is performed to reconstruct each chunk, which can then be combined in order to reconstruct the final state S.

[0295] Practically, the separation of the final rANS state may be achieved by applying an XOR mod 2moperation (where m is the size of the chunk in bits) to the final rANS state S, or using any other suitable operation that separates the final rANS state into n chunks of size m bits. For example, in some cases the final rANS state may have a relatively high minimum bit size (e.g., 224). This value may be subtracted from the final state to realise a bit saving. This approach has substantively the same effect as the XOR operation.

[0296] Another area where further bit savings can be made is in the amount of metadata included in the bitstream to indicate certain parameters indicative of one or more properties of the rANS state(s) and / or how they have been encoded.

[0297] For example, when using parallel rANS, each final rANS state is represented by some number of bits. A flag or metadata indicating this number for each final rANS state may be written into the bitstream to indicate to the decoder on decoding where each final rANS state starts and ends in the bitstream. The data object representing this flag or metadata takes up some number of bits as well. For example, it may be an 8-, 16-, 32-bit or more data object (depending on the size(s) of the final rANS state(s)). In a similar manner as described above, when using parallel rANS methods, the overhead from this and other kinds of metadata associated with each final rANS state can be very high, particularly when there are many parallel threads. ToImproved rANS / WO

[0298] at least somewhat reduce this overhead, the flag or metadata indicating the size or length (in bits) of the last final rANS state encoded in the bitstream is omitted and instead the decoder is configured to continue rANS decoding until it reaches the end of the part of the bitstream in which the final rANS states are contained, thereby giving a 8-, 16-, 32- or more bit saving. In the context of Al-based compression, this may be a 8-, 16-, 32- or more bit saving per frame.

[0299] Another area where further bit savings can be made is in the approach used to handle the encoding of outlier symbols in the distribution of symbols of the message being encoded. To introduce this concept, we first describe what outliers are in the context of rANS encoding. As will be appreciated, rANS encoding uses a known distribution of a set of symbols used in the underlying message to efficiently entropy encode that message. When the message contains only a small number of symbols all roughly equally likely (a dense distribution of symbols), rANS encoding is a very effective lossless encoding technique. However, if the distribution has a small number of rare symbols, the distribution becomes sparse and it takes a disproportionate number of bits to encode this rare, sparse symbols with rANS. These symbols are referred to herein as outliers. More specifically, outliers require a greater rANS range to be encoded, increasing the size of the final rANS state. In the context of Al-based compression, a latent representation may be mostly representable by some tensor of pixel values, for example RGB channels with values between 0 to 255. If the tensor contains mostly values between 0-5, but there are some sparse, rare symbols such as a few occurrences of outlier values between 245-255, for example:

[0300] 0,0, 0,0, 1,5, 1,4, 3,0,...,255,4,1,3,247,...

[0301] then the rANS range has to be large to capture the full range of symbols 0-255. But for the outliers, the rANS range would be able to be 0-5, and the bitstream would be able to be significantly smaller in size.

[0302] To address this problem, outliers may be identified in the bitstream heuristically or statistically or in any other way and their position identified with an outlier flag, or their values replaced by an outlier flag. All the outliers can then be encoded using a different lossless encoding method such as Golomb encoding, or other loss encoding method, or simply written to the bitstream directly (e.g., where slower encode and decode runtimes are of importance to the use case). The Golomb (or other lossless encoding method) encoded outliers can be concatenated to the rANS encoded bitstream portion, giving a partially rANS encoded and partially Golomb (or other method) encoded bitstream. Using the toy example above, the sequence of RGB channel values:

[0303] 0,0, 0,0, 1,5, 1,4, 3,0,...,255,4,1,3,247,...Improved rANS / WO

[0304] may have the outlier values of 255 and 247 replaced by, for example, the flag 6 (which is whatever the actual rANS range of the distribution of values is, in this case 0-5 plus 1). This means the tensor being rANS encoded in this toy example would be:

[0305] 0,0, 0,0, 1,5, 1,4, 3,0,...,6,4,1, 3, 6,...

[0306] The outlier values are collected in order:

[0307] 255,247,...

[0308] and Golomb encoded into the bitstream. Alternatively, these outliers may be written directly to the bitstream without any encoding.

[0309] During decoding, each time an outlier flag value is found in the rANS encoded part of the bitstream, a next, unsed outlier in the Golomb encoded part of the bitstream that is Golomb decoded is inserted in the place of the flag to reconstruct the original tensor. Whilst the metadata associated with the number of outliers may be written into the bitstream, by ensuring the outliers are encoded and decoded in the order they are found in the rANS encoded message, there will always be a one-to-one ordered correspondence between the outliers and outlier flags so it is not necessary to write the number of outliers or other such metadata into the bitstream, thereby making another 8-bit, 16-bit, 32- or more bit saving per final rANS state. As above, when using parallel rANS methods, this saving per final rANS state can be quite substantial.

[0310] For completeness, the compression efficiency and other parameters of Golomb encoding may be predetermined, or optimised a priori based on typical distributions of outliers in the latent representations on which Golomb encoding is to be used. It is also envisaged that alternatives to Golomb encoding may be used, for example run-length encoding, Huffman coding, and / or other methods. In the present case, the combination of (parallel) rANS encoding but with Golomb encoding for outliers is chosen as it has an overall low runtime overhead which is particularly advantageous for use in real- or near real-time Al-based compression, where resources are already constrained by the heavy compute requirements of the lossy part of the pipeline and increases in runtime are generally to be avoided.

[0311] Taking each of the above bit saving methods into account results in a typical bit-saving in an Al-based compression pipeline of around 32-48 bits per final rANS state, and around 16-bits per latent representation where the final rANS state chunks are separated using a XOR mod 232operation where typical latent representation final rANS states are represented with data objects that are 32-40 bits in size.

[0312] Figure 5 illustrates constructing a bitstream using the above methods together. As already described, it is envisaged that each of the above methods may be used alone or together. The toy example in Figure 5 uses them together for the sake of illustration only.Improved rANS / WO

[0313] The data objects representing a latent representation 500a of an Al-based compression pipeline (for example, one or more tensors representing values the latent representation produced from an input frame in an RGB or YUV colourspace) are separated into chunks and processed using a parallel rANS approach such as that desribed above to produce a plurality of final rANS states 500b. In this toy example there are four final rANS states.

[0314] In order to avoid storing the final rANS states as high-bit (e.g., 64-bit) data objects, an XOR mod 232operation is performed to separate each final state into a plurality of 32-bit sub states data objects and a (or no) 16-bit remainder state. In Figure 5, the first final rANS state fits entirely into a single 32-bit data object 501, resulting in a bit saving of 32 bits, represented by the dotted line 502. The second final rANS state fits into a one 32-bit sub state 503 with a 16-bit remainder state 504, representing a 16-bit saving. The third final rANS state requires two 32-bit data objects 505, 506, so no saving is made here. The fourth and final rANS state can again be represented by a single 32-bit data object 507, representing a 16-bit saving.

[0315] Thus, the latent representation is represented by 32 + 32 + 16 + 32 + 32 + 32 = 176 bits which is significantly smaller than the 256 bits that would have been needed if each final rANS state was captured naively by four 64-bit data objects.

[0316] The data objects are then concatenated into a first bitstream portion 508, in this case joined with a 16-bit flag indicating their respective bit lengths 508a, 508b, 508c. However, as described above, to make a 16-bit saving, the flag associated with the final state 508d is omitted.

[0317] Also shown in Figure 5 are the identified outliers 509a, 509b, 509c, which are Golomb (or other entropy encoding method) encoded into a second bitstream portion 510, which is combined with the first bitstream portion to produce the combined bitstream. Not shown are any additional metadata or other parameters that may also be included in the bitstream, for example one or more separators indicating where the rANS encoded portion and the outlier portion of the bitstream starts and ends and so on. Also, not shown in Figure 5 is an optional step of rANS encoding the some or all of the final rANS states 500b into a higher level final rANS state, before joining whatever data objects are the result of that step into the first bit stream portion.

[0318] On the decode side, the reverse operations are performed. That is, the bitstream is received, the first and second portions of the bitstream are identified and the final rANS states are decoded in using the rANS decoding steps described earlier herein. After each final rANS state is decoded and the latent representation is being reconstructed, any outlier flags are replaced with the Golomb decoded values. These may all be decoded in one go beforeImproved rANS / WO

[0319] performing the rANS decoding, or these may be decoded as / when the next outlier flag is reached when reconstructing the latent representation.

[0320] The subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. The subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory program carrier for execution by, or to control the operation of, data processing apparatus. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. The computer storage medium is not, however, a propagated signal.

[0321] The term "data processing apparatus" encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0322] A computer program (which may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on oneImproved rANS / WO

[0323] computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

[0324] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).

[0325] Computers suitable for the execution of a computer program include, by way of example, can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a VR headset, a game console, a Global Positioning System (GPS) receiver, a server, a mobile phones, a tablet computer, a notebook computer, a music player, an e-book reader, a laptop or desktop computer, a PDAs, a smart phone, or other stationary or portable devices, that includes one or more processors and computer readable media, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0326] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

[0327] The subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combinationImproved rANS / WO

[0328] of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (" LAN") and a wide area network (" WAN"), e.g., the Internet.

[0329] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0330] While this specification contains many specific implementation details, these should be construed as descriptions of features that may be specific to particular examples of particular inventions. Certain features that are described in this specification in the context of separate examples can also be implemented in combination in a single example. Conversely, various features that are described in the context of a single example can also be implemented in multiple examples separately or in any suitable sub-combination.

[0331] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the examples described above should not be understood as requiring such separation in all examples, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0332] Finally, it will be appreciated that the concept of using rANS encoding in combination with another type of encoding, such as Golomb encoding, run-length encoding, Huffman coding, and so on, is envisaged to provide standalone advantages in that only those parts of the latent representation(s) which can be efficiently rANS encoded end up being rANS encoded, and those parts of the latent representation(s) where other types of lossless encoding are advantageous can be losslessly encoded with the other type of lossless encoding.

Claims

Improved rANS / WOCLAIMS1. A method for lossy image or video encoding, transmission and decoding, the method comprising:receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latent representation;range asymetric numeral system (rANS) encoding the latent representation to produce a bitstream and transmitting the bitstream to a second computer system;at the second computer system, rANS decoding the bitstream to produce the latent representation;decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image, and wherein rANS encoding comprises producing a final rANS state, and splitting the final rANS state into a plurality of data objects, wherein at least one of the data objects is stored using fewer bits than the other data objects.

2. The method of claim 1, wherein producing the bitstream comprises writing the plurality of data objects to the bitstream.

3. The method of claim 1 or 2, wherein splitting the final rANS state comprises performing one or more bitwise XOR operations on the final rANS state to produce the plurality of data objects, and wherein the at least one of the data objects is stored using fewer bits comprises a remainder of the one or more bitwise XOR operations.

4. The method of any of claims 1 to 3, wherein the least one data object stored using fewer bits is stored using 16 bits or 8 bits.

5. The method of claim 4, wherein the other of the plurality of data objects are stored using 64 bits or 32 bits.

6. The method of any of claims 1 to 5, comprising, at the first computer system, modifying the latent representation by identifying values of the latent representation outside of a range, and substituting identified values with a flag comprising a value inside of the range.

7. The method of any of claims 1 to 6, comprising at the first computer system, while rANS encoding, modifying the latent representation by identifying values of the latent representationImproved rANS / WOoutside of a range, removing the identified values from the latent representation, and writing one or more position flags into the bitstream indicative of positions of removed identified values.

8. The method of claim 6, comprising, at the second computer system, before the decoding with the second trained neural network, modifying the latent representation by substituting the flags with the identified values.

9. The method of claim 7, comprising, at the second computer system, while rANS decoding, inserting the identified values into the latent representation at the positions of the removed identified values.

10. The method of claim 6, comprising, at the first computer system, losslessly encoding the identified values into the bitstream, and, at the second computer system, losslessly decoding the bitstream to produce the identified values.

11. The method of claim 10, wherein the lossless encoding and decoding of the identified values comprises Golomb encoding and decoding.

12. The method of any of claims 1 to 11, comprising writing metadata into the bitstream indicative of respective lengths of all but one of the plurality of data objects.

13. The method of any of claims 1 to 12, wherein rANS decoding comprises: joining the plurality of data objects to produce the final rANS state.

14. A method for lossy image or video encoding and transmission, the method comprising: receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latent representation;range asymetric numeral system (rANS) encoding the latent representation to produce a bitstream and transmitting the bitstream to a second computer system;wherein rANS encoding comprises producing a final rANS state, and splitting the final rANS state into a plurality of data objects, wherein at least one of the data objects is stored using fewer bits than the other data objects.

15. A method for lossy image or video receipt and decoding, the method comprising:Improved rANS / WOat a second computer system, rANS decoding a bitstream to produce a latent representation, the bitstream produced by encoding an input image at a first computer system using a first trained neural network to produce a latent representation, and rANS encoding the latent representation to produce the bitstream, wherein rANS encoding comprises: producing a final rANS state, and splitting the final rANS state into a plurality of data objects, wherein at least one of the data objects is stored using fewer bits than the other data objects;at the second computer system, rANS decoding the bitstream to produce the latent representation; anddecoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image.

16. The method of claim 14 or 15, wherein producing the bitstream comprises writing the plurality of data objects to the bitstream.

17. The method of any of claims 14 to 16, wherein splitting the final rANS state comprises performing one or more bitwise XOR operations on the final rANS state to produce the plurality of data objects, and wherein the at least one of the data objects is stored using fewer bits comprises a remainder of the one or more bitwise XOR operations.

18. The method of any of claims 14 to 17, wherein the least one data object stored using fewer bits is stored using 16 bits or 8 bits.

19. The method of claim 18, wherein the other of the plurality of data objects are stored using 64 bits or 32 bits.

20. The method of any of claims 14 to 19, comprising, at the first computer system, modifying the latent representation by identifying values of the latent representation outside of a range, and substituting identified values with a flag comprising a value inside of the range.

21. The method of any of claims 14 to 20, comprising at the first computer system, while rANS encoding, modifying the latent representation by identifying values of the latent representation outside of a range, removing the identified values from the latent representation, and writing one or more position flags into the bitstream indicative of positions of removed identified values.Improved rANS / WO22. The method of claim 20, comprising, at the second computer system, before the decoding with the second trained neural network, modifying the latent representation by substituting the flags with the identified values.

23. The method of claim 21, comprising, at the second computer system, while rANS decoding, inserting the identified values into the latent representation at the positions of the removed identified values.

24. The method of claim 20, comprising, at the first computer system, losslessly encoding the identified values into the bitstream, and, at the second computer system, losslessly decoding the bitstream to produce the identified values.

25. The method of claim 24, wherein the losslessly encoding and decoding of the identified values comprises Golomb encoding and decoding.

26. The method of any of claims 14 to 25, comprising writing metadata into the bitstream indicative of respective lengths of all but one of the plurality of data objects.

27. The method of any of claims 14 to 26, wherein rANS decoding comprises: joining the plurality of data objects to produce the final rANS state.

28. A method for lossy image or video encoding, transmission and decoding, the method comprising:receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latent representation;range asymetric numeral system (rANS) encoding the latent representation to produce a bitstream and transmitting the bitstream to a second computer system;at the second computer system, rANS decoding the bitstream to produce the latent representation;decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image; and wherein rANS encoding comprises producing a final rANS state, and splitting the final rANS state into a plurality of data objects; and wherein producing the bitstream comprises writing metadata into the bitstream indicative of respective lengths of all but one of the plurality of data objects.Improved rANS / WO29. The method of claim 28, wherein rANS decoding comprises: joining the plurality of data objects to produce the final rANS state.

30. The method of claim 28 or 29, wherein producing the bitstream comprises writing the plurality of data objects to the bitstream.

31. The method of any of claims 28 to 30, wherein splitting the final rANS state comprises performing one or more bitwise XOR operations on the final rANS state to produce the plurality of data objects, and wherein the at least one of the data objects is stored using fewer bits comprises a remainder of the one or more bitwise XOR operations.

32. The method of any of claims 28 to 31, wherein the least one data object stored using fewer bits is stored using 16 bits or 8 bits.

33. The method of claim 32, wherein the other of the plurality of data objects are stored using 64 bits or 32 bits.

34. The method of any of claims 28 to 33, comprising, at the first computer system, modifying the latent representation by identifying values of the latent representation outside of a range, and substituting identified values with a flag comprising a value inside of the range.

35. The method of any of claims 28 to 34, comprising at the first computer system, while rANS encoding, modifying the latent representation by identifying values of the latent representation outside of a range, removing the identified values from the latent representation, and writing one or more position flags into the bitstream indicative of positions of removed identified values.

36. The method of claim 34, comprising, at the second computer system, before the decoding with the second trained neural network, modifying the latent representation by substituting the flags with the identified values.

37. The method of claim 35, comprising, at the second computer system, while rANS decoding, inserting the identified values into the latent representation at the positions of the removed identified values.Improved rANS / WO38. The method of claim 34, comprising, at the first computer system, losslessly encoding the identified values into the bitstream, and, at the second computer system, losslessly decoding the bitstream to produce the identified values.

39. The method of claim 35, wherein the lossless encoding and decoding of the identified values comprises Golomb encoding and decoding.

40. A method for lossy image or video encoding and transmission, the method comprising: receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latent representation;range asymetric numeral system (rANS) encoding the latent representation to produce a bitstream and transmitting the bitstream to a second computer system;wherein rANS encoding comprises producing a final rANS state, and splitting the final rANS state into a plurality of data object; and wherein the method comprises writing metadata into the bitstream indicative of respective lengths of all but one of the plurality of data objects.

41. A method for lossy image or video receipt and decoding, the method comprising:at a second computer system, range asymetric numeral system (rANS) decoding a bitstream to produce a latent representation, the bitstream produced by encoding an input image at a first computer system using a first trained neural network to produce a latent representation, and rANS encoding the latent representation to produce the bitstream, wherein rANS encoding comprises producing a final rANS state, and splitting the final rANS state into a plurality of data object; and the producing the bitstream comprises writing metadata into the bitstream indicative of respective lengths of all but one of the plurality of data objects;at the second computer system, rANS decoding the bitstream to produce the latent representation; anddecoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image.

42. The method of claim 40 or 41, wherein rANS decoding comprises: joining the plurality of data objects to produce the final rANS state.

43. The method of any of claims 40 to 42, wherein producing the bitstream comprises writing the plurality of data objects to the bitstream.Improved rANS / WO44. The method of any of claims 40 to 43, wherein splitting the final rANS state comprises performing one or more bitwise XOR operations on the final rANS state to produce the plurality of data objects, and wherein the at least one of the data objects is stored using fewer bits comprises a remainder of the one or more bitwise XOR operations.

45. The method of any of claims 40 to 44, wherein the least one data object stored using fewer bits is stored using 16 bits or 8 bits.

46. The method of claim 45, wherein the other of the plurality of data objects are stored using 64 bits or 32 bits.

47. The method of any of claims 40 to 46, comprising, at the first computer system, modifying the latent representation by identifying values of the latent representation outside of a range, and substituting identified values with a flag comprising a value inside of the range.

48. The method of any of claims 40 to 47, comprising at the first computer system, while rANS encoding, modifying the latent representation by identifying values of the latent representation outside of a range, removing the identified values from the latent representation, and writing one or more position flags into the bitstream indicative of positions of removed identified values.

49. The method of claim 47, comprising, at the second computer system, before the decoding with the second trained neural network, modifying the latent representation by substituting the flags with the identified values.

50. The method of claim 48, comprising, at the second computer system, while rANS decoding, inserting the identified values into the latent representation at the positions of the removed identified values.

51. The method of claim 47, comprising, at the first computer system, losslessly encoding the identified values into the bitstream, and, at the second computer system, losslessly decoding the bitstream to produce the identified values.

52. The method of claim 48, wherein the lossless encoding and decoding of the identified values comprises Golomb encoding and decoding.Improved rANS / WO53. A method for lossy image or video encoding, transmission and decoding, the method comprising:receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latent representation;range asymetric numeral system (rANS) encoding the latent representation to produce a bitstream, the rANS encoding comprising modifying the latent representation based on a range of values of the latent representation;transmitting the bitstream to a second computer system;at the second computer system, rANS decoding the bitstream to produce the latent representation, the rANS decoding comprising modifying the latent representation based on the range of values of the latent representation;decoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image.

54. The method of claim 53, wherein the modifying the latent representation comprises identifying values of the latent representation outside of the range, and substituting identified values with a flag comprising a value inside of the range.

55. The method of claim 54, comprising, at the first computer system, losslessly encoding the identified values into the bitstream, and, at the second computer system, losslessly decoding the bitstream to produce the identified values.

56. The method of claim 55, comprising, at the second computer system, before the decoding with the second trained neural network, modifying the latent representation by substituting the flags with the identified values.

57. The method of claim 55 or 56, wherein the losslessly encoding and decoding of the identified values comprises Golomb encoding and decoding.

58. The method of claim 53, comprising at the first computer system modifying the latent representation by identifying values of the latent representation outside of the range, removing the identified values from the latent representation, and writing one or more position flags into the bitstream indicative of positions of removed identified values.Improved rANS / WO59. The method of claim 58, comprising at the second computer system inserting the identified values into the latent representation at the positions of the removed identified values.

60. A method for lossy image or video encoding and transmission, the method comprising: receiving an input image at a first computer system; andencoding the input image using a first trained neural network to produce a latent representation;range asymetric numeral system (rANS) encoding the latent representation to produce a bitstream, the rANS encoding comprising modifying the latent representation based on a range of values of the latent representation; andtransmitting the bitstream to a second computer system.

61. A method for lossy image or video receipt and decoding, the method comprising:at a second computer system, range asymetric numeral system (rANS) decoding a bitstream to produce a latent representation, the bitstream produced by encoding an input image at a first computer system using a first trained neural network to produce a latent representation, and rANS encoding the latent representation to produce the bitstream, the rANS encoding comprising modifying the latent representation based on a range of values of the latent representation;wherein the rANS decoding comprises modifying the latent representation based on the range of values of the latent representation; anddecoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image.

62. The method of claim 60 or 61, wherein the modifying the latent representation comprises identifying values of the latent representation outside of the range, and substituting identified values with a flag comprising a value inside of the range.

63. The method of claim 62, comprising, at the first computer system, losslessly encoding the identified values into the bitstream, and, at the second computer system, losslessly decoding the bitstream to produce the identified values.

64. The method of claim 63, comprising, at the second computer system, before the decoding with the second trained neural network, modifying the latent representation by substituting the flags with the identified values.Improved rANS / WO65. The method of claim 63 or 64, wherein the lossless encoding and decoding of the identified values comprises Golomb encoding and decoding.

66. The method of claim 60 or 61, comprising at the first computer system modifying the latent representation by identifying values of the latent representation outside of the range, removing the identified values from the latent representation, and writing one or more position flags into the bitstream indicative of positions of removed identified values.

67. The method of claim 66, comprising at the second computer system inserting the identified values into the latent representation at the positions of the removed identified values.

68. A method for lossy image or video encoding, transmission and decoding, the method comprising:receiving an input image at a first computer system;encoding the input image using a first trained neural network to produce a latent representation;range asymetric numeral system (rANS) encoding a first portion of the latent representation and entropy encoding a second portion of the latent representation using a second entropy encoding method to produce a bitstream and transmitting the bitstream to a second computer system;at the second computer system, rANS decoding and entropy decoding, using a second entropy decoding method, the bitstream to produce the first and second portions of the latent representation, and combining the first and second portions to produce the latent representation; anddecoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image.

69. The method of claim 68, wherein the second entropy encoding and decoding methods comprise non-rANS entropy encoding and decoding methods.

70. The method of claim 68 or 69, wherein the non-rANS entropy encoding and decoding methods comprises Golomb encoding and decoding.

71. A method for lossy image or video encoding and transmission, the method comprising: receiving an input image at a first computer system;Improved rANS / WOencoding the input image using a first trained neural network to produce a latent representation;range asymetric numeral system (rANS) encoding a first portion of the latent representation and entropy encoding a second portion of the latent representation using a second entropy encoding method to produce a bitstream and transmitting the bitstream to a second computer system.

72. A method for lossy image or video receipt and decoding, the method comprising:receiving a bitstream at a second computer system, the bitstream produced by range asymetric numeral system (rANS) encoding a first portion of a latent representation and entropy encoding a second portion of the latent representation using a second entropy encoding method to produce the bitstream, the latent representation produced by encoding an input image using a first trained neural network,at the second computer system, rANS decoding and entropy decoding, using a second entropy decoding method, the bitstream to produce the first and second portions of a latent representation, and combining the first and second portions to produce the latent representation; anddecoding the latent representation using a second trained neural network to produce an output image, wherein the output image is an approximation of the input image.

73. The method of claim 71 or 72, wherein the second entropy encoding and decoding methods comprise non-rANS entropy encoding and decoding methods.

74. The method of any of claims 71 to 73, wherein the non-rANS entropy encoding and decoding methods comprises Golomb encoding and decoding.

75. A data processing apparatus configured to perform the method of any of claims 1 to 74.

76. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any of claims 1 to 74.

77. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of any of claims 1 to 74.

Images