Method and data processing system for lossy image or video encoding, transmission and decoding
The method and system leverage neural networks and hyper-networks for entropy encoding to address inefficiencies in existing lossy image and video compression, achieving improved data reduction and visual quality by optimizing rate and distortion trade-offs.
Patent Information
- Application Number
- PCT/US2026/027037
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2025-05-08
- Filing Date
- 2026-05-07
- Publication Date
- 2026-11-12
AI Technical Summary
Existing lossy image and video compression techniques face challenges in efficiently reducing data transmission demands while maintaining visual quality, particularly in handling spatial and temporal redundancies, with AI-based methods often resulting in poor compression outcomes.
A method and system utilizing neural networks for lossy image or video encoding and decoding, employing entropy encoding based on central tendency measures of pixel values, and incorporating hyper-networks for improved compression efficiency.
Enhances compression efficiency by minimizing data transmission while maintaining visual quality through end-to-end learning of entropy parameters and hyper-networks, optimizing the trade-off between rate and distortion.
Smart Images

Figure US2026027037_12112026_PF_FP_ABST
Abstract
Description
[0001] METHOD AND DATA PROCESSING SYSTEM FOR LOSSY IMAGE OR VIDEO ENCODING, TRANSMISSION AND DECODING
[0002] CROSS REFERENCE TO RELATED APPLICATIONS
[0003] This application claims priority to United Kingdom Patent Application No. GB 2507042.6, filed May 08, 2025, which is incorporated herein by reference in its entirety.
[0004] BACKGROUND
[0005] This invention relates to a method and system for lossy image or video encoding, transmission and decoding, a method, apparatus, computer program and computer readable storage medium for lossy image or video encoding and transmission, and a method, apparatus, computer program and computer readable storage medium for lossy image or video receipt and decoding.
[0006] There is increasing demand from users of communications networks for images and video content. Demand is increasing not just for the number of images viewed, and for the playing time of video; demand is also increasing for higher resolution content. This places increasing demand on communications networks and increases their energy use because of the larger amount of data being transmitted.
[0007] To reduce the impact of these issues, image and video content is compressed for transmission across the network. The compression of image and video content can be lossless or lossy compression. In lossless compression, the image or video is compressed such that all of the original information in the content can be recovered on decompression. However, when using lossless compression there is a limit to the reduction in data quantity that can be achieved. In lossy compression, some information is lost from the image or video during the compression process. Known compression techniques attempt to minimise the apparent loss of information by the removal of information that results in changes to the decompressed image or video that is not particularly noticeable to the human visual system. JPEG, JPEG2000, AVC, HEVC and AVI are examples of compression processes for image and / or video files.
[0008] In general terms, known lossy image compression techniques use the spatial correlations between pixels in images to remove redundant information during compression. For example, in an image of a blue sky, if a given pixel is blue, there is a high likelihood that the neighbouring pixels, and their neighbouring pixels, and so on, are also blue. There is accordingly no need to retain all the raw pixel data. Instead, we can retain only a subset of the pixels which take up fewer bits and infer the pixel values of the other pixels using information derived from spatial correlations.A similar approach is applied in known lossy video compression techniques. That is, spatial correlations between pixels allow the removal of redundant information during compression. However, in video compression, there is further information redundancy in the form of temporal correlations. For example, in a video of an aircraft flying across a blue-sky background, most of the pixels of the blue sky do not change at all between frames of the video. The most of the blue sky pixel data for the frame at position t = 0 in the video is identical to that at position t = 10. Storing this identical, temporally correlated, information is inefficient. Instead, only the blue sky pixel data for a subset of the frames is stored and the rest are inferred from information derived from temporal correlations.
[0009] In the realm of lossy video compression in particular, the removal of redundant temporally correlated information in a video sequence is known as inter-frame redundancy.
[0010] One technique using inter-frame redundancy that is widely used in standard video compression algorithms involves the categorization of video frames into three types: I-frames, P-frames, and B-frames. Each frame type carries distinct properties concerning their encoding and decoding process, playing different roles in achieving high compression ratios while maintaining acceptable visual quality.
[0011] 1-frames, or intra-coded frames, serve as the foundation of the video sequence. These frames are self-contained, each one encoding a complete image without reference to any other frame. In terms of compression, 1-frames are least compressed among all frame types, thus carrying the most data. However, their independence provides several benefits, including being the starting point for decompression and enabling random access, crucial for functionalities like fast-forwarding or rewinding the video.
[0012] P-frames, or predictive frames, utilize temporal redundancy in video sequences to achieve greater compression. Instead of encoding an entire image like an I-frame, a P-frame represents the difference between itself and the closest preceding I- or P-frame. The process, known as motion compensation, identifies and encodes only the changes that have occurred, thereby significantly reducing the amount of data transmitted. Nonetheless, P-frames are dependent on previous frames for decoding. Consequently, any error during the encoding or transmission process may propagate to subsequent frames, impacting the overall video quality.
[0013] B-frames, or bidirectionally predictive frames, represent the highest level of compression. Unlike P-frames, B-frames use both the preceding and following frames as references in their encoding process. By predicting motion both forwards and backwards in time, B-frames encode only the differences that cannot be accurately anticipated from the previous and next frames, leading to substantial data reduction. Although this bidirectional prediction makes B-frames morecomplex to generate and decode, it does not propagate decoding errors since they are not used as references for other frames. Artificial intelligence (Al) based compression techniques achieve compression and decompression of images and videos through the use of trained neural networks in the compression and decompression process. Typically, during training of the neutral networks, the difference between the original image and video and the compressed and decompressed image and video is analyzed and the parameters of the neural networks are modified to reduce this difference while minimizing the data required to transmit the content. However, Al based compression methods may achieve poor compression results in terms of the appearance of the compressed image or video or the amount of information required to be transmitted.
[0014] An example of an Al based image compression process comprising a hyper-network is described in Balle, Johannes, et al. "Variational image compression with a scale hyperprior." arXiv preprint arXiv: 1802.01436 (2018), which is incorporated herein by reference.
[0015] An example of an Al based video compression approach is shown in Agustsson, E., Minnen, D., Johnston, N., Balle, J., Hwang, S. J., and Toderici, G. (2020), Scale-space flow for end-to-end optimized video compression. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 8503-8512), which is incorporated herein by reference.
[0016] A further example of an Al based video compression approach is shown in Mentzer, F., Agustsson, E., Balle, J., Minnen, D., Johnston, N., and Toderici, G. (2022, November). Neural video compression using gans for detail synthesis and propagation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVI (pp. 562-578), which is incorporated herein by reference.
[0017] SUMMARY
[0018] According to an aspect, there is provided a method for lossy image or video encoding and transmission, the method comprising the steps of receiving a first image of a sequence of images at a first computer system, with a first neural network, producing one or more latent representations associated with the first image, entropy encoding the one or more latent representations to produce a bitstream using one or more entropy parameters associated with the one or more latent representations and transmitting the bitstream to a second computer system, wherein one or more of the entropy parameters is based on a central tendency measure of one or more pixel values of the one or more latent representations.
[0019] According to an aspect, there is provided a data processing apparatus comprising a processor, the processer configured to perform receiving a first image of a sequence of images ata first computer system, with a first neural network, producing one or more latent representations associated with the first image, entropy encoding the one or more latent representations to produce a bitstream using one or more entropy parameters associated with the one or more latent representations and transmitting the bitstream to a second computer system, wherein one or more of the entropy parameters is based on a central tendency measure of one or more pixel values of the one or more latent representations.
[0020] Optionally wherein one or more of the entropy parameters is based on a mean of one or more pixel values of the latent representations.
[0021] Optionally comprising, for a plurality of pixel coordinates of at least one of the latent representations, calculating a mean of pixel values at and / or around each of the pixel coordinates to produce one or more entropy parameters associated with the at least one latent representation.
[0022] Optionally comprising using the calculated mean values as the one or more entropy parameters.
[0023] Optionally wherein the one or more entropy parameters comprise one or more scale values, and wherein the method comprises using the one or more scale values to perform range asymmetric numeral system (rANS) encoding of the one or more latent representations to produce the bitstream.
[0024] Optionally comprising, for at least one pixel coordinate of one of the latent representations performing said entropy encoding with a learned entropy parameter.
[0025] Optionally wherein the first neural network comprises an encoder neural network part, a hyper encoder neural network part and a hyper hyper encoder neural network part, and wherein the method comprises using the hyper encoder neural network part to produce a hyper latent representation of the first image and using the hyper hyper encoder neural network part to produce a hyper hyper latent representation of the first image, wherein the one or more latent representations comprises the hyper latent representation and the hyper hyper latent representation.
[0026] Optionally comprising computing the central tendency measure from one or more pixel values of the hyper hyper latent representation, and using the computed central tendency measure as the one or more entropy parameters associated with the hyper hyper latent representation.
[0027] Optionally wherein the first neural network comprises a decoder neural network part, a hyper decoder neural network part and a hyper hyper decoder neural network part, and wherein the method comprises using the hyper hyper decoder neural network part to decode the hyper hyper latent representation to produce the one or more entropy parameters associated with the hyper latent representation.Optionally wherein the first neural network comprises a plurality of convolution layers and activation layers,
[0028] Optionally wherein at least one of the convolution layers comprises a bitwise accurate convolution kernel.
[0029] According to an aspect, there is provided a method for lossy image or video receipt and decoding, the method comprising the steps of receiving a bitstream at a second computer system, the bitstream produced by entropy encoding one or more latent representations using one or more entropy parameters associated with the one or more latent representations, the one or more latent representations produced with a first neural network using a first image of a sequence of images entropy decoding the bitstream to produce the one or more latent representations using the one or more parameters and with a second neural network, decoding the one or more latent representations to produce an output image, wherein the output image is an approximation of the first image wherein one or more of the entropy parameters is based on a central tendency measure of one or more pixel values of the one or more latent representations.
[0030] According to an aspect, there is provided a data processing apparatus comprising a processor, the processer configured to perform receiving a bitstream at a second computer system, the bitstream produced by entropy encoding one or more latent representations using one or more entropy parameters associated with the one or more latent representations, the one or more latent representations produced with a first neural network using a first image of a sequence of images entropy decoding the bitstream to produce the one or more latent representations using the one or more parameters and with a second neural network, decoding the one or more latent representations to produce an output image, wherein the output image is an approximation of the first image, wherein one or more of the entropy parameters is based on a central tendency measure of one or more pixel values of the one or more latent representations.
[0031] Optionally wherein one or more of the entropy parameters is based on a mean of one or more pixel values of the latent representations.
[0032] Optionally comprising, for a plurality of pixel coordinates of at least one of the latent representations, calculating a mean of pixel values at and / or around each of the pixel coordinates to produce one or more entropy parameters associated with the at least one latent representation.
[0033] Optionally comprising using the calculated mean values as the one or more entropy parameters.
[0034] Optionally wherein entropy decoding the bitstream comprises: at the second computer system: calculating a mean of previously decoded pixel values around at least a first pixelcoordinate of one of the latent representations, and using said mean as an entropy parameter to entropy decode from the bitstream a pixel value for the first pixel coordinate.
[0035] Optionally wherein entropy decoding the bitstream comprises: repeating for further pixel coordinates of the one of the latent representations the steps of: calculating a mean of previously decoded pixel values around each further pixel coordinate and using each mean as an entropy parameter to entropy decode from the bitstream a pixel value for each said further pixel coordinates.
[0036] Optionally comprising, for at least one pixel coordinate of one of the latent representations performing said entropy decoding with a learned entropy parameter.
[0037] Optionally wherein the second neural network comprises a plurality of convolution layers and activation layers,
[0038] Optionally wherein at least one of the convolution layers comprises a bitwise accurate convolution kernel.
[0039] According to an aspect, there is provided a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out any of the above methods.
[0040] According to an aspect, there is provided a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out any of the above methods.
[0041] According to an aspect, there is provided a method for lossy image or video encoding and transmission, and decoding, the method comprising the steps of:
[0042] receiving a first image of a sequence of images at a first computer system;
[0043] with a first neural network, producing one or more latent representations associated with the first image;
[0044] entropy encoding the one or more latent representations to produce a bitstream using one or more entropy parameters associated with the one or more latent representations;
[0045] transmitting the bitstream to a second computer system;
[0046] entropy decoding the bitstream to produce the one or more latent representations using the one or more entropy parameters;
[0047] with a second neural network, decoding the one or more latent representations to produce an output image, wherein the output image is an approximation of the first image; and wherein the method comprises estimating one or more of the entropy parameters with a context model.According to an aspect, there is provided a method for lossy image or video encoding and transmission, and decoding, the method comprising the steps of:
[0048] receiving a first image of a sequence of images at a first computer system;
[0049] with a first neural network, producing one or more latent representations associated with the first image;
[0050] entropy encoding the one or more latent representations to produce a bitstream using one or more entropy parameters associated with the one or more latent representations;
[0051] transmitting the bitstream to a second computer system;
[0052] entropy decoding the bitstream to produce the one or more latent representations using the one or more entropy parameters;
[0053] with a second neural network, decoding the one or more latent representations to produce an output image, wherein the output image is an approximation of the first image;
[0054] wherein the first neural network and the second neural network comprise a plurality of convolution layers and activation layers; and
[0055] wherein at least one of the convolution layers comprises a bitwise accurate convolution kernel.
[0056] Optionally, the first neural network comprises an encoder neural network part, a hyper encoder neural network part and a hyper hyper encoder neural network part, and wherein the method comprises:
[0057] using the hyper encoder neural network part to produce a hyper latent representation of the first image; and
[0058] using the hyper hyper encoder neural network part to produce a hyper hyper latent representation of the first image,
[0059] wherein the one or more latent representations comprises the hyper latent representation and the hyper hyper latent representation.
[0060] Optionally, the first neural network and / or the second neural network comprises a decoder neural network part, a hyper decoder neural network part and a hyper hyper decoder neural network part.
[0061] Optionally, comprising using the hyper hyper decoder neural network part to decode the hyper hyper latent representation to produce the one or more entropy parameters associated with the hyper latent representation.
[0062] Optionally, at least one convolution layer of the hyper hyper decoder neural network part comprises a bitwise accurate convolution kernel.Optionally, the method comprises using the hyper decoder neural network part to decode the hyper latent representation to produce the one or more entropy parameters associated with the latent representation.
[0063] Optionally, at least one convolution layer of the hyper decoder neural network part comprises a bitwise accurate convolution kernel.
[0064] According to an aspect, there is provided a method for lossy image or video encoding and transmission, the method comprising the steps of:
[0065] receiving a first image of a sequence of images at a first computer system;
[0066] with a first neural network, producing one or more latent representations associated with the first image;
[0067] entropy encoding the one or more latent representations to produce a bitstream using one or more entropy parameters associated with the one or more latent representations; and transmitting the bitstream to a second computer system;
[0068] wherein the first neural network comprises a plurality of convolution layers and activation layers; and
[0069] wherein at least one of the convolution layers comprises a bitwise accurate convolution kernel.
[0070] According to an aspect, there is provided a method for lossy image or video receipt and decoding, the method comprising the steps of:
[0071] receiving a bitstream at a second computer system, the bitstream produced by entropy encoding one or more latent representations using one or more entropy parameters associated with the one or more latent representations, the one or more latent representations produced with a first neural network using a first image of a sequence of images;
[0072] entropy decoding the bitstream to produce the one or more latent representations using the one or more entropy parameters; and
[0073] with a second neural network, decoding the one or more latent representations to produce an output image, wherein the output image is an approximation of the first image;
[0074] wherein the first neural network comprises a plurality of convolution layers and activation layers; and
[0075] wherein at least one of the convolution layers comprises a bitwise accurate convolution kernel.
[0076] According to an aspect, there is provided a data processing apparatus configured to perform any of the above methods.According to an aspect, there is provided a computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out any of the above methods.
[0077] According to an aspect, there is provided a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out any of the above methods.
[0078] BRIEF DESCRIPTION OF THE DRAWINGS
[0079] Aspects of the invention will now be described by way of examples, with reference to the following figures in which:
[0080] Figure 1 illustrates an example of an image or video compression, transmission and decompression pipeline;
[0081] Figure 2 illustrates a further example of an image or video compression, transmission and decompression pipeline including a hyper-network;
[0082] Figure 3 illustrates an example of a video compression, transmission and decompression pipeline;
[0083] Figure 4 illustrates an example of a video compression, transmission and decompression system;
[0084] Figure 5 illustrates an example of a part of a video compression, transmission and decompression pipeline.
[0085] Figure 6 illustrates an example of a part of a video compression, transmission and decompression pipeline.
[0086] Figure 7 illustrates an example of a part of a video compression, transmission and decompression pipeline.
[0087] Figure 8 illustrates an example of a part of a video compression, transmission and decompression pipeline.
[0088] Figure 9 illustrates steps in part of a video compression, transmission and decompression pipeline.
[0089] Figure 10 illustrates an example of a part of a video compression, transmission and decompression pipeline.
[0090] Figure 11 illustrates an example of a part of a video compression, transmission and decompression pipeline.
[0091] Figure 12 illustrates an example of a part of a video compression, transmission and decompression pipeline.DETAILED DESCRIPTION OF THE DRAWINGS
[0092] Compression processes may be applied to any form of information to reduce the amount of data, or file size, required to store that information. Image and video information is an example of information that may be compressed. The file size required to store the information, particularly during a compression process when referring to the compressed file, may be referred to as the rate. In general, compression can be lossless or lossy. In both forms of compression, the file size is reduced. However, in lossless compression, no information is lost when the information is compressed and subsequently decompressed. This means that the original file storing the information is fully reconstructed during the decompression process. In contrast to this, in lossy compression information may be lost in the compression and decompression process and the reconstructed file may differ from the original file. Image and video files containing image and video data are common targets for compression.
[0093] In a compression process involving an image, the input image may be represented as x. The data representing the image may be stored in a tensor of dimensions H x W x C, where H represents the height of the image, W represents the width of the image and C represents the number of channels of the image. Each H x W data point of the image represents a pixel value of the image at the corresponding location. Each channel C of the image represents a different component of the image for each pixel which are combined when the image file is displayed by a device. For example, an image file may have 3 channels with the channels representing the red, green and blue component of the image respectively. In this case, the image information is stored in the RGB colour space, which may also be referred to as a model or a format. Other examples of colour spaces or formats include the CMKY and the YCbCr colour models. However, the channels of an image file are not limited to storing colour information and other information may be represented in the channels. As a video may be considered a series of images in sequence, any compression process that may be applied to an image may also be applied to a video. Each image making up a video may be referred to as a frame of the video.
[0094] The output image may differ from the input image and may be represented by x. The difference between the input image and the output image may be referred to as distortion or a difference in image quality. The distortion can be measured using any distortion function which receives the input image and the output image and provides an output which represents the difference between input image and the output image in a numerical way. An example of such a method is using the mean square error (MSE) between the pixels of the input image and the output image, but there are many other ways of measuring distortion, as will be known to the person skilled in the art. The distortion function may comprise a trained neural network.Typically, the rate and distortion of a lossy compression process are related. An increase in the rate may result in a decrease in the distortion, and a decrease in the rate may result in an increase in the distortion. Changes to the distortion may affect the rate in a corresponding manner. A relation between these quantities for a given compression technique may be defined by a ratedistortion equation.
[0095] Al based compression processes may involve the use of neural networks. A neural network is an operation that can be performed on an input to produce an output. A neural network may be made up of a plurality of layers. The first layer of the network receives the input. One or more operations may be performed on the input by the layer to produce an output of the first layer. The output of the first layer is then passed to the next layer of the network which may perform one or more operations in a similar way. The output of the final layer is the output of the neural network.
[0096] Each layer of the neural network may be divided into nodes. Each node may receive at least part of the input from the previous layer and provide an output to one or more nodes in a subsequent layer. Each node of a layer may perform the one or more operations of the layer on at least part of the input to the layer. For example, a node may receive an input from one or more nodes of the previous layer. The one or more operations may include a convolution, a weight, a bias and an activation function. Convolution operations are used in convolutional neural networks. When a convolution operation is present, the convolution may be performed across the entire input to a layer. Alternatively, the convolution may be performed on at least part of the input to the layer.
[0097] Each of the one or more operations is defined by one or more parameters that are associated with each operation. For example, the weight operation may be defined by a weight matrix defining the weight to be applied to each input from each node in the previous layer to each node in the present layer. In this example, each of the values in the weight matrix is a parameter of the neural network. The convolution may be defined by a convolution matrix, also known as a kernel. In this example, one or more of the values in the convolution matrix may be a parameter of the neural network. The activation function may also be defined by values which may be parameters of the neural network. The parameters of the network may be varied during training of the network.
[0098] Other features of the neural network may be predetermined and therefore not varied during training of the network. For example, the number of layers of the network, the number of nodes of the network, the one or more operations performed in each layer and the connections between the layers may be predetermined and therefore fixed before the training process takes place. These features that are predetermined may be referred to as the hyperparameters of the network. These features are sometimes referred to as the architecture of the network.To train the neural network, a training set of inputs may be used for which the expected output, sometimes referred to as the ground truth, is known. The initial parameters of the neural network are randomized and the first training input is provided to the network. The output of the network is compared to the expected output, and based on a difference between the output and the expected output the parameters of the network are varied such that the difference between the output of the network and the expected output is reduced. This process is then repeated for a plurality of training inputs to train the network. The difference between the output of the network and the expected output may be defined by a loss function. The result of the loss function may be calculated using the difference between the output of the network and the expected output to determine the gradient of the loss function. Back-propagation of the gradient descent of the loss function may be used to update the parameters of the neural network using the gradients dL / dy of the loss function. A plurality of neural networks in a system may be trained simultaneously through back-propagation of the gradient of the loss function to each network.
[0099] In the context of image or video compression, this type of system, where simultaneous training with back-propagation through each element or the whole network architecture may be referred to as end-to-end, learned image or video compression. Unlike in traditional compression algorithms that use primarily handcrafted, manually constructed steps, an end-to-end learned system learns itself during training what combination of parameters best achieves the goal of minimising the loss function. This approach is advantageous compared to systems that are not end-to-end learned because an end-to-end system has a greater flexibility to learn weights and parameters that might be counter-intuitive to someone handcrafting features.
[0100] It will be appreciated that the term "training" or "learning" as used herein means the process of optimizing an artificial intelligence or machine learning model, based on a given set of data. This involves iteratively adjusting the parameters of the model to minimize the discrepancy between the model's predictions and the actual data, represented by the above-described ratedistortion loss function.
[0101] The training process may comprise multiple epochs. An epoch refers to one complete pass of the entire training dataset through the machine learning algorithm. During an epoch, the model's parameters are updated in an effort to minimize the loss function. It is envisaged that multiple epochs may be used to train a model, with the exact number depending on various factors including the complexity of the model and the diversity of the training data.
[0102] Within each epoch, the training data may be divided into smaller subsets known as batches. The size of a batch, referred to as the batch size, may influence the training process. A smaller batch size can lead to more frequent updates to the model's parameters, potentially leading to fasterconvergence to the optimal solution, but at the cost of increased computational resources. Conversely, a larger batch size involves fewer updates, which can be more computationally efficient but might converge slower or even fail to converge to the optimal solution.
[0103] The learnable parameters are updated by a specified amount each time, determined by the learning rate. The learning rate is a hyperparameter that decides how much the parameters are adjusted during the training process. A smaller learning rate implies smaller steps in the parameter space and a potentially more accurate solution, but it may require more epochs to reach that solution. On the other hand, a larger learning rate can expedite the training process but may risk overshooting the optimal solution or causing the training process to diverge.
[0104] The training described herein may involve use of a validation set, which is a portion of the data not used in the initial training, which is used to evaluate the model's performance and to prevent overfitting. Overfitting occurs when a model learns the training data too well, to the point that it fails to generalize to unseen data. Regularization techniques, such as dropout or L1 / L2 regularization, can also be used to mitigate overfitting.
[0105] It will be appreciated that training a machine learning model is an iterative process that may comprise selection and tuning of various parameters and hyperparameters. As will be appreciated, the specific details, such as hyper parameters and so on, of the training process may vary and it is envisaged that producing a trained model in this way may achieved in a number of different ways with different epochs, batch sizes, learning rates, regularisations, and so on, the details of which are not essential to enabling the advantages and effects of the present disclosure, except where stated otherwise. The point at which an "untrained" neural network is considered be "trained" is envisaged to be case specific and depend on, for example, on a number of epochs, a plateauing of any further learning, or some other metric and is not considered to be essential in achieving the advantages described herein.
[0106] More details of an end-to-end, learned compression process will now be described. It will be appreciated that in some cases, end-to-end, learned compression processes may be combined with one or more components that are handcrafted or trained separately.
[0107] In the case of Al based image or video compression, the loss function may be defined by the rate distortion equation. The rate distortion equation may be represented by Loss = D + A * R, where D is the distortion function, is a weighting factor, and R is the rate loss. may be referred to as a lagrange multiplier. The langrange multiplier provides as weight for a particular term of the loss function in relation to each other term and can be used to control which terms of the loss function are favoured when training the network.In the case of Al based image or video compression, a training set of input images may be used. An example training set of input images is the KODAK image set (for example at www.cs.albany.edu / xypan / research / snr / Kodak.html). An example training set of input images is the IMAX image set. An example training set of input images is the Imagenet dataset (for example at www.image-net.org / download). An example training set of input images is the CLIC Training Dataset P ("professional") and M ("mobile") (for example at http : / / ch al 1 enge . compre ssi on . cc / tasks / ) .
[0108] An example of an Al based compression, transmission and decompression process 100 is shown in Figure 1. As a first step in the Al based compression process, an input image 5 is provided. The input image 5 is provided to a trained neural network 110 characterized by a function fg acting as an encoder. The encoder neural network 110 produces an output based on the input image. This output is referred to as a latent representation of the input image 5. In a second step, the latent representation is quantised in a quantisation process 140 characterised by the operation Q, resulting in a quantized latent. The quantisation process transforms the continuous latent representation into a discrete quantized latent. An example of a quantization process is a rounding function.
[0109] In a third step, the quantized latent is entropy encoded in an entropy encoding process 150 to produce a bitstream 130. The entropy encoding process may be for example, range or arithmetic encoding. In a fourth step, the bitstream 130 may be transmitted across a communication network.
[0110] In a fifth step, the bitstream is entropy decoded in an entropy decoding process 160. The quantized latent is provided to another trained neural network 120 characterized by a function g0acting as a decoder, which decodes the quantized latent. The trained neural network 120 produces an output based on the quantized latent. The output may be the output image of the Al based compression process 100. The encoder-decoder system may be referred to as an autoencoder.
[0111] Entropy encoding processes such as range or arithmetic encoding are typically able to losslessly compress given input data up to close to the fundamental entropy limit of that data, as determined by the total entropy of the distribution of that data. Accordingly, one way in which end-to-end, learned compression can minimise the rate loss term of the rate-distortion loss function and thereby increase compression effectiveness is to learn autoencoder parameter values that produce low entropy latent representation distributions. Producing latent representations distributed with as low an entropy as possible allows entropy encoding to compress the latent distributions as close to or to the fundamental entropy limit for that distribution. The lower the entropy of the distribution, the more entropy encoding can losslessly compress it and the lower the amount of data in the corresponding bitstream. In some cases where the latent representation isdistributed according to a gaussian or Laplacian distribution, this learning may comprise learning optimal location and scale parameters of the gaussian or Laplacian distributions, in other cases, it allows the learning of more flexible latent representation distributions which can further help to achieve the minimising of the rate-distortion loss function in ways that are not intuitive or possible to do with handcrafted features. Examples of these and other advantages are described in WO2021 / 220008 Al, which is incorporated herein by reference in its entirety.
[0112] Something which is closely linked to the entropy encoding of the latent distribution and which accordingly also has an effect on the effectiveness of compression of end-to-end learned approaches is the quantisation step. During inference, a rounding function may be used to quantise a latent representation distribution into bins of given sizes, a rounding function is not differentiable everywhere. Rather, a rounding function is effectively one or more step functions whose gradient is either zero (at the top of the steps) or infinity (at the boundary between steps). Back propagating a gradient of a loss function through a rounding function is challenging. Instead, during training, quantisation by rounding function is replaced by one or more other approaches. For example, the functions of a noise quantisation model are differentiable everywhere and accordingly do allow backpropagation of the gradient of the loss function through the quantisation parts of the end-to-end, learned system. Alternatively, a straight-through estimator (STE) quantisation model or one other quantisation models may be used. It is also envisaged that different quantisation models may be used for during evaluation of different term of the loss function. For example, noise quantisation may be used to evaluate the rate or entropy loss term of the rate-distortion loss function while STE quantisation may be used to evaluate the distortion term.
[0113] In a similar manner to how learning parameters top produce certain distributions of the latent representation facilitates achieving better rate loss term minimisation, end-to-end learning of the quantisation process achieves a similar effect. That is, learnable quantisation parameters provide the architecture with a further degree of freedom to achieve the goal of minimising the loss function. For example, parameters corresponding to quantisation bin sizes may be learned which is likely to result in an improved rate-distortion loss outcome compared to approaches using hand-crafted quantisation bin sizes.
[0114] Further, as the rate-distortion loss function constantly has to balance a rate loss term against a distortion loss term, it has been found that the more degrees of freedom the system has during training, the better the architecture is at achieving optimal rate and distortion trade off.
[0115] The system described above may be distributed across multiple locations and / or devices. For example, the encoder 110 may be located on a device such as a laptop computer, desktop computer, smart phone or server. The decoder 120 may be located on a separate device which maybe referred to as a recipient device. The system used to encode, transmit and decode the input image 5 to obtain the output image 6 may be referred to as a compression pipeline.
[0116] The Al based compression process may further comprise a hyper-network 105 for the transmission of meta-information that improves the compression process. The hyper-network 105 comprises a trained neural network 115 acting as a hyper-encoder fg and a trained neural network 125 acting as a hyper-decoder g^1. An example of such a system is shown in Figure 2. Components of the system not further discussed may be assumed to be the same as discussed above. The neural network 115 acting as a hyper-encoder receives the latent that is the output of the encoder 110. The hyper-encoder 115 produces an output based on the latent representation that may be referred to as a hyper-latent representation. The hyper-latent is then quantized in a quantization process 145 characterised by Qhto produce a quantized hyper-latent. The quantization process 145 characterised by Qhmay be the same as the quantisation process 140 characterised by Q discussed above.
[0117] In a similar manner as discussed above for the quantized latent, the quantized hyper-latent is then entropy encoded in an entropy encoding process 155 to produce a bitstream 135. The bitstream 135 may be entropy decoded in an entropy decoding process 165 to retrieve the quantized hyper-latent. The quantized hyper-latent is then used as an input to the trained neural network 125 acting as a hyper-decoder. However, in contrast to the compression pipeline 100, the output of the hyper-decoder may not be an approximation of the input to the hyper-encoder 115. Instead, the output of the hyper-decoder is used to provide parameters for use in the entropy encoding process 150 and entropy decoding process 160 in the main compression process 100. For example, the output of the hyper-decoder 125 can include one or more of the mean, standard deviation, variance or any other parameter used to describe a probability model for the entropy encoding process 150 and entropy decoding process 160 of the latent representation. In the example shown in Figure 2, only a single entropy decoding process 165 and hyper-decoder 125 is shown for simplicity. However, in practice, as the decompression process usually takes place on a separate device, duplicates of these processes will be present on the device used for encoding to provide the parameters to be used in the entropy encoding process 150.
[0118] Further transformations may be applied to at least one of the latent and the hyper-latent at any stage in the Al based compression process 100. For example, at least one of the latent and the hyper latent may be converted to a residual value before the entropy encoding process 150,155 is performed. The residual value may be determined by subtracting the mean value of the distribution of latents or hyper-latents from each latent or hyper latent. The residual values may also be normalised.To perform training of the Al based compression process described above, a training set of input images may be used as described above. During the training process, the parameters of both the encoder 110 and the decoder 120 may be simultaneously updated in each training step. If a hyper-network 105 is also present, the parameters of both the hyper-encoder 115 and the hyperdecoder 125 may additionally be simultaneously updated in each training step. The training process may further include a generative adversarial network (GAN). When applied to an Al based compression process, in addition to the compression pipeline described above, an additional neutral network acting as a discriminator is included in the system. The discriminator receives an input and outputs a score based on the input providing an indication of whether the discriminator considers the input to be ground truth or fake. For example, the indicator may be a score, with a high score associated with a ground truth input and a low score associated with a fake input. For training of a discriminator, a loss function is used that maximizes the difference in the output indication between an input ground truth and input fake.
[0119] When a GAN is incorporated into the training of the compression process, the output image 6 may be provided to the discriminator. The output of the discriminator may then be used in the loss function of the compression process as a measure of the distortion of the compression process. Alternatively, the discriminator may receive both the input image 5 and the output image 6 and the difference in output indication may then be used in the loss function of the compression process as a measure of the distortion of the compression process. Training of the neural network acting as a discriminator and the other neutral networks in the compression process may be performed simultaneously. During use of the trained compression pipeline for the compression and transmission of images or video, the discriminator neural network is removed from the system and the output of the compression pipeline is the output image 6.
[0120] Incorporation of a GAN into the training process may cause the decoder 120 to perform hallucination. Hallucination is the process of adding information in the output image 6 that was not present in the input image 5. In an example, hallucination may add fine detail to the output image 6 that was not present in the input image 5 or received by the decoder 120. The hallucination performed may be based on information in the quantized latent received by decoder 120.
[0121] Details of a video compression process will now be described. As discussed above, a video is made up of a series of images arranged in sequential order. Al based compression process 100 described above may be applied multiple times to perform compression, transmission and decompression of a video. For example, each frame of the video may be compressed, transmitted and decompressed individually. The received frames may then be grouped to obtain the original video.The frames in a video may be labelled based on the information from other frames that is used to decode the frame in a video compression, transmission and decompression process. As described above, frames which are decoded using no information from other frames may be referred to as I-frames. Frames which are decoded using information from past frames may be referred to as P-frames. Frames which are decoded using information from past frames and future frames may be referred to as B-frames. Frames may not be encoded and / or decoded in the order that they appear in the video. For example, a frame at a later time step in the video may be decoded before a frame at an earlier time.
[0122] The images represented by each frame of a video may be related. For example, a number of frames in a video may show the same scene. In this case, a number of different parts of the scene may be shown in more than one of the frames. For example, objects or people in a scene may be shown in more than one of the frames. The background of the scene may also be shown in more than one of the frames. If an object or the perspective is in motion in the video, the position of the object or background in one frame may change relative to the position of the object or background in another frame. The transformation of a part of the image from a first position in a first frame to a second position in a second frame may be referred to as flow, warping or motion compensation. The flow may be represented by a vector. One or more flows that represent the transformation of at least part of one frame to another frame may be referred to as a flow map.
[0123] An example Al based video compression, transmission, and decompression process 200 is shown in Figure 3. The process 200 shown in Figure 3 is divided into an I-frame part 201 for decompressing I-frames, and a P-frame part 202 for decompressing P-frames. It will be understood that these divisions into different parts are arbitrary and the process 200 may also be considered as a single, end-to-end pipeline.
[0124] As described above, I-frames do not rely on information from other frames so the 1-frame part 201 corresponds to the compression, transmission, and decompression process illustrated in Figures 1 or 2. The specific details will not be repeated here but, in summary, an input image x0is passed into an encoder neural network 203 producing a latent representation which is quantised and entropy encoded into a bitstream 204. The subscript 0 in x0indicates the input image corresponds to a frame of a video stream at position t = 0. This may be the first frame of an entire video stream or the first frame of a chunk of a video stream made up of, for example, an 1-frame and a plurality of subsequent P-frames and / or B-frames. The bitstream 204 is then entropy decoded and passed into a decoder neural network 205 to reproduce a reconstructed image x0which in this case is an 1-frame. The decoding step may be performed both locally at the same location as where the input image compression occurs as well as at the location where the decompression occurs.This allows the reconstructed image x0to be available for later use by components of both the encoding and decoding sides of the pipeline.
[0125] In contrast to I-frames, P-frames (and B-frames) do rely on information from other frames. Accordingly, the P-frame part 202 at the encoding side of the pipeline takes as input not only the input image xtthat is to be compressed (corresponding to a frame of a video stream at position t), but also one or more previously reconstructed images xt-1from an earlier frame t-1. As described above, the previously reconstructed xt-1is available at both the encode and decode side of the pipeline and can accordingly be used for various purposes at both the encode and decode sides.
[0126] At the encode side, previously reconstructed images may be used for generating a flow map containing information indicative of inter-frame movement of pixels between frames. In the example of Figure 3, both the image being compressed xtand the previously reconstructed image from an earlier frame xt-1are passed into a flow module part 206 of the pipeline. The flow module part 206 comprises an autoencoder such as that of the autoencoder systems of Figures 1 and 2 but where the encoder neural network 207 has been trained to produce a latent representation of a flow map from inputs xt-1and xt, which is indicative of inter-frame movement of pixels or pixel groups between xt-1and xt. The latent representation of the flow map is quantised and entropy encoded to compress it and then transmitted as a bitstream 208. On the decode side, the bitstream is entropy decoded and passed to a decoder neural network 209 to produce a reconstructed flow map f.
[0127] The reconstructed flow map f is applied to the previously reconstructed image xt-1to generate a warped image xt-l w. It is envisaged that any suitable warping technique may be used, for example bi-linear or tri-linear warping, as is described in Agustsson, E., Minnen, D., Johnston, N., Balle, J., Hwang, S. J., and Toderici, G. (2020), Scale-space flow for end-to-end optimized video compression. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (pp. 8503-8512), which is incorporated herein by reference in its entirety. It is further envisaged that a scale-space flow approach as described in the above paper may also optionally be used. The warped imaget-l wis a prediction of how the previously reconstructed image xt-1might have changed between frame positions t-1 and t, based on the output flow map produced by the flow module part 206 autoencoder system from the inputs of xtand xt-1.
[0128] As with the I-frame, the reconstructed flow map f and corresponding warped image
[0129]
[0130] may be produced both on the encode side and the decode side of the pipeline so they are available for use by other components of the pipeline on both the encode and decode sides.
[0131] In the example of Figure 3, both the image being compressed xtand thet-l ware passed into a residual module part 210 of the pipeline. The residual module part 210 comprises an autoencoder system such as that of the autoencoder systems of Figures 1 and 2 but where theencoder neural network 211 has been trained to produce a latent representation of a residual map indicative of differences between the input mage xtand the warped image
[0132]
[0133] The latent representation of the residual map is then quantised and entropy encoded into a bitstream 212 and transmitted. The bitstream 212 is then entropy decoded and passed into a decoder neural network 213 which reconstructs a residual map r from the decoded latent representation.
[0134] Alternatively, a residual map may first be pre-calculated between xtand the
[0135]
[0136] and the pre-calculated residual map may be passed into an autoencoder for compression only. This handcrafted residual map approach is computationally simpler, but reduces the degrees of freedom with which the architecture may learn weights and parameters to achieve its goal during training of minimising the rate-distortion loss function.
[0137] Thus, for a block of video frames comprising an I-frame and n subsequent P-frames, the bitstream may contain (i) a quantised, entropy encoded latent representation of the 1-frame image, and (ii) a quantised, entropy encoded latent representation of a flow map and residual map of each P-frame image. For completeness, whilst not illustrated in Figure 3, any of the autoencoder systems of Figure 3 may comprise hyper and hyper-hyper networks such as those described in connection with Figure 2. Accordingly, the bitstream may also contain hyper and hyper-hyper parameters, their latent quantised, entropy encoded latent representations and so on, of those networks as applicable.
[0138] Finally, the above approach may generally also be extended to B-frames, for example as is described in Pourreza, R., and Cohen, T. (2021). Extending neural p-frame codecs for b-frame coding. In Proceedings of the IEEE / CVF International Conference on Computer Vision (pp. 6680-6689).
[0139] The above-described flow and residual based approach is highly effective at reducing the amount of data that needs to be transmitted because, as long as at least one reconstructed frame (e.g., 1-frame xt-1) is available, the encode side only needs to compress and transmit a flow map and a residual map (and any hyper or hyper-hyper parameter information, as applicable) to reconstruct a subsequent frame.
[0140] Figure 4 shows an example of an Al image or video compression process such as that described above in connection with Figures 1-3 implemented in a video streaming system 400. The system 400 comprises a first device 401 and a second device 402. The first and second devices 401, 402 may be user devices such as smartphones, tablets, AR / VR headsets or other portable devices. In contrast to known systems which primarily perform inference on GPUs such as Nvidia A100, Geforce 3090, Geforce 4090 GPU cards, the system 400 of Figure 4 performs inference on a CPU of the first and second devices respectively. That is, compute for performing both encodingand decoding are performed by the respective CPUs of the first and second devices 401, 402. This places very different power usage, memory and runtime constraints on the implementation of the above methods than when implementing Al-based compression methods on GPUs. In one example, the CPU of first and second devices 401, 402 may comprise a Qualcomm Snapdragon CPU.
[0141] The first device 401 comprises a media capture device 403, such as a camera, arranged to capture a plurality of images, referred to hereafter as a video stream 404, of a scene 405. The video stream 404 is passed to a pre-processing module 406 which splits the video stream into blocks of frames, various frames of which will be designated as I-frames, P-frames, and / or B-frames. The blocks of frames are then compressed by an Al-compression module 407 comprising the encode side of the Al-based video compression pipeline of Figure 3. The output of the Al-compression module is accordingly a bitstream 408a which is transmitted from the first device 401, for example via a communications channel, for example over one or more of a WiFi, 3G, 4G or 5G channel, which may comprise internet or cloud-based 409 communications.
[0142] The second device 402 receives the communicated bitstream 408b which is passed to an Al-decompression module 410 comprising the decode side of the Al-based video compression pipeline of Figure 3. The output of the Al-decompression module 410 is the reconstructed I-frames, P-frames and / or B-frames which are passed to a post-processing module 411 where they can prepared, for example passed into a buffer, in preparation for streaming 412 to and rendering on a display device 413 of the second device 402.
[0143] It is envisaged that the system 400 of Figure 4 may be used for live video streaming at 30fps of a 1080p video stream, which means a cumulative latency of both the encode and decode side is below substantially 50ms, for example substantially 30ms or less. Achieving this level of runtime performance with only CPU compute on user devices presents challenges which are not addressed by known methods and systems or in the wider Al-compression literature.
[0144] For example, execution of different parts of the compression pipeline during inference may be optimized by adjusting the order in which operations are performed using one or more known CPU scheduling methods. Efficient scheduling can allow for operations to be performed in parallel, thereby reducing the total execution time. It is also envisaged that efficient management of memory resources may be implemented, including optimising caching methods such as storing frequently-accessed data in faster memory locations, and memory reuse, which minimizes memory allocation and deallocation operations.
[0145] A number of concepts related to the Al compression processes and / or their implementation in a hardware system discussed above will now be described. Although each concept is describedseparately, one or more of the concepts described below may be applied in an Al based compression process as described above.
[0146] Concept 1: Model determinism
[0147] Figures 5 to 7 illustratively show components of a flow residual Al-based compression, transmission and decompression pipeline, such as that of Figure 3.
[0148] Figure 5 illustrates the components of an I-frame part comprising an encoder neural network 501 (such as the encoder neural network 203 of Figure 3) and a decoder neural network 502 (such as the decoder neural network 205 of Figure 3)
[0149] The encoder neural network 501 in Figure 5 may be considered as a single neural network or as a plurality of neural network components made up of an encoder 503, a hyper encoder 504, and a hyper hyper encoder 505, as well as a decoder 506, a hyper decoder 507, and a hyper hyper decoder 508, with corresponding entropy encoding modules 509, 510, 511 that together use the outputs of the decoders 506, 507, 508 to entropy encode the latent representation outputs of the encoders 503, 504, 505 into a bitstream.
[0150] The decoder neural network 502 in Figure 5 may be considered as a single neural network or as a plurality of neural network components made up of a decoder 506, a hyper decoder 507, and a hyper hyper decoder 508 with corresponding entropy decoding modules 512, 513, 514 that together use the bitstreams and respective outputs of each other to entropy decode the bitstream and reconstruct the various latent representations into an output image x0, which is a lossy approximation of an input image x0.
[0151] Figure 6 illustrates the components of a flow part of a pipeline comprising a flow encoder neural network 601 (such as the encoder neural network 207 of Figure 3) and a flow decoder neural network 602 (such as the decoder neural network 209 of Figure 3)
[0152] The flow encoder neural network 601 in Figure 6 may be considered as a single neural network or as a plurality of neural network components made up of: a flow encoder 606, a flow hyper encoder 608, a flow hyper hyper encoder 609, a flow decoder 610, a flow hyper decoder 611 and a flow hyper hyper decoder neural network 612. Also provided are respective entropy encoding modules 613, 614, 615 that together use the outputs of the flow decoders 610, 611, 612 to entropy encode the latent representation outputs of the flow encoders 606, 608, 609 into a bitstream.
[0153] The flow decoder neural network 602 in Figure 6 may be considered as a single neural network or as a plurality of neural network components made up of a flow decoder 610, a flow hyper decoder 611, and a flow hyper hyper decoder 612 with corresponding entropy decodingmodules 616, 617, 618 that together use the bitstreams and respective outputs of each other to entropy decode the bitstream and reconstruct the various latent representations into a representation of optical flow information indicative of a difference between an input previously decoded image xt-1and an input current image xt. The representation of optical flow information is used to warp the previously decoded image xt-to produce a warped representation of the previously decoded image
[0154]
[0155] for later use by the residual part of the pipeline.
[0156] Figure 7 illustrates the components of a residual part of a pipeline comprising an encoder neural network 701 (such as the encoder neural network 211 of Figure 3) and a decoder neural network 702 (such as the decoder neural network 213 of Figure 3)
[0157] The encoder neural network 701 in Figure 7 may be considered as a single neural network or as a plurality of neural network components made up of: a residual encoder neural network 704, a residual hyper encoder neural network 705, a residual hyper hyper encoder neural network 706, a residual decoder neural network 707, a residual hyper decoder neural network 708 and a residual hyper hyper decoder neural network 709, together with corresponding entropy encoding modules 710, 711, 712 that together use the outputs of the decoders 707, 708, 709 to entropy encode the latent representation outputs of the encoders 704, 705, 706 into a bitstream. As shown in Figure 7, the latent representation outputs of the encoders are based on the current image xtand the warped representation of the previously decoded image
[0158]
[0159] produced by the flow part of the pipeline.
[0160] The residual decoder neural network 702 in Figure 7 may be considered as a single neural network or as a plurality of neural network components made up of a residual decoder 707, a residual hyper decoder 708 and a residual hyper hyper decoder 709, with corresponding entropy decoding modules 713, 714, 715 that together use the bitstreams and respective outputs of each other to entropy decode the bitstream and reconstruct the various latent representations into an output image xtwhich is a lossy approximation of the input current image xtfed into the residual encoder 704. Note that the operation of the one or more of the residual decoder 707, the residual hyper decoder 708, the residual hyper hyper decoder 709 of the residual decoder neural network 702 may be conditioned on the warped representation of the previously decoded image
[0161]
[0162] produced by the flow part of the pipeline, for example by combining its tensor into an input or output of one or more layers of the networks.
[0163] Note also for completeness that the warped representation of the previously decoded image xt-l wmay be produced from a single image or from multiple images of the image sequence. This may be in either ground truth form xt-1and / or in reconstructed form xt-1. This need not be limited to a strictly temporally "past" frame either. For example, in a pipeline where B-frames are being used, the previously decoded image may be a "future" frame in the image sequence that has beenencoded and decoded before the current frame as would be applicable in a random access encoding and decoding scheme. Thus the previously decoded image may be said to be a reference image that may be selected from or produced from one or more images of the image sequence being encoded and decoded. It is in particular envisaged that a reference frame predictor neural network comprising a plurality of convolution and / or activation layers to receive as input the one or more previously decoded images, and to output a tensor that combines all these images into a single composite image. This composite image may then be used as a reference image against which to compare the current image when producing optical flow information and / or residual information indicative of a difference between the current frame and the one or more previously decoded images. Thus, the term "past" as used herein can be may refer not only to a "past" frame (in the sense of a temporal order of a sequence of images) but also to a future frame (as in B-frames). Thus throughout all of the present description, the term xt-lwcan mean xt-nw, and the term ft-1can mean ft-m, where n and m can be both positive and negative, and / or relate to multiple images of a sequence.
[0164] The flow part and residual part of the pipeline shown in Figures 6 and 7 may be combined into a P- and / or B-frame module, which may be combined with the I-frame part of the pipeline, which together may be used as an end-to-end, Al-based I-frame, P- and / or B-frame video compression pipeline. That is, the I-frame part may compress, transmit, and reconstruct a first frame of an image sequence as an 1-frame, and then a plurality of other frames of the image sequence may be compressed, transmitted, and reconstructed as P- and / or B-frames, together forming a group of pictures (GOP).
[0165] When deploying Al-based compression pipelines such as any of those described above, guaranteeing model determinism across different hardware platforms is non-trivial. For example, different CPU, GPU, and NPU chip set manufacturers permit different extents of control of compiler optimization choices, extents of machine level instruction set choices, and so on. Indeed the extent of control is rarely consistent across different chip sets and, in many cases, granular details of what optimizations are applied when different compiler preset options are selected are not exposed to, let alone controllable by, developers implementing programs to run on these chip sets. This lack of consistent control makes it difficult to implement neural network models in a way that guarantees that encoding and / or decoding will produce the same outputs when some or all of the neural network models are running on different chip sets. For example, a compiler for one chip set might always try to aggressively fuse some convolution operations but not others, make approximations and assumptions about data types in some places but not in others, rearrange orders of operations such as reordering sums to improve parallelization or reduce register pressure,transforming expressions for better instruction scheduling, grouping multiplications to reduce total computational steps, and so on. A different compiler on a different chip set might run with different optimization approaches. These types of differences can produce rounding errors and other errors that can propagate and cause significant differences in outputs as the information moves through the layers of the neural network. A consequence of this is that it can be challenging to guarantee model determinism when the encode side of an Al-based compression pipeline is running on one manufacturer's chip set while the decode side is running on a different manufacturer's chip set. This phenomenon is referred to herein as model non-determini sm. For example, model nondeterminism may occur when an encode side of the pipeline is running on one of Apple, Intel, Qualcomm silicon, and so on, while the decode side is running on a different one of those.
[0166] Model non-determini sm can also occur in other scenarios, for example where errors are introduced into a bitstream as a result of the physical transmission process (for example due to a noisy communication channel), as a result of anomalous hardware behaviour, or in any scenario where an input to the neural network parts is being mapped in a non-deterministic way to an output.
[0167] The inventors have found that a cause of high sensitivity of Al-based compression pipelines to model non-determini sm is when model non-determini sm occurs in at least the hyper network part, but particularly in the hyper hyper network part of the pipeline. That is, when it occurs in the hyper encoder neural network part, the hyper encoder neural network part, the hyper hyper encoder neural network part and / or in the hyper hyper decoder neural network part, with the effect becoming worse deeper into the nested structure.
[0168] Recall that a purpose of the hyper hyper decoder, is to take as input the hyper latent representation and to produce as output entropy parameters that describe the distribution of the pixel values of the hyper latent representation For example each pixel coordinate of the hyper latent representation may have a scale and location parameter associated with it that defines a distribution indicative of what the pixel value at that pixel coordinate is likely to be. These parameters allow the set of values (i.e. symbols) of the hyper latent representation to be entropy encoded as efficiently as possible (i.e. as close to the Shannon entropy limit as possible) into a bitstream, and then entropy decoded back into the hyper latent representation on the decode side, for example using range asymmetric numerical system (rANS) encoding and decoding. It will also be recalled that a purpose of the hyper encoder neural network part and the hyper decoder neural network part is to perform the same exercise but now for producing the entropy parameters associated with the distribution of pixels of the latent representation to alow it to be entropy encoded and decoded as efficiently as possible. It is when the latent representation is finally available on the decode side that the input image can be reconstructed.The various latent representations in the (base) networks are dependent on the pixel values of the hyper latent representation, which in turn are dependent on the pixel values hyper hyper latent representation. The hyper hyper latent representation has smaller spatial dimensions than the hyper latent representation, which in turn has smaller spatial dimensions than the latent representation. Changes to pixel values of the hyper hyper latent representation (and to a lesser extent the hyper latent representation ) have a disproportionately greater effect on the values of the various downstream latent representation. For example, even very small changes in pixel values of the hyper hyper latent representation can result in very large changes in the downstream hyper latent representation which in turn can result in even larger changes in the latent representation.
[0169] Conceptually, model non-determini sm in the hyper hyper network and hyper network can be likened to an error being introduced into the entropy parameters, for example into the scale and / or location values, used for entropy encoding and / or decoding. Consider, for example, a scenario where the encode side is running on one chip set while the decode side is running on another chip set. In this toy scenario, model non-determini sm can cause the entropy parameters used for the encoding of the various latent representations into the bitstream to be different to the entropy parameters used for entropy decoding the bitstream back into the various latent representations. Even though entropy encoding and decoding is intended to be lossless, one effect of model non-determini sm is to make entropy encoding and decoding lossy, with the effect strengthening deeper into the nested hyper and hyper hyper networks.
[0170] Recall also that at the deepest level of the nested structure, the entropy parameters used for entropy encoding and decoding the deepest level latent representation (e.g. here the hyper hyper latent representation) may be learned during pipeline training. That is, a tensor of entropy parameters, for example scale and / or location values, may be optimized for whatever training data set is used, in the hope that the learned values are end up generalizing well enough to a validation data set.
[0171] The inventors have found that the problems associated with model non-determini sm can at least be partially mitigated by introducing an entropy parameter context model into one or more of the nested network structures.
[0172] Whilst it is envisaged that this may be in any of the networks, for example in the (base) network, the hyper network, and / or the hyper hyper network, the example described below is based on introducing such a model into the deepest level network structures, in this case that is into the hyper hyper network of a residual part of an Al-based compression pipeline. This facilitates the calculation of entropy parameters for the residual hyper hyper latent representation in way thateach pixel of the entropy parameter tensor takes into account the context of surrounding pixel values of the hyper hyper latent representation tensor at a corresponding pixel coordinate.
[0173] Figure 8 illustrates the components of a residual part of a pipeline based on that of Figure 7. Like-numbered references refer to like-numbered elements.
[0174] As in Figure 7, the residual part of Figure 8 comprises an encoder neural network 701 (such as the encoder neural network 211 of Figure 3) and a decoder neural network 702 (such as the decoder neural network 213 of Figure 3)
[0175] The encoder neural network 701 in Figure 7 may be considered as a single neural network or as a plurality of neural network components made up of: a residual encoder neural network 704, a residual hyper encoder neural network 705, a residual hyper hyper encoder neural network 706, a residual decoder neural network 707, a residual hyper decoder neural network 708 and a residual hyper hyper decoder neural network 709, together with corresponding entropy encoding modules 710, 711, 712 that together use the outputs of the decoders 707, 708, 709 to entropy encode the latent representation outputs of the encoders 704, 705, 706 into a bitstream. As shown in Figure 7, the latent representation outputs of the encoders are based on the current image xtand the warped representation of the previously decoded image
[0176]
[0177] produced by the flow part of the pipeline.
[0178] The residual decoder neural network 702 in Figure 7 may be considered as a single neural network or as a plurality of neural network components made up of a residual decoder 707, a residual hyper decoder 708 and a residual hyper hyper decoder 709, with corresponding entropy decoding modules 713, 714, 715 that together use the bitstreams and respective outputs of each other to entropy decode the bitstream and reconstruct the various latent representations into an output image xtwhich is a lossy approximation of the input current image xtfed into the residual encoder 704. Note that the operation of the one or more of the residual decoder 707, the residual hyper decoder 708, the residual hyper hyper decoder 709 of the residual decoder neural network 702 may be conditioned on the warped representation of the previously decoded image
[0179]
[0180] produced by the flow part of the pipeline, for example by combining its tensor into an input or output of one or more layers of the networks.
[0181] Unlike in Figure 7, the residual part of Figure 8 also comprises an entropy parameter context model 801, denoted by m(^), on both the encode side and decode side of the pipeline.
[0182] The operation of the entropy parameter context model 801 on the encode side of the pipeline is described first. The entropy parameter context model 801 takes as input the hyper hyper latent representation tensor produced by the hyper hyper encoder neural network part 706 and, from this, constructs a tensor of entropy parameters for each pixel coordinate of the hyper hyper latent representation tensor. In this specific example, the specific entropy parameters predicted bythe entropy parameter context model 801 is a tensor of scale values. The entropy parameter context model 801 does this by moving a window across the hyper hyper latent representation tensor with stride 1 and calculating a central tendency measure, for example a mean or average or other central tendency measure, of the pixel values within the window. The resulting central tendency measure at each pixel coordinate is then used as the scale value for that pixel coordinate. The entropy parameter tensor is then passed together with the hyper hyper latent representation tensor into the entropy encoding module 712, where the entropy parameter tensor is used to entropy encode the pixel values (symbols) of the hyper hyper latent representation into the bitstream. On the encode side of the pipeline, the whole hyper hyper latent representation tensor is available so the central tendency measure calculations can be optionally parallelized for a run time speed up. Optionally, a location parameter tensor may be used to shift the center points of the pixel value distributions of the hyper latent representation before the entropy encoding occurs. In this case, the location parameters are obtained from a learned tensor (i.e. values learned during training that do not take into account pixel value context). Note for completeness that, just as the scale parameter defines for each pixel coordinate of the hyper hyper latent representation the shape of the probability distribution of the pixel values at that pixel coordinate, the location parameter for each pixel coordinate defines the center point of the pixel value probability distribution. Location shifting these center points to 0 can be beneficial for compression rate performance. The learned location parameter tensor contains a learned set of values that together facilitate this location shifting to improve compression rate performance.
[0183] The operation of the entropy parameter context model 801 on the decode side of the pipeline will now described. On the decode side of the pipeline the bitstream is received and entropy decoding can begin. For the first pixel of the hyper hyper latent representation tensor, no previously entropy decoded pixels are available so we use a learned entropy parameter value (that is, one that is learned during training of the networks and which does not use any context of surrounding pixels) for entropy decoding the first pixel. For entropy decoding the next pixel of the hyper hyper latent representation, for example an immediately adjacent pixel, we have the context of the first pixel available. The central tendency measure is calculated by taking the mean of the window including the first pixel, in this case giving a scale value corresponding to the pixel value of the first pixel, that can now be used to entropy decode this next pixel. This process is then repeated in a sliding window manner across the pixels of the hyper hyper latent representation, with each next pixel using as its scale parameter for entropy decoding the mean of the pixel values in a predetermined window shape. For example, if the window size is a 3x3 window shape and the window is moving from left to right across each row in order, then each next pixel will have ascale value for entropy decoding calculated by taking the mean of up to 4 pixels: the 3 pixels in the row above, and the 1 pixel immediately to its left.
[0184] Figure 9 illustratively shows two steps of a 3x3 sliding window as it moves across a toy example set of pixels of a hyper hyper latent representation tensor during entropy decoding. In the first step 901a, the scale parameter for entropy decoding a first pixel is computed by calculating a central tendency measure, for example a mean or average, and taking the value of this measure as the scale parameter. This allows that pixel to be entropy decoded, and made available for use context in the next step, as shown in step 901b. In step 901b, the window has moved one pixel to the right and is using the previously decoded hyper hyper latent representation pixel (together with the 3 previously decoded pixels from the row above) as context for calculating the scale parameter for entropy decoding the current pixel. For completeness, at the edges the window shape may be changed to accommodate only pixels within the pixel coordinates in the image, or padding may be implemented so the window may stay the same shape.
[0185] One advantage of using a central tendency measure estimate as the entropy parameter context model is that it has a very low computational overhead. For example, the computational overhead associated with performing at most a tensor worth of mean computations is significantly faster than a purely neural network context based approach, such as using transformer and / or attention mechanisms for context modeling. This is particularly the case when the entropy parameter context model is introduced in the hyper hyper network, where the spatial dimensions are significantly smaller than those of the input image, the latent representation and / or the hyper latent representation, and so the number of computations used to estimate the entropy parameters is small. This approach to entropy parameter context modeling thus helps to facilitate real time or near real time encoding and / or decoding with Al-based compression pipelines. Despite the low computational overhead, the effect of the introduction of an entropy parameter context model such as that described above is to significantly mitigate the effects of model non-determini sm in an AI-based compression pipeline with a nested hyper and hyper hyper network structure. Indeed, it is envisaged that the same advantages may be realised in other Al-based compression architectures as well.
[0186] It will be appreciated that where pixel values of pixel coordinates of a representation of some information is referred to (for example, pixel values of a latent, hyper latent, and / or hyper hyper latent representation), this means that the representation comprises those values arranged in some way with each value having a coordinate. For example, if a latent representation comprises a tensor of some shape defined by a channel dimension, a height dimension and a width dimension, such as C x H x W, more compactly written as (C, H, W). Then there is a unique pixel coordinatedefined by a triplet of indices (c, h, w) where c represents the channel (from 0 to C), h represents the vertical position or height (from 0 to H-l), and w represents the horizontal position or width (from 0 to W-l). Each coordinate maps to a specific pixel value within the tensor. The collection of all such coordinates and their associated values constitutes the complete latent representation. It is generally envisaged that the hyper hyper latent representation tensor will have smaller height and width dimensions, and thus fewer pixel coordinates, than the hyper latent representation tensor, which in turn will have smaller height and width dimensions, and thus fewer pixel coordinates, than the latent representation tensor.
[0187] Whilst the example giving above is based on introducing an entropy parameter context model into the hyper hyper network of a residual part of an Al-based compression pipeline, the model may additionally or alternatively be introduced into any of the other networks of an AI-based compression pipeline as an alternative to either using learned entropy parameters (e.g. a tensor of scale values learned during training of the pipeline) or as an alternative to using one or more of the hyper or hyper hyper networks for producing one or more entropy parameters to entropy encode and decode one or more of the latent, the hyper latent and / or the hyper hyper latent representations, in any of the pipelines described and illustrated in Figures 1 to 7.
[0188] We now describe a series of further additional or alternative approaches to mitigating model non-determinism in Al-based compression pipeline using bitwise accurate convolution kernels in one or more convolution layers of the (base) networks, the hyper networks, and / or the hyper hyper networks of the pipeline. As was the case with the entropy parameter context model, it has been found that introducing a bitwise accurate convolution kernel into one or more convolution layers of the hyper hyper networks has the effect of at least partially (indeed disproportionately) mitigating the effects of model non-determinism in Al-based image and video compression. This is particularly the case when using such convolutions for computing of one or more of the entropy parameters (e.g. scale and / or location) towards the output ends or layers of the hyper decoders, and / or hyper hyper decoders of any of the I-frame, flow, and / or residual modules.
[0189] Figure 10 illustrates a portion 1000 of an I-frame module of an Al-based compression pipeline that has one or both of a hyper hyper network architectural component and a hyper network architectural component. This may be, for example, an architecture such as that shown and described with reference to Figure 5. The portion 1000 illustrated in Figure 10 shows (i) the decoding of the I-frame hyper hyper latent representation by the I-frame hyper hyper decoder neural network 1001 to produce the entropy parameters
[0190]
[0191] (by way of reminder these are respectively the location, scale and quantisation parameters associated with the I-frame hyper latent representation, (ii) the decoding of the I-frame hyper latent representation by the I-framehyper decoder neural network 1002 to produce the entropy parameters p, o, 8 associated with the I-frame latent representation, and (iii) the decoding of the I-frame latent representation by the (base) I-frame decoder neural network 1003 to produce the output image x, which is the lossy reconstruction of an input image x.
[0192] Each of the 1-frame hyper hyper decoder neural network 1001, the I-frame hyper decoder neural network 1002 and the (base) 1-frame decoder neural network 1003 comprises one or more convolution and activation layers. At least one of the convolution layers in at least one of the networks 1001, 1002, 1003 is implemented using a deterministic convolution operation, for example a convolution operation that is implemented using a bitwise accurate convolution kernel.
[0193] In the example of Figure 10, the hyper hyper decoder neural network 1001 comprises between 1 and 16 convolution layers, for example between 5 and 11 convolution layers, for example 8 convolution layers. The hyper hyper decoder neural network further comprises between 1 and 10 activation layers, for example between 4 and 6 activation layers, for example 5 activation layers. At least one of the convolution layers of the hyper hyper decoder neural network is envisaged to be implemented using a deterministic convolution, for example a convolution that is implemented using a bitwise accurate convolution kernel. That is, the steps of the convolution layer (and indeed any other multiplication, division, addition, subtraction, and so on, operations) are explicitly defined without using native chipset platform optimizations, compiler specific optimizations, and so on. In the example of Figure 10, it is envisaged that all of the convolution kernels of the hyper hyper decoder neural network 1001 are deterministic convolution kernels, and any other operations in the hyper hyper decoder neural network are implemented in a deterministic way. As input, the hyper hyper decoder neural network receives the hyper hyper latent representation tensor which is entropy decoded (ED) from the portion of the bitstream comprising hyper hyper latent representation information. This entropy decoding is performed using learned, parametric entropy parameters <7, 6. The output of the hyper hyper decoder neural netowrk 1001 is the entropy parameters <7, <5 associated with the hyper latent representation, and these entropy parameters are used to entropy decode the portion of the bitstream comprising the hyper latent representation information to produce the hyper latent representation tensor, which is then fed into the hyper decoder neural network 1002.
[0194] The hyper decoder neural network 1002 in Figure 10 comprises between 1 and 16 convolution layers, for example between 5 and 11 convolution layers, for example 8 convolution layers. The hyper decoder neural network further comprises between 1 and 10 activation layers, for example between 4 and 6 activation layers, for example 5 activation layers. At least one of the convolution layers of the hyper decoder neural network 1002 is envisaged to be implemented usinga deterministic convolution, for example a convolution that is implemented using a bitwise accurate convolution kernel. In the example of Figure 10, only those convolutions (and indeed any multiplication, division, addition, subtraction, and so on, operations) that are performed to produce the output scale entropy parameter <7 are implemented so as to be deterministic. That is, in the hyper decoder neural network 1002, an initial single processing stream comprising, for example, a gain unit (element-wise multiplication of an input tensor with a learned tensor to control the rate distortion trade off during inference), a haar transform, may split off into separate processing streams for producing the various output entropy parameters such as the scale parameters <7, the location parameters . and so on. It is envisaged that the operations in the processing stream that outputs the scale parameters o in particular are implemented to be deterministic. As described above, non-determinism in the scale parameters estimation has been found to be particularly problematic for compression performance. Accordingly, targeting the determinism of the operations used to produce the scale parameter processing stream by making the convolution operations (and any other operations) deterministic, has the effect of significantly mitigating the negative effects on compression performance that model non-determinism causes. Whilst the processing streams that produce the other entropy parameters may also be implemented in a deterministic way, the inventors have found that the determinism of the scale entropy parameter <7 has a disproportionately greater effect on compression performance. The output of the hyper decoder neural network 1002 is the entropy parameters
[0195]
[0196] associated with the latent representation, and these entropy parameters are used to entropy decode the portion of the bitstream comprising the latent representation information to produce the latent representation tensor, which is then fed into the (base) decoder neural network 1003.
[0197] The (base) decoder neural network 1003 in Figure 10 comprises between 1 and 14 convolution layers, for example between 4 and 10 convolution layers, for example 7 convolution layers. The decoder neural network comprises between 1 and 10 activation layers, for example between 4 and 6 activation layers, for example 5 activation layers. Whilst it is envisaged that one or more operations in the decoder neural network 1003 may be implemented deterministically, a primary purpose of the decoder neural network is to produce the lossy reconstruction x of the input image x, so the effects of model non-determinism are observed to be less significant and deterministic implementation of the operations is less important in the decoder neural network 1003. As described above, the decoder neural network 1003 takes as input the latent representation tensor, and outputs the lossy reconstructed image x, which may be used as an I-frame in an AI-based compression pipeline.Figure 11 illustrates a portion 1100 of a flow module of an Al-based compression pipeline that has one or both of a hyper hyper network architectural component and a hyper network architectural component. This may be, for example, an architecture such as that shown and described with reference to Figure 6. The portion 1100 illustrated in Figure 11 shows (i) the decoding of the flow hyper hyper latent representation by the flow hyper hyper decoder neural network 1101 to produce the entropy parameters associated with the flow hyper latent representation, (ii) the decoding of the flow hyper latent representation by the flow hyper decoder neural network 1102 to produce the entropy parameters associated with the flow latent representation, and (iii) the decoding of the flow latent representation by the (base) flow decoder neural network 1103 to produce the output optical flow information.
[0198] Each of the flow hyper hyper decoder neural network 1101, the flow hyper decoder neural network 1102 and the (base) flow flow decoder neural network 1103 comprises one or more convolution and activation layers. At least one of the convolution layers is implemented using a deterministic convolution operation, for example a convolution operation that is implemented using a bitwise accurate convolution kernel.
[0199] In the example of Figure 11, the flow hyper hyper decoder neural network 1101 comprises between 1 and 16 convolution layers, for example between 5 and 11 convolution layers, for example 8 convolution layers. The flow hyper hyper decoder neural network further comprises between 1 and 10 activation layers, for example between 4 and 6 activation layers, for example 5 activation layers. At least one of the convolution layers of the flow hyper hyper decoder neural network 1101 is envisaged to be implemented using a deterministic convolution, for example a convolution that is implemented using a bitwise accurate convolution kernel. That is, the steps of the convolution layer (and indeed any other multiplication, division, addition, subtraction, and so on, operations) are explicitly defined without using native chipset platform optimizations, compiler specific optimizations, and so on. In the example of Figure 11, it is envisaged that all of the convolution kernels of the flow hyper hyper decoder neural network 1101 are deterministic convolution kernels whereby the flow hyper hyper decoder neural network 1101 may be said to be deterministic. As input, the flow hyper hyper decoder neural network receives the flow hyper hyper latent representation tensor which is entropy decoded (ED) from the portion of the bitstream comprising the flow hyper hyper latent representation information. This entropy decoding is performed using learned, parametric entropy parameters <7, <5. The output of the flow hyper hyper decoder neural netowrk 1001 is the entropy parameters <7, <5 associated with the flow hyper latent representation, and these entropy parameters are used to entropy decode the portion of thebitstream comprising the flow hyper latent representation information to produce the flow hyper latent representation tensor, which is then fed into the flow hyper decoder neural network 1102.
[0200] The flow hyper decoder neural network 1102 in Figure 11 comprises between 1 and 16 convolution layers, for example between 5 and 11 convolution layers, for example 8 convolution layers. The flow hyper decoder neural network 1102 further comprises between 1 and 10 activation layers, for example between 4 and 6 activation layers, for example 5 activation layers. At least one of the convolution layers of the flow hyper decoder neural network 1102 is envisaged to be implemented using a deterministic convolution, for example a convolution that is implemented using a bitwise accurate convolution kernel. In the example of Figure 11, only those convolutions (and indeed any multiplication, division, addition, subtraction, and so on, operations) that are performed to produce the output scale entropy parameter are implemented to be deterministic. That is, in the flow hyper decoder neural network 1102, an initial single processing stream may split off into separate processing streams for producing the various output entropy parameters such as the scale parameters, the location parameters and so on. It is envisaged that the convolutions and other operations in the processing stream that outputs the scale parameters are implemented to be deterministic. As described above, non-determini sm in the scale parameters estimation has been found to be particularly problematic for compression performance. Accordingly, targeting the determinism in the scale parameter processing stream in particular, by making the convolution and other operations deterministic, has the effect of significantly mitigating the negative effects on compression performance that model non-determini sm causes. Whilst the processing streams that produce the other entropy parameters may also be implemented in a deterministic way, the inventors have found that the determinism of the scale entropy parameter has a disproportionately greater effect on compression performance. Note also that in the example of Figure 11, the gain unit(s) element-wise multiplication for granular control of the rate distortion trade off that is present in the I-frame architecture in Figure 10 is not provided, although it is envisaged this may be included. The output of the flow hyper decoder neural network 1102 is the entropy parameters p, o, 8 associated with the flow latent representation, and these entropy parameters are used to entropy decode the portion of the bitstream comprising the flow latent representation information to produce the flow latent representation tensor, which is then fed into the (base) flow decoder neural network 1103.
[0201] The (base) flow decoder neural network 1103 in Figure 10 comprises between 1 and 8 convolution layers, for example between 2 and 6 convolution layers, for example 4 convolution layers. The decoder neural network comprises between 1 and 5 activation layers, for example between 1 and 3 activation layers, for example 2 activation layers. Whilst it is envisaged that oneor more operations in the flow decoder neural network 1103 may be implemented deterministically, a primary purpose of the flow decoder neural network is to produce the reconstruction f of the flow between a reference image xref and a current input image x, so the effects of model non-determini sm are observed to be less significant and deterministic implementation of the operations is less important in the decoder neural network 1103. As described above, the flow decoder neural network 1103 takes as input the flow latent representation tensor, and outputs the reconstructed flow , which may be used by a residual part of an Al-based compression pipeline to produce a P- and / or B-frame.
[0202] Figure 12 illustrates a portion 1200 of a residual module of an Al-based compression pipeline that has one or both of a hyper hyper network architectural component and a hyper network architectural component. This may be, for example, an architecture such as that shown and described with reference to Figure 7. The portion 1200 illustrated in Figure 12 shows (i) the decoding of the residual hyper hyper latent representation by the residual hyper hyper decoder neural network 1201 to produce the entropy parameters associated with the residual hyper latent representation, (ii) the decoding of the residual hyper latent representation by the residual hyper decoder neural network 1202 to produce the entropy parameters associated with the residual latent representation, and (iii) the decoding of the residual latent representation by the (base) residual decoder neural network 1203 to produce the output reconstructed image which is a lossy reconstruction of an input image x, conditioned on a (warped representation of) a reference image xrejw. That is the output of the portion 1200 of the residual module may be a P- and / or B-frame of the compression pipeline.
[0203] Each of the flow hyper hyper decoder neural network 1201, the flow hyper decoder neural network 1202 and the (base) flow flow decoder neural network 1203 comprises one or more convolution and activation layers. At least one of the convolution layers is implemented using a deterministic convolution operation, for example a convolution operation that is implemented using a bitwise accurate convolution kernel.
[0204] In the example of Figure 12, the residual hyper hyper decoder neural network 1201 comprises between 1 and 18 convolution layers, for example between 6 and 14 convolution layers, for example 9 convolution layers. The residual hyper hyper decoder neural network 1201 further comprises between 1 and 10 activation layers, for example between 4 and 6 activation layers, for example 5 activation layers. At least one of the convolution layers of the residual hyper hyper decoder neural network 1201 is envisaged to be implemented using a deterministic convolution, for example a convolution that is implemented using a bitwise accurate convolution kernel. That is, the steps of the convolution layer (and indeed any other multiplication, division, addition,subtraction, and so on, operations) are explicitly defined without using native chipset platform optimizations, compiler specific optimizations, and so on. In the example of Figure 12, it is envisaged that all of the convolution kernels of the residual hyper hyper decoder neural network 1201 are deterministic convolution kernels and any other operations in the hyper hyper decoder neural network are implemented in a deterministic way. As input, the residual hyper hyper decoder neural network 1201 receives the residual hyper hyper latent representation tensor which is entropy decoded (ED) from the portion of the bitstream comprising residual hyper hyper latent representation information. This entropy decoding is performed using learned, parametric entropy parameters p, a, 6. The output of the residual hyper hyper decoder neural network 1201 is the entropy parameters
[0205]
[0206] associated with the hyper latent representation, and these entropy parameters are used to entropy decode the portion of the bitstream comprising the residual hyper latent representation information to produce the residual hyper latent representation tensor, which is then fed into the residual hyper decoder neural network 1202.
[0207] The residual hyper decoder neural network 1202 in Figure 12 comprises between 1 and 16 convolution layers, for example between 5 and 11 convolution layers, for example 8 convolution layers. The hyper decoder neural network further comprises between 1 and 10 activation layers, for example between 4 and 6 activation layers, for example 5 activation layers. At least one of the convolution layers of the residual hyper decoder neural network 1202 is envisaged to be implemented using a deterministic convolution, for example a convolution that is implemented using a bitwise accurate convolution kernel. In the example of Figure 12, only those convolutions that are performed up to the point of producing the output scale entropy parameter are implemented as deterministic convolutions. That is, in the residual hyper decoder neural network 1202, an initial single processing stream (comprising, as in Figure 10, a gain unit component and a Haar transform component) may split off into separate processing streams for producing the various output entropy parameters such as the scale parameters <7, the location parameters p and so on. It is envisaged that the convolutions (and indeed any multiplication, division, addition, subtraction, and so on, operations) in the processing stream that outputs the scale parameters <7 are implemented in a deterministic way. As described above, non-determini sm in the scale parameters estimation has been found to be particularly problematic for compression performance. Accordingly, targeting the determinism of the convolutions in the scale parameter processing stream in particular, by making the operations deterministic, has the effect of significantly mitigating the negative effects on compression performance that model non-determini sm causes. Whilst the processing streams that produce the other entropy parameters may also be implemented in a deterministic way, theinventors have found that the determinism of the scale entropy parameter has a disproportionately greater effect on compression performance.
[0208] In any of Figures 10 to 12, implementing a bitwise accurate convolution kernel, and / or any of deterministic element-wise multiplications, divisions, additions, subtractions, and so on, operations (such as for example in the gain unit(s) and / or Haar transforms) may comprise one or more of the steps set out below.
[0209] In some implementations it is envisaged that the flow of data to, in and from the processor of the CPU and / or NPU, the register, the cache and the main memory may be controlled explicitly in the convolution kernel. For example, buffer allocation on the CPU side may be explicitly defined using Swift or Objective-C or another suitable language, where buffers are created for input data, weights, and outputs that get passed to the kernel through buffer attributes. Within the kernel itself, pre-allocated buffers may be accessed using device pointers to indicate read-only memory and to allow for read-write operations. For convolution operations, predetermined memory access patterns may be defined. Memory alignment requirements, local / shared memory for frequently accessed data like weights, and consistent ordering of floating-point (or fixed point) operations to maintain deterministic results may also be specified explicitly in the control flow. Explicitly defined optimizations such as tiling, unrolling small inner loops, and pre-computing of buffer indices may also be implemented without using platform native, higher level language optimizations.
[0210] In some implementations, all native chipset platform optimizations (e.g. compiler optimizations and higher level code library optimizations) are switched off.
[0211] In some implementations, forced sequential processing of the arithmetic operations in the kernel may be switched on.
[0212] The following clauses are presented in claim-like form to illustrate an aspect:
[0213] 1. A method for lossy image or video encoding and transmission, and decoding, the method comprising the steps of:
[0214] receiving a first image of a sequence of images at a first computer system;
[0215] with a first neural network, producing one or more latent representations associated with the first image;
[0216] entropy encoding the one or more latent representations to produce a bitstream using one or more entropy parameters associated with the one or more latent representations;
[0217] transmitting the bitstream to a second computer system;
[0218] entropy decoding the bitstream to produce the one or more latent representations using the one or more parameters;with a second neural network, decoding the one or more latent representations to produce an output image, wherein the output image is an approximation of the first image;
[0219] wherein the first neural network and the second neural network comprise a plurality of convolution layers and activation layers; and
[0220] wherein at least one of the convolution layers comprises a bitwise accurate convolution kernel.
[0221] 2. The method of clause 1, wherein the first neural network comprises an encoder neural network part, a hyper encoder neural network part and a hyper hyper encoder neural network part, and wherein the method comprises:
[0222] using the hyper encoder neural network part to produce a hyper latent representation of the first image; and
[0223] using the hyper hyper encoder neural network part to produce a hyper hyper latent representation of the first image,
[0224] wherein the one or more latent representations comprises the hyper latent representation and the hyper hyper latent representation.
[0225] 3. The method of clause 2, wherein the first neural network and / or the second neural network comprises a decoder neural network part, a hyper decoder neural network part and a hyper hyper decoder neural network part.
[0226] 4. The method of clause 3, comprising using the hyper hyper decoder neural network part to decode the hyper hyper latent representation to produce the one or more entropy parameters associated with the hyper latent representation.
[0227] 5. The method of clause 4, wherein at least one convolution layer of the hyper hyper decoder neural network part comprises a bitwise accurate convolution kernel.
[0228] 6. the method of any of clause 2 to 5, comprising using the hyper decoder neural network part to decode the hyper latent representation to produce the one or more entropy parameters associated with the latent representation.
[0229] 7. The method of clause 6, wherein at least one convolution layer of the hyper decoder neural network part comprises a bitwise accurate convolution kernel.8. A method for lossy image or video encoding and transmission, the method comprising the steps of:
[0230] receiving a first image of a sequence of images at a first computer system;
[0231] with a first neural network, producing one or more latent representations associated with the first image;
[0232] entropy encoding the one or more latent representations to produce a bitstream using one or more entropy parameters associated with the one or more latent representations; and transmitting the bitstream to a second computer system;
[0233] wherein the first neural network comprises a plurality of convolution layers and activation layers; and
[0234] wherein at least one of the convolution layers comprises a bitwise accurate convolution kernel.
[0235] 9. A method for lossy image or video receipt and decoding, the method comprising the steps of:
[0236] receiving a bitstream at a second computer system, the bitstream produced by entropy encoding one or more latent representations using one or more entropy parameters associated with the one or more latent representations, the one or more latent representations produced with a first neural network using a first image of a sequence of images;
[0237] entropy decoding the bitstream to produce the one or more latent representations using the one or more parameters; and
[0238] with a second neural network, decoding the one or more latent representations to produce an output image, wherein the output image is an approximation of the first image;
[0239] wherein the first neural network comprises a plurality of convolution layers and activation layers; and
[0240] wherein at least one of the convolution layers comprises a bitwise accurate convolution kernel.
[0241] 10. A data processing apparatus configured to perform the method of any of clauses 1 to 9.
[0242] 11. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any of clauses 1 to 9.12. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of any of clauses 1 to 9.
[0243] The subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly -embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. The subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory program carrier for execution by, or to control the operation of, data processing apparatus. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. The computer storage medium is not, however, a propagated signal.
[0244] The term "data processing apparatus" encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can also include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.
[0245] A computer program (which may also be referred to or described as a program, software, a software application, a module, a software module, a script, or code) can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computerprogram can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.
[0246] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by, and apparatus can also be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit).
[0247] Computers suitable for the execution of a computer program include, by way of example, can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a VR headset, a game console, a Global Positioning System (GPS) receiver, a server, a mobile phones, a tablet computer, a notebook computer, a music player, an e-book reader, a laptop or desktop computer, a PDAs, a smart phone, or other stationary or portable devices, that includes one or more processors and computer readable media, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.
[0248] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.
[0249] The subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of thesystem can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network ("LAN") and a wide area network ("WAN"), e.g., the Internet.
[0250] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
[0251] While this specification contains many specific implementation details, these should be construed as descriptions of features that may be specific to particular examples of particular inventions. Certain features that are described in this specification in the context of separate examples can also be implemented in combination in a single example. Conversely, various features that are described in the context of a single example can also be implemented in multiple examples separately or in any suitable sub-combination.
[0252] Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the examples described above should not be understood as requiring such separation in all examples, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.
Claims
CLAIMS1. A method for lossy image or video encoding and transmission, the method comprising the steps of:receiving a first image of a sequence of images at a first computer system;with a first neural network, producing one or more latent representations associated with the first image;entropy encoding the one or more latent representations to produce a bitstream using one or more entropy parameters associated with the one or more latent representations; and transmitting the bitstream to a second computer system;wherein one or more of the entropy parameters is based on a central tendency measure of one or more pixel values of the one or more latent representations.
2. A data processing apparatus comprising a processor, the processer configured to perform:receiving a first image of a sequence of images at a first computer system;with a first neural network, producing one or more latent representations associated with the first image;entropy encoding the one or more latent representations to produce a bitstream using one or more entropy parameters associated with the one or more latent representations; and transmitting the bitstream to a second computer system;wherein one or more of the entropy parameters is based on a central tendency measure of one or more pixel values of the one or more latent representations.
3. The method of claim 1 or the data processing apparatus of claim 2, wherein one or more of the entropy parameters is based on a mean of one or more pixel values of the latent representations.
4. The method of claim 1 or 3 or the data processing apparatus of claim 2 or 3, comprising, for a plurality of pixel coordinates of at least one of the latent representations, calculating a mean of pixel values at and / or around each of the pixel coordinates to produce one or more entropy parameters associated with the at least one latent representation.
5. The method claim 4 or the data processing apparatus of claim 4, comprising using the calculated mean of pixel values as the one or more entropy parameters.
6. The method of claim 1 or 3 to 5 or the data processing apparatus of claim 2 to 5, wherein the one or more entropy parameters comprise one or more scale values, and wherein the method comprises using the one or more scale values to perform range asymmetric numeral system (rANS) encoding of the one or more latent representations to produce the bitstream.
7. The method of claim 1 or 3 to 6 or the data processing apparatus of claim 2 to 6, comprising, for at least one pixel coordinate of one of the latent representations performing said entropy encoding with a learned entropy parameter.
8. The method of claim 1 or 3 to 7 or the data processing apparatus of claim 2 to 7, wherein the first neural network comprises an encoder neural network part, a hyper encoder neural network part and a hyper hyper encoder neural network part, and wherein the method comprises:using the hyper encoder neural network part to produce a hyper latent representation of the first image; andusing the hyper hyper encoder neural network part to produce a hyper hyper latent representation of the first image,wherein the one or more latent representations comprises the hyper latent representation and the hyper hyper latent representation.
9. The method of claim 8 or the data processing apparatus of claim 8, comprising computing the central tendency measure from one or more pixel values of the hyper hyper latent representation, and using the computed central tendency measure as the one or more entropy parameters associated with the hyper hyper latent representation.
10. The method of claim 9 the data processing apparatus of claim 9, wherein the first neural network comprises a decoder neural network part, a hyper decoder neural network part and a hyper hyper decoder neural network part, and wherein the method comprises using the hyper hyper decoder neural network part to decode the hyper hyper latent representation to produce the one or more entropy parameters associated with the hyper latent representation.
11. The method of claim 1 or 3 to 10 or the data processing apparatus of claim 2 to 10, wherein the first neural network comprises a plurality of convolution layers and activation layers,12. The method of claim 11 or the data processing apparatus of claim 11, wherein at least one of the convolution layers comprises a bitwise accurate convolution kernel.
13. A method for lossy image or video receipt and decoding, the method comprising the steps of:receiving a bitstream at a second computer system, the bitstream produced by entropy encoding one or more latent representations using one or more entropy parameters associated with the one or more latent representations, the one or more latent representations produced with a first neural network using a first image of a sequence of images;entropy decoding the bitstream to produce the one or more latent representations using the one or more parameters; andwith a second neural network, decoding the one or more latent representations to produce an output image, wherein the output image is an approximation of the first image;wherein one or more of the entropy parameters is based on a central tendency measure of one or more pixel values of the one or more latent representations.
14. A data processing apparatus comprising a processor, the processer configured to perform:receiving a bitstream at a second computer system, the bitstream produced by entropy encoding one or more latent representations using one or more entropy parameters associated with the one or more latent representations, the one or more latent representations produced with a first neural network using a first image of a sequence of images;entropy decoding the bitstream to produce the one or more latent representations using the one or more parameters; andwith a second neural network, decoding the one or more latent representations to produce an output image, wherein the output image is an approximation of the first image;wherein one or more of the entropy parameters is based on a central tendency measure of one or more pixel values of the one or more latent representations.
15. The method of claim 13 or the data processing apparatus of claim 14, wherein one or more of the entropy parameters is based on a mean of one or more pixel values of the latent representations.
16. The method of claim 13 or 15 the data processing apparatus of claim 14 or 15, comprising, for a plurality of pixel coordinates of at least one of the latent representations, calculating a mean of pixel values at and / or around each of the pixel coordinates to produce one or more entropy parameters associated with the at least one latent representation.
17. The method of claim 16 or the data processing apparatus of claim 16, comprising using the calculated mean of pixel values as the one or more entropy parameters.
18. The method of claim 13 or 15 to 17 or the data processing apparatus of claim 14 to 17, wherein entropy decoding the bitstream comprises: at the second computer system: calculating a mean of previously decoded pixel values around at least a first pixel coordinate of one of the latent representations, and using said mean as an entropy parameter to entropy decode from the bitstream a pixel value for the first pixel coordinate.
19. The method of claim 18 or the data processing apparatus of claim 18, wherein entropy decoding the bitstream comprises: repeating for further pixel coordinates of the one of the latent representations the steps of: calculating a mean of previously decoded pixel values around each further pixel coordinate and using each mean as an entropy parameter to entropy decode from the bitstream a pixel value for each said further pixel coordinates.
20. The method of claim 13 or 15 to 19 or the data processing apparatus of claim 14 to 19, comprising, for at least one pixel coordinate of one of the latent representations performing said entropy decoding with a learned entropy parameter.
21. The method of claim 13 or 15 to 20 or the data processing apparatus of claim 14 to 20, wherein the second neural network comprises a plurality of convolution layers and activation layers.
22. The method of claim 21 or the data processing apparatus of claim 21, wherein at least one of the convolution layers comprises a bitwise accurate convolution kernel.
23. A computer program comprising instructions which, when the program is executed by a computer, cause the computer to carry out the method of any of claims 1, 3 to 13 or 15 to 22.
24. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method of any of claims 1, 3 to 13 or 15 to 22.
