Extreme overfitting

Abstract

A video encoding / decoding computing device, including a processor in communication with a memory device is provided. The processor may be configured to: (i) clustering received input data into one or more clusters; (ii) for at least one of the one or more clusters, selecting one or more aspects of an overfitting process, where the one or more aspects are associated to the at least one of the one or more clusters; and (iii) for at least one of the one or more clusters, overfitting one or more parameters of a neural network using the selected one or more aspects of the overfitting process associated to the at least one of the one or more clusters and using data in the at least one of the one or more clusters. Overfitting of the neural network may be based on one or more outputs of an auxiliary neural network.

Description

EXTREME OVERFITTINGTECHNICAL FIELD

[0001] The example and non-limiting embodiments relate generally to optimizing neural networks and, more particularly, to overfitting neural networks or signals that are input to neural networks on input data.BRIEF DESCRIPTION OF PRIOR DEVELOPMENTS

[0002] A neural network (NN) may be described as a computation graph consisting of several layers of computation. Each layer may consist of one or more units, where each unit performs an elementary computation. A unit is connected to one or more other units, and the connection may be associated with a weight. The weight may be used for scaling the signal passing through the associated connection. Weights are learnable parameters, i.e., values which can be learned from training data. There may be other learnable parameters, such as those of batch-normalization layers.

[0003] In some neural networks, such as convolutional neural networks for image classification, initial layers (those close to the input data) extract semantically low-level features such as edges and textures in images, whereas intermediate layers extract more high-level features. After the feature extraction layers there may be one or more layers performing a certain task, such as classification, semantic segmentation, object detection, denoising, style transfer, super-resolution, etc.SUMMARY OF THE INVENTION

[0004] The following summary is merely intended to be an example. The summary is not intended to limit the scope of the claims.

[0005] In accordance with one aspect, a computing device including at least one processor in communication with a memory device may be provided. The at least one processor may beconfigured to cause the apparatus to perform: (i) grouping received input data into one or more clusters, (ii) for at least one of the one or more clusters, selecting one or more aspects of an overfitting process, where the one or more aspects are associated to the at least one of the one or more clusters, and (iii) for at least one of the one or more clusters, adapting one or more parameters of a neural network using the selected one or more aspects of the overfitting process associated to the at least one of the one or more clusters and using data in the at least one of the one or more clusters.

[0006] In accordance with another aspect, a computing device including at least one processor in communication with a memory device may be provided. The at least one processor may be configured to cause the apparatus to perform: (i) receiving reconstruction data and uncompressed data, (ii) applying the reconstruction data to a neural network filter to obtain first filtered data, (iii) computing a main loss based on the first filtered data and the uncompressed data, (iv) storing a copy of at least a portion of the neural network filter, (v) applying the reconstruction data to the neural network filter to obtain second filtered data, (vi) computing an overfitting loss based on the second filtered data and the uncompressed data, (vii) computing one or more gradients of the overfitting loss with respect to one or more overfittable parameters of the neural network filter, (viii) using the one or more gradients to update the one or more overfittable parameters of the neural network filter, obtaining an adapted neural network filter, (ix) computing an overfit progress loss based on the adapted neural network filter, (x) restoring the neural network filter after the neural network filter had been adapted with overfitting, based on the copy of at least a portion of the neural network filter, and (xi) determining a total loss based on a combination of the main loss and the overfit progress loss and updating the neural network filter.

[0007] In accordance with another aspect, a computing device including at least one processor in communication with a memory device may be provided. The at least one processor may be configured to cause the apparatus to perform: (i) determining one or more ground truth options for an input picture, (ii) training an auxiliary neural network for one or more tasks based at least in part on the one or more ground truth options, and (iii) adapting a neural network based at least in part on one or more outputs of the auxiliary neural network.

[0008] In accordance with another aspect, a computing device including at least one processor in communication with a memory device may be provided. The at least one processor may be configured to cause the apparatus to perform: (i) obtaining auxiliary information from an auxiliary neural network trained using one or more ground truth options of an input picture, (ii) adapting a neural network using the auxiliary information, and (iii) training the neural network based on an overfit loss determined during adapting of the neural network using the auxiliary information.

[0009] In accordance with another aspect, a computing device including at least one processor in communication with a memory device may be provided. The at least one processor may be configured to cause the apparatus to perform: (i) determining one or more ground truth options for an input picture, (ii) training an auxiliary neural network for one or more tasks based at least in part on the one or more ground truth options, and (iii) adapting a neural network based at least in part on one or more outputs of the auxiliary neural network.

[0010] In accordance with another aspect, a computing device including at least one processor in communication with a memory device may be provided. The at least one processor may be configured to cause the apparatus to perform: (i) extracting, using a first neural network, information based on an input picture, (ii) selecting, based on the extracted information, at least one pretrained neural network from a plurality of pretrained neural networks, and (iii) constructing a second neural network based at least in part on the selected pretrained neural network.

[0011] In accordance with another aspect, a computing device including at least one processor in communication with a memory device may be provided. The at least one processor may be configured to cause the apparatus to perform: (i) extracting, using a first neural network, information based on an input picture, (ii) generating ground-truth information of the extracted information using a genetic algorithm, (iii) selecting, based on the ground-truth information, at least one pretrained neural network from a plurality of pretrained neural networks, and constructing a second neural network based at least in part on the selected pretrained neural network.

[0012] According to some aspects, there is provided the subject matter of the independent claims. Some further aspects are provided in subject matter of the dependent claims.BRIEF DESCRIPTION OF DRAWINGS

[0013] The foregoing aspects and other features are explained in the following description, taken in connection with the accompanying drawings, wherein:

[0014] FIG. 1 shows schematically an electronic device employing embodiments of the examples described herein.

[0015] FIG. 2 shows schematically a user equipment suitable for employing embodiments of the examples described herein.

[0016] FIG. 3 further shows schematically electronic devices employing embodiments of the examples described herein connected using wireless and wired network connections.

[0017] FIG. 4 shows schematically a block diagram of an encoder on a general level.

[0018] FIG. 5 shows an example neural network-based end-to-end learned coding system in accordance with one or more example embodiments of the disclosure.

[0019] FIGs. 6A and 6B show example overfitting processes in accordance with one or more example embodiments of the disclosure.

[0020] FIG. 7 illustrates example variations of an auxiliary neural network in accordance with one or more example embodiments of the disclosure.

[0021] FIGs. 8A and 8B illustrate example instances of an auxiliary neural network in accordance with one or more example embodiments of the disclosure.

[0022] FIG. 9 illustrates training and inference of a neural network to be overfit in accordance with one or more example embodiments of the disclosure.

[0023] FIG. 10 illustrates ground truth generation in accordance with one or more example embodiments of the disclosure.

[0024] FIG. 11 illustrates ground truth generation with a genetic algorithm in accordance with one or more example embodiments of the disclosure.

[0025] FIG. 12 illustrates example inference with a genetic algorithm in accordance with one or more example embodiments of the disclosure.

[0026] FIG. 13 illustrates example fitness calculations of an overfit neural network in accordance with one or more example embodiments of the disclosure.

[0027] FIG. 14A-14G illustrate example flow diagrams in accordance with one or more example embodiments of the disclosure.DETAILED DESCRIPTIONApparatus and Mechanism for a Video / lmage Encoding Process

[0028] The following describes in detail a suitable apparatus and possible mechanisms for a video / image encoding process according to embodiments. In this regard reference is first made to FIG. 1 and FIG. 2, where FIG. 1 shows an example block diagram of an apparatus 50. The apparatus may be an Internet of Things (loT) apparatus configured to perform various functions, such as for example, gathering information by one or more sensors, receiving or transmitting information, analyzing information gathered or received by the apparatus, or the like. The apparatus may comprise a video coding system, which may incorporate a codec. FIG.2 shows a layout of an apparatus according to an example embodiment. The elements of FIG.1 and FIG. 2 are explained next.

[0029] The electronic device 50 may for example be a mobile terminal or user equipment of a wireless communication system, a sensor device, a tag, or other lower power device. However, it would be appreciated that embodiments of the examples described herein may be implemented within any electronic device or apparatus which may process data by neural networks.

[0030] The apparatus 50 may comprise a housing 30 for incorporating and protecting the device. The apparatus 50 further may comprise a display 32 in the form of a liquid crystal display. In other embodiments of the examples described herein the display may be any suitable display technology suitable to display an image or video. The apparatus 50 may further comprise a keypad 34. In other embodiments of the examples described herein any suitable data or user interface mechanism may be employed. For example the user interfacemay be implemented as a virtual keyboard or data entry system as part of a touch-sensitive display.

[0031] The apparatus may comprise a microphone 36 or any suitable audio input which may be a digital or analog signal input. The apparatus 50 may further comprise an audio output device which in embodiments of the examples described herein may be any one of: an earpiece 38, speaker, or an analog audio or digital audio output connection. The apparatus 50 may also comprise a battery (or in other embodiments of the examples described herein the device may be powered by any suitable mobile energy device such as solar cell, fuel cell or clockwork generator). The apparatus may further comprise a camera capable of recording or capturing images and / or video. The apparatus 50 may further comprise an infrared port for short range line of sight communication to other devices. In other embodiments the apparatus 50 may further comprise any suitable short range communication solution such as for example a Bluetooth wireless connection or a USB / firewire wired connection.

[0032] The apparatus 50 may comprise a controller 56, processor or processor circuitry for controlling the apparatus 50. The controller 56 may be connected to memory 58 which in embodiments of the examples described herein may store both data in the form of image and audio data and / or may also store instructions for implementation on the controller 56. The controller 56 may further be connected to codec circuitry 54 suitable for carrying out coding and / or decoding of audio and / or video data or assisting in coding and / or decoding carried out by the controller.

[0033] The apparatus 50 may further comprise a card reader 48 and a smart card 46, for example a UICC and UICC reader for providing user information and being suitable for providing authentication information for authentication and authorization of the user at a network.

[0034] The apparatus 50 may comprise radio interface circuitry 52 connected to the controller and suitable for generating wireless communication signals for example for communication with a cellular communications network, a wireless communications system or a wireless local area network. The apparatus 50 may further comprise an antenna 44 connected to the radio interface circuitry 52 for transmitting radio frequency signalsgenerated at the radio interface circuitry 52 to other apparatus(es) and / or for receiving radio frequency signals from other apparatus(es).

[0035] The apparatus 50 may comprise a camera capable of recording or detecting individual frames which are then passed to the codec 54 or the controller for processing. The apparatus may receive the video image data for processing from another device prior to transmission and / or storage. The apparatus 50 may also receive either wirelessly or by a wired connection the image for coding / decoding. The structural elements of apparatus 50 described above represent examples of means for performing a corresponding function.

[0036] With respect to FIG. 3, an example of a system within which embodiments of the examples described herein can be utilized is shown. The system 10 comprises multiple communication devices which can communicate through one or more networks. The system 10 may comprise any combination of wired or wireless networks including, but not limited to a wireless cellular telephone network (such as a GSM, UMTS, CDMA, LTE, 4G, 5G network etc.), a wireless local area network (WLAN) such as defined by any of the IEEE 802.x standards, a Bluetooth personal area network, an Ethernet local area network, a token ring local area network, a wide area network, and the Internet.

[0037] The system 10 may include both wired and wireless communication devices and / or apparatus 50 suitable for implementing embodiments of the examples described herein.

[0038] For example, the system shown in FIG. 3 shows a mobile telephone network 11 and a representation of the internet 28. Connectivity to the internet 28 may include, but is not limited to, long range wireless connections, short range wireless connections, and various wired connections including, but not limited to, telephone lines, cable lines, power lines, and similar communication pathways.

[0039] The example communication devices shown in the system 10 may include, but are not limited to, an electronic device or apparatus 50, a combination of a personal digital assistant (PDA) and a mobile telephone 14, a PDA 16, an integrated messaging device (IMD) 18, a desktop computer 20, a notebook computer 22. The apparatus 50 may be stationary or mobile when carried by an individual who is moving. The apparatus 50 may also be located in a mode of transport including, but not limited to, a car, a truck, a taxi, a bus, a train, a boat,an airplane, a bicycle, a motorcycle or any similar suitable mode of transport. A head mounted display (HMD) may also be configured to implement the methods as described herein, such as a caching and clearing mechanism for deep convolutional neural networks.

[0040] The embodiments may also be implemented in a set-top box; i.e. a digital TV receiver, which may / may not have a display or wireless capabilities, in tablets or (laptop) personal computers (PC), which have hardware and / or software to process neural network data, in various operating systems, and in chipsets, processors, DSPs and / or embedded systems offering hardware / software based coding.

[0041] Some or further apparatus may send and receive calls and messages and communicate with service providers through a wireless connection 25 to a base station 24. The base station 24 may be connected to a network server 26 that allows communication between the mobile telephone network 11 and the internet 28. The system may include additional communication devices and communication devices of various types.

[0042] The communication devices may communicate using various transmission technologies including, but not limited to, code division multiple access (CDMA), global systems for mobile communications (GSM), universal mobile telecommunications system (UMTS), time divisional multiple access (TDMA), frequency division multiple access (FDMA), transmission control protocol-internet protocol (TCP-IP), short messaging service (SMS), multimedia messaging service (MMS), email, instant messaging service (IMS), Bluetooth, IEEE 802.11, 3GPP Narrowband loT and any similar wireless communication technology. A communications device involved in implementing various embodiments of the examples described herein may communicate using various media including, but not limited to, radio, infrared, laser, cable connections, and any suitable connection.

[0043] In telecommunications and data networks, a channel may refer either to a physical channel or to a logical channel. A physical channel may refer to a physical transmission medium such as a wire, whereas a logical channel may refer to a logical connection over a multiplexed medium, capable of conveying several logical channels. A channel may be used for conveying an information signal, for example a bitstream, from one or several senders (or transmitters) to one or several receivers.

[0044] The embodiments may also be implemented in so-called loT devices. The Internet of Things (loT) may be defined, for example, as an interconnection of uniquely identifiable embedded computing devices within the existing Internet infrastructure. The convergence of various technologies has and may enable many fields of embedded systems, such as wireless sensor networks, control systems, home / building automation, etc. to be included in the Internet of Things (loT). In order to utilize the Internet loT devices are provided with an IP address as a unique identifier. loT devices may be provided with a radio transmitter, such as a WLAN or Bluetooth transmitter or a RFID tag. Alternatively, loT devices may have access to an IP-based network via a wired network, such as an Ethernet-based network or a power-line connection (PLC).

[0045] An MPEG-2 transport stream (TS), specified in ISO / IEC 13818-1 or equivalently in ITU-T Recommendation H.222.0, is a format for carrying audio, video, and other media as well as program metadata or other metadata, in a multiplexed stream. A packet identifier (PID) is used to identify an elementary stream (a.k.a. packetized elementary stream) within the TS. Hence, a logical channel within an MPEG-2 TS may be considered to correspond to a specific PID value.

[0046] Available media file format standards include ISO base media file format (ISO / IEC 14496-12, which may be abbreviated ISOBMFF) and file format for NAL unit structured video (ISO / IEC 14496-15), which derives from the ISOBMFF.

[0047] A video codec consists of an encoder that transforms the input video into a compressed representation suited for storage / transmission and a decoder that can uncompress the compressed video representation back into a viewable form. A video encoder and / or a video decoder may also be separate from each other, i.e. need not form a codec. Typically the encoder discards some information in the original video sequence in order to represent the video in a more compact form (that is, at lower bitrate).

[0048] Typical hybrid video encoders, for example many encoder implementations of ITU-T H.263 and H.264, encode the video information in two phases. Firstly pixel values in a certain picture area (or "block") are predicted for example by motion compensation means (finding and indicating an area in one of the previously coded video frames that corresponds closely to the block being coded) or by spatial means (using the pixel values around the block to becoded in a specified manner). Secondly the prediction error, i.e. the difference between the predicted block of pixels and the original block of pixels, is coded. This is typically done by transforming the difference in pixel values using a specified transform (e.g. Discrete Cosine Transform (DCT) or a variant of it), quantizing the coefficients and entropy coding the quantized coefficients. By varying the fidelity of the quantization process, encoder can control the balance between the accuracy of the pixel representation (picture quality) and size of the resulting coded video representation (file size or transmission bitrate).

[0049] In temporal prediction, the sources of prediction are previously decoded pictures (a.k.a. reference pictures). In intra block copy (IBC; a.k.a. intra-block-copy prediction and current picture referencing), prediction is applied similarly to temporal prediction but the reference picture is the current picture and previously decoded samples can be referred in the prediction process. Inter-layer or inter-view prediction may be applied similarly to temporal prediction, but the reference picture is a decoded picture from another scalable layer or from another view, respectively. In some cases, inter prediction may refer to temporal prediction, while in other cases inter prediction may refer collectively to temporal prediction and any of intra block copy, inter-layer prediction, and inter-view prediction provided that they are performed with the same or similar process as temporal prediction. Inter prediction or temporal prediction may sometimes be referred to as motion compensation or motion-compensated prediction.

[0050] Inter prediction, which may also be referred to as temporal prediction, motion compensation, or motion-compensated prediction, reduces temporal redundancy. In inter prediction the sources of prediction are previously decoded pictures. Intra prediction utilizes the fact that adjacent pixels within the same picture are likely to be correlated. Intra prediction can be performed in the spatial or transform domain, i.e., either sample values or transform coefficients can be predicted. Intra prediction is typically exploited in intra coding, where no inter prediction is applied.

[0051] One outcome of the coding procedure is a set of coding parameters, such as motion vectors and quantized transform coefficients. Many parameters can be entropy-coded more efficiently if they are predicted first from spatially or temporally neighboring parameters. For example, a motion vector may be predicted from spatially adjacent motion vectors and thedifference relative to the motion vector predictor may be coded. Prediction of coding parameters and intra prediction may be collectively referred to as in-picture prediction.

[0052] FIG. 4 shows a block diagram of a general structure of a video encoder. FIG. 4 presents an encoder for two layers, but it would be appreciated that presented encoder could be similarly extended to encode more than two layers. FIG. 4 illustrates a video encoder comprising a first encoder section 500 for a base layer and a second encoder section 502 for an enhancement layer. Each of the first encoder section 500 and the second encoder section 502 may comprise similar elements for encoding incoming pictures. The encoder sections 500, 502 may comprise a pixel predictor 302, 402, prediction error encoder 303, 403 and prediction error decoder 304, 404. FIG. 4 also shows an embodiment of the pixel predictor 302, 402 as comprising an inter-predictor 306, 406 (Pinter), an intra-predictor 308, 408 (Pintra), a mode selector 310, 410, a filter 316, 416 (F), and a reference frame memory 318, 418 (RFM). The pixel predictor 302 of the first encoder section 500 receives 300 base layer images (10, n) of a video stream to be encoded at both the inter-predictor 306 (which determines the difference between the image and a motion compensated reference frame 318) and the intra-predictor 308 (which determines a prediction for an image block based on the already processed parts of the current frame or picture). The output of both the interpredictor and the intra-predictor are passed to the mode selector 310. The intra-predictor 308 may have more than one intra-prediction modes. Hence, each mode may perform the intra-prediction and provide the predicted signal to the mode selector 310. The mode selector 310 also receives a copy of the base layer picture 300. Correspondingly, the pixel predictor 402 of the second encoder section 502 receives 400 enhancement layer images (11, n) of a video stream to be encoded at both the inter-predictor 406 (which determines the difference between the image and a motion compensated reference frame 418) and the intra-predictor 408 (which determines a prediction for an image block based on the already processed parts of the current frame or picture). The output of both the inter-predictor and the intra-predictor are passed to the mode selector 410. The intra-predictor 408 may have more than one intraprediction modes. Hence, each mode may perform the intra-prediction and provide the predicted signal to the mode selector 410. The mode selector 410 also receives a copy of the enhancement layer picture 400.

[0053] Depending on which encoding mode is selected to encode the current block, the output of the inter-predictor 306, 406 or the output of one of the optional intra-predictor modes or the output of a surface encoder within the mode selector is passed to the output of the mode selector 310, 410. The output of the mode selector is passed to a first summing device 321, 421. The first summing device may subtract the output of the pixel predictor 302, 402 from the base layer picture 300 / enhancement layer picture 400 to produce a first prediction error signal 320, 420 (Dn) which is input to the prediction error encoder 303, 403.

[0054] The pixel predictor 302, 402 further receives from a preliminary reconstructor 339, 439 the combination of the prediction representation of the image block 312, 412 (P'n) and the output 338, 438 (D'n) of the prediction error decoder 304, 404. The preliminary reconstructed image 314, 414 (I'n) may be passed to the intra-predictor 308, 408 and to the filter 316, 416. The filter 316, 416 receiving the preliminary representation may filter the preliminary representation and output a final reconstructed image 340, 440 (R'n) which may be saved in a reference frame memory 318, 418. The reference frame memory 318 may be connected to the inter-predictor 306 to be used as the reference image against which a future base layer picture 300 is compared in inter-prediction operations. Subject to the base layer being selected and indicated to be the source for inter-layer sample prediction and / or interlayer motion information prediction of the enhancement layer according to some embodiments, the reference frame memory 318 may also be connected to the inter-predictor 406 to be used as the reference image against which a future enhancement layer picture 400 is compared in inter-prediction operations. Moreover, the reference frame memory 418 may be connected to the inter-predictor 406 to be used as the reference image against which a future enhancement layer picture 400 is compared in inter-prediction operations.

[0055] Filtering parameters from the filter 316 of the first encoder section 500 may be provided to the second encoder section 502 subject to the base layer being selected and indicated to be the source for predicting the filtering parameters of the enhancement layer according to some embodiments.

[0056] The prediction error encoder 303, 403 comprises a transform unit 342, 442 (T) and a quantizer 344, 444 (Q). The transform unit 342, 442 transforms the first prediction error signal 320, 420 to a transform domain. The transform is, for example, the DCT transform. Thequantizer 344, 444 quantizes the transform domain signal, e.g. the DCT coefficients, to form quantized coefficients.

[0057] The prediction error decoder 304, 404 receives the output from the prediction error encoder 303, 403 and performs the opposite processes of the prediction error encoder 303, 403 to produce a decoded prediction error signal 338, 438 which, when combined with the prediction representation of the image block 312, 412 at the second summing device 339, 439, produces the preliminary reconstructed image 314, 414. The prediction error decoder 304, 404 may be considered to comprise a dequantizer 346, 446 (Q-l), which dequantizes the quantized coefficient values, e.g. DCT coefficients, to reconstruct the transform signal and an inverse transformation unit 348, 448 (T-l), which performs the inverse transformation to the reconstructed transform signal wherein the output of the inverse transformation unit 348, 448 contains reconstructed block(s). The prediction error decoder may also comprise a block filter which may filter the reconstructed block(s) according to further decoded information and filter parameters.

[0058] The entropy encoder 330, 430 (E) receives the output of the prediction error encoder 303, 403 and may perform a suitable entropy encoding / variable length encoding on the signal to provide error detection and correction capability. The outputs of the entropy encoders 330, 430 may be inserted into a bitstream e.g. by a multiplexer 508 (M).

[0059] The following abbreviations that may be found in the specification and / or the drawing figures are defined as follows:GA Genetic AlgorithmMAE Mean Absolute ErrormAP Mean Average PrecisionML Machine LearningMSE Mean Square ErrorNN Neural NetworkQP Quantization ParameterRFM Reference Frame MemoryROI Region of InterestSEI Supplemental Enhancement InformationSSIM Structure SimilarityVCM Video Coding for MachinesFundamentals of Neural Networks (NNs)

[0060] A neural network (NN) may be described as a computation graph consisting of several layers of computation. Each layer may consist of one or more units, where each unit performs an elementary computation. A unit is connected to one or more other units, and the connection may be associated with a weight. The weight may be used for scaling the signal passing through the associated connection. Weights are learnable parameters, i.e., values which can be learned from training data. There may be other learnable parameters, such as those of batch-normalization layers.

[0061] One property of NNs (and other machine learning tools) is that they are able to learn properties from input data, e.g., in supervised way or in unsupervised way. Such learning is a result of a training algorithm, or of a meta-level NN providing the training signal.

[0062] In general, the training algorithm may consist of changing some properties of the NN so that its output is as close as possible to a desired output. For example, in the case of classification of objects in images, the output of the NN can be used to derive a class or category index which indicates the class or category that the object in the input image belongs to. Training usually happens by minimizing or decreasing the output's error, also referred to as the loss or loss function. Examples of losses are mean squared error (MSE), cross-entropy, etc. In recent deep learning techniques, training is an iterative process, where at each iteration the algorithm modifies the weights of the NN to make a gradual improvement of the network's output, i.e., to gradually decrease the loss, by means of gradient descent technique. In one example, at each training iteration, gradients of the loss function with respect to oneor more weights or parameters of the NN are computed, for example by backpropagation technique; the computed gradients may then be used by an optimization routine, such as Adam or Stochastic Gradient Descent (SGD) to obtain an update to the one or more weights or parameters.

[0063] In this disclosure, the terms "model", "neural network", "neural net" and "network" may be used interchangeably. Additionally, the weights of neural networks are sometimes referred to as learnable parameters or simply as parameters.

[0064] Training a NN is an optimization process, but the final goal may be different from the typical goal of optimization. In optimization, the only goal is to minimize a function. In machine learning, the goal of the optimization or training process is to make the model learn the properties of the data distribution from a limited training dataset. In otherwords, the goal is to learn to use a limited training dataset in orderto learn to generalize to previously unseen data, i.e., data which was not used for training the model. This is usually referred to as generalization. In practice, data is usually split into at least two sets, the training set and the validation set. The training set is used for training the network, i.e., to modify its learnable parameters in order to minimize the loss. The validation set is used for checking the performance of the network on data which was not used to minimize the loss, as an indication of the final performance of the model. In particular, the errors on the training set and on the validation set are monitored during the training process to understand the following things:

[0065] If the network is learning at all - in this case, the training set error should decrease, otherwise the model is in the regime of underfitting.

[0066] If the network is learning to generalize - in this case, also the validation set error needs to decrease and to be not too much higher than the training set error. If the training set error is low, but the validation set error is much higher than the training set error, or it does not decrease, or it even increases, the model may be in the regime of overfitting. This means that the model hasjust memorized the training set's propertiesand performs well only on that set, but performs poorly on a set not used for tuning its parameters.Fundamentals of Video / lmage

[0067] Video codec consists of an encoder that transforms the input video into a compressed representation suited for storage / transmission and a decoder that can decompress the compressed video representation back into a viewable form. Typically, the encoder discards some information in the original video sequence in order to represent the video in a more compact form (that is, at lower bitrate).

[0068] Typical hybrid video codecs, for example ITU-T H.263 and H.264, encode the video information in two phases. Firstly pixel values in a certain picture area (or "block") are predicted for example by motion compensation means (finding and indicating an area in one of the previously coded video frames that corresponds closely to the block being coded) or by spatial means (using the pixel values around the block to be coded in a specified manner). Secondly the prediction error, i.e. the difference between the predicted block of pixels and the original block of pixels, is coded. This is typically done by transforming the difference in pixel values using a specified transform (e.g. Discrete Cosine Transform (DCT) or the like), quantizing the coefficients and entropy coding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the accuracy of the pixel representation (picture quality) and size of the resulting coded video representation (file size or transmission bitrate).

[0069] Inter prediction, which may also be referred to as temporal prediction, motion compensation, or motion-compensated prediction, exploits temporal redundancy. In inter prediction the sources of prediction are previously decoded pictures (a.k.a. reference pictures).

[0070] In temporal inter prediction, the sources of prediction are previously decoded pictures in the same scalable layer. In intra block copy (IBC; a.k.a. intra-block-copy prediction), prediction may be applied similarly to temporal inter prediction but the reference picture is the current picture and only previously decoded samples can be referred in the prediction process. Inter-layer or inter-view prediction may be applied similarly to temporal inter prediction, but the reference picture is a decoded picture from another scalable layer or from another view, respectively. In some cases, inter prediction may refer to temporal interprediction only, while in other cases inter prediction may refer collectively to temporal inter prediction and any of intra block copy, inter-layer prediction, and inter-view prediction provided that they are performed with the same or similar process than temporal prediction. Inter prediction, temporal inter prediction, or temporal prediction may sometimes be referred to as motion compensation or motion-compensated prediction.

[0071] Intra prediction utilizes the fact that adjacent pixels within the same picture are likely to be correlated. Intra prediction may be performed in spatial or transform domain, i.e., either sample values or transform coefficients can be predicted. Intra prediction is typically exploited in intra coding, where no inter prediction is applied.

[0072] One outcome of the coding procedure is a set of coding parameters, such as motion vectors and quantized transform coefficients. Many parameters may be entropy-coded more efficiently if they are predicted first from spatially or temporally neighboring parameters. For example, a motion vector may be predicted from spatially adjacent motion vectors and only the difference relative to the motion vector predictor may be coded. Prediction of coding parameters and intra prediction may be collectively referred to as in-picture prediction.

[0073] The decoder reconstructs the output video by applying prediction means similar to the encoder to form a predicted representation of the pixel blocks (using the motion or spatial information created by the encoder and stored in the compressed representation) and prediction error decoding (inverse operation of the prediction error coding recovering the quantized prediction error signal in spatial pixel domain). After applying prediction and prediction error decoding means the decoder sums up the prediction and prediction error signals (pixel values) to form the output video frame. The decoder (and encoder) can also apply additional filtering means to improve the quality of the output video before passing it for display and / or storing it as prediction reference for the forthcoming frames in the video sequence.

[0074] In typical video codecs the motion information is indicated with motion vectors associated with each motion compensated image block. Each of these motion vectors represents the displacement of the image block in the picture to be coded (in the encoder side) or decoded (in the decoder side) and the prediction source block in one of the previously coded or decoded pictures. In order to represent motion vectors efficiently those are typicallycoded differentially with respect to block specific predicted motion vectors. In typical video codecs the predicted motion vectors are created in a predefined way, for example calculating the median of the encoded or decoded motion vectors of the adjacent blocks. Another way to create motion vector predictions is to generate a list of candidate predictions from adjacent blocks and / or co-located blocks in temporal reference pictures and signaling the chosen candidate as the motion vector predictor. In addition to predicting the motion vector values, the reference index of previously coded / decoded picture may be predicted. The reference index is typically predicted from adjacent blocks and / or or co-located blocks in temporal reference picture. Moreover, typical high efficiency video codecs employ an additional motion information coding / decoding mechanism, often called merging / merge mode, where all the motion field information, which includes motion vector and corresponding reference picture index for each available reference picture list, is predicted and used without any modification / correction. Similarly, predicting the motion field information is carried out using the motion field information of adjacent blocks and / or co-located blocks in temporal reference pictures and the used motion field information is signaled among a list of motion field candidate list filled with motion field information of available adjacent / co-located blocks.

[0075] In typical video codecs the prediction residual after motion compensation is first transformed with a transform kernel, such as DCT, and then coded. The reason for this is that often there still exists some correlation among the residual and transform can in many cases help reduce this correlation and provide more efficient coding.

[0076] Typical video encoders may use Lagrangian cost functions to find optimal coding modes, e.g. the desired Macroblock mode and associated motion vectors. This kind of cost function uses a weighting factor A. to tie together the (exact or estimated) image distortion due to lossy coding methods and the (exact or estimated) amount of information that is required to represent the pixel values in an image area:C = D + ARwhere C is the Lagrangian cost to be minimized, D is the image distortion (e.g. Mean Squared Error) with the mode and motion vectors considered, and R the number of bits needed torepresent the required data to reconstruct the image block in the decoder (including the amount of data to represent the candidate motion vectors).

[0077] Video coding specifications may enable the use of supplemental enhancement information (SEI) messages or the like. Some video coding specifications include SEI NAL units, and some video coding specifications contain both prefix SEI NAL units and suffix SEI NAL units, where the former type can start a picture unit or alike and the latter type can end a picture unit or alike. An SEI NAL unit contains one or more SEI messages, which are not required for the decoding of output pictures but may assist in related processes, such as picture output timing, post-processing of decoded pictures, rendering, error detection, error concealment, and resource reservation. Several SEI messages are specified in H.264 / AVC, H.265 / HEVC, H.266 / VVC, and H.274 / VSEI standards, and the user data SEI messages enable organizations and companies to specify SEI messages for their own use. The standards may contain the syntax and semantics for the specified SEI messages but a process for handling the messages in the recipient might not be defined. Consequently, encoders may be required to follow the standard specifying a SEI message when they create SEI message(s), and decoders might not be required to process SEI messages for output order conformance. One of the reasons to include the syntax and semantics of SEI messages in standards is to allow different system specifications to interpret the supplemental information identically and hence interoperate. It is intended that system specifications can require the use of particular SEI messages both in the encoding end and in the decoding end, and additionally the process for handling particular SEI messages in the recipient can be specified.Neural Network Based Image / Video Coding

[0078] Neural networks (NNs) have been used in the context of image and video compression, by following mainly two approaches.

[0079] In a first approach, NNs are used to replace one or more of the components of a traditional codec such as a VVC / H.266-compliant codec. Here, "traditional" means those codecs whose components and parameters are typically not learned from data by means of machine learning techniques. Examples of components that may be implemented as neural networks are:• An in-loop filter, for example a NN that works as an additional in-loop filter with respect to the traditional loop filters, or a NN that works as the only additional in-loop filter, thus replacing any other in-loop filter.• Intra-frame prediction.• Inter-frame prediction.• Transform and / or inverse transform.• Probability model for lossless coding.

[0080] In a second approach, commonly referred to as "end-to-end learned compression" (or end-to-end learned codec), NNs are used as the main components of the image / video codecs. However, the codec may still comprise components which are not based on machine learning techniques. In this second approach, two design options are as follows:

[0081] Option 1: re-use the traditional video coding pipeline, but replace most or all the components with NNs. In this example, the forward and inverse transforms may be replaced with two neural networks. Additionally, or alternatively, the loop filter may be neural network.

[0082] Option 2 (also referred to as end-to-end learned coding): re-design the whole pipeline as a neural network auto-encoder with a quantization and lossless coding in the middle part, as follows:• Encoder NN (also referred to as neural network based encoder, or NN encoder):performs a non-linear transformation of the input. The output is typically referred to as latent tensor.• Quantization and lossless encoding of the encoder NN's output.• Lossless decoding and dequantization.• Decoder NN (also referred to as neural network based decoder, or NN decoder):performs a non-linear inverse transformation from dequantized latent tensor to a reconstructed input.

[0083] It is to be understood that even in end-to-end learned approaches, there may be components which are not learned from data, such as the arithmetic codec.Neural Network-based End-to-end Learned Video Coding

[0084] FIG. 5 illustrates diagram 550 of an example neural network-based end-to-end learned coding system, such as an end-to-end learned video coding system or an end-to-end learned image coding system. Even though some examples are provided with respect to coding images or videos, it is to be understood that other types of data may be coded in a similar way, such as audio, speech, text, features, etc. As shown in FIG. 5, a typical neural network-based end-to-end learned coding system 550 contains an encoder 552 and a decoder 554.

[0085] The encoder 552 comprises an encoder NN 556, a quantizer or quantization 558, a probability model 560, a lossless encoder 562, such as an arithmetic encoder. The decoder 554 comprises a lossless decoder 564, such as an arithmetic decoder, a probability model 566, a dequantizer or dequantization 568, and a decoder NN 570.

[0086] It is to be noted that the probability model 560 present at encoder side and the probability model 566 present at decoder side may be same or substantially the same. For example, they may be two copies of the same probability model.

[0087] The lossless encoder 562 and the lossless decoder 564 form a lossless codec 574. Lossless codec 574 may be, for example, an entropy-based lossless codec. An example of lossless codec is an arithmetic codec, such as a context-adaptive binary arithmetic coding (CABAC).

[0088] Encoder NN 552 and decoder NN 554 may be two neural networks, or mainly comprise of neural network components.

[0089] Probability model 560 or 566 may also be a neural network and / or comprise mainly neural network components, and may be referred to as neural network based probability model or learned probability model.

[0090] Sometimes, the term lossless codec may refer to a system that comprise also the probability model, in addition to, for example, an arithmetic encoder and an arithmetic decoder.

[0091] The quantizer 558, dequantizer 568 and lossless codec 574 are typically not based on neural network components, but they may also comprise neural network components.

[0092] The encoder NN 556 takes an input x, which may comprise, for example, an image to be compressed. The encoder NN 556 outputs a latent tensor z. In one example, the latent tensor may be a 3D tensor, where the three dimensions of such tensor represent a channel dimension, a vertical dimension (also sometimes referred to as height dimension) and a horizontal dimension (also sometimes referred to as width dimension). In another example, the latent tensor may be a 4D tensor, where the four dimensions of such tensor represent sample dimension (also sometimes referred to as batch dimension, which is the dimension along which different samples of data can be placed), a channel dimension, a vertical dimension (also sometimes referred to as height dimension) and a horizontal dimension (also sometimes referred to as width dimension). The latent tensor is input to a quantization operation 558, obtaining a quantized latent tensor zq. The quantized latent tensor may be lossless-encoded into a bitstream b by the lossless encoder 562, based also on the output of the probability model 560. In particular, the probability model 560 takes as input at least part of the quantized latent tensor and outputs an estimate of a probability or an estimate of a probability distribution or an estimate of one or more parameters of a probability distribution for one or more elements of the quantized latent tensor. The bitstream represents an encoded or compressed version of the input x.

[0093] The bitstream may be lossless-decoded by the lossless decoder 564 also based on the output of the probability model 566 present at decoder side 554, obtaining a quantized latent tensor z<,. The quantized latent tensor is dequantized 568, obtaining a reconstructed latent tensor z. The reconstructed latent tensor is input to a decoder NN, obtaining a reconstructed input x, i.e., a reconstructed version of the input x. The reconstructed input may also be referred to as reconstructed data, or reconstruction, or decoded data, or decoded input, or decoded output, or the like.

[0094] This is a simplified description of an end-to-end learned codec, and it is to be understood that more sophisticated designs or variations of this design may be possible.

[0095] In some embodiments, the neural network components, or a subset of the neural network components, of an end-to-end learned codec may be trained by minimizing a ratedistortion loss function:L = D + XR,where D is a distortion loss term, R is a rate loss term, and X is a weight that controls the balance between the two losses.

[0096] The distortion loss term may be referred to also as reconstruction loss term, or simply reconstruction loss. The rate loss term may be referred to simply as rate loss. The distortion loss term measures the quality of the reconstructed or decoded output, and may comprise (but may not be limited to) one or more of the following:• Mean square error (MSE)• Structure similarity (SSIM)• MS-SSIM• Losses derived from the use of a pretrained neural network. For example, error(fl, f2), where fl and f2 are the features extracted by a pretrained neural network for the input data and the decoded data, respectively, and error() is an error or distance function, such as LI norm or L2 norm.• Losses derived from the use of a neural network that is trained simultaneously with the end-to-end learned codec. For example, adversarial loss can be used, which is the loss provided by a discriminator neural network that is trained adversarially with respect to the codec, following the settings proposed in the context of Generative Adversarial Networks (GANs) and their variants.• Loss that is related to a performance of one or more machine analysis tasks or to an estimated performance of one or more machine analysis tasks, where the one or more machine analysis tasks may comprise classification, object detection, image segmentation, instance segmentation, etc. In one example, the estimatedperformance of one or more machine analysis tasks may comprise a distortion computed based at least on a first set of features extracted from an output of the decoder and a second set of features extracted from a respective ground truth data, where the first set of features and the second set of features are output by one or more layers of a pretrained feature-extraction neural network.

[0097] Multiple distortion losses may be used and integrated into D, such as a weighted sum of MSE and SSIM.

[0098] The rate loss term may be used to train the encoder NN to output a low-entropy latent tensor, or a latent tensor such that the quantized latent tensor has low entropy, or a latent tensor such that the probability distribution of the quantized latent tensor can be better estimated or predicted by the probability model.

[0099] The rate loss term may be used to train the probability model to better estimate or predict the probability distribution of the quantized latent tensor.

[0100] Examples of the rate loss terms are the following:• The rate loss term may be derived from the output of the probability model, and it represents the estimated entropy of the quantized latent representation, which indicates the number of bits necessary to represent the quantized latent tensor. • A sparsification loss, such as a loss that encourages the quantized latent tensor to comprise many zeros. Examples are L0 norm, LI norm, LI norm divided by L2 norm.

[0101] To train the neural network components, or a subset of the neural network components, of an end-to-end learned codec, one or more of reconstruction losses may be used, and one or more rate losses may be used. In one example, the one or more reconstruction losses and / or one or more rate losses may be combined by means of a weighted sum. Typically, the different loss terms may be weighted using different weights, and these weights determine how the final system performs in terms of rate-distortion performance. For example, if more weight is given to the reconstruction losses with respect to the rate losses, the system may learn to compress less but to reconstruct with higher accuracy (as measured by a metric that correlates with the reconstruction losses). These weights are usually considered to be hyper-parameters of the training process, and may beset manually by the person designing the training process, or automatically for example by grid search or by using additional neural networks.

[0102] In one example process, the training process may be performed jointly with respect to the distortion loss D and the rate loss R. In another example, the training process may be performed in two alternating phases, where in a first phase only the distortion loss D may be used, and in a second phase only the rate loss R may be used.

[0103] For lossless video / image compression, the system would comprise only the probability model and lossless encoder and lossless decoder. The loss function would comprise only the rate loss, since the distortion loss is always zero (i.e., no loss of information).

[0104] In this disclosure, inference phase, or inference stage, or inference time, or test time is referred to as the phase when a neural network or a codec is used for its purpose, such as encoding and decoding an input image.Video Coding for Machines (VCMs)

[0105] Reducing the distortion in image and video compression is often intended to increase human perceptual quality, as humans are considered to be the end users, i.e. consuming / watching the decoded images or videos. Recently, with the advent of machine learning, especially deep learning, there is a rising number of machines (i.e., autonomous agents) that analyze data independently from humans and that may even take decisions based on the analysis results without human intervention. Examples of such analysis are object detection, scene classification, semantic segmentation, video event detection, anomaly detection, pedestrian tracking, etc. For example, such analysis tasks may be performed by neural networks.

[0106] It is likely that the device where the analysis takes place has multiple "machines" or neural networks (N Ns). These multiple machines may be used in a certain combination which is for example determined by an orchestrator sub-system. The multiple machines may be used for example in succession, based on the output of the previously used machine, and / or in parallel. For example, a video may be analyzed by one machine (NN) for detectingpedestrians, by another machine (another NN) for detecting cars, and by another machine (another NN) for estimating the depth of all the pixels in the frames.

[0107] Example use cases and applications are self-driving cars, video surveillance cameras and public safety, smart sensor networks, smart TV and smart advertisement, person reidentification, smart traffic monitoring, drones, etc. In addition to image and video data, automatic analysis and processing is increasingly been performed for other types of data, such as audio, speech, text.

[0108] Compressing (and decompressing) data where the end user comprises machines (e.g., neural networks) is commonly referred to as compression or coding for machines. In the case of video data, it is referred to as video compression or coding for machines (VCM).

[0109] Compressing for machines may differ from compressing for humans for example with respect to the algorithms and technology used in the codec, or the training losses used to train any neural network components of the codec, or the evaluation methodology of codecs.

[0110] It is to be understood that, when considering the case of coding for machines, the terms "receiver-side" or "decoder-side" refer to the physical or abstract entity or device which contains one or more machines, and runs these one or more machines on some encoded and eventually decoded video representation which is encoded by another physical or abstract entity or device, the "encoder-side device".

[0111] A VCM encoder may be used to encode input video into a bitstream. A bitrate may be computed from the bitstream, as a measure of the size of the bitstream. A VCM decoder decodes the bitstream that was produced by the VCM encoder. Output data may be considered as the decoded or reconstructed video. However, in some implementations of this pipeline, this data may not have same or similar characteristics as the original video which was input to the VCM encoder. For example, this data may not be easily understandable by a human by simply rendering the data onto a screen. The output of VCM decoder is then input to one or more task neural networks. One goal of VCM may be to obtain a low bitrate while guaranteeing that the task-NNs still perform well in terms of the evaluation metric associated to each task.

[0112] It is to be understood that, in some cases, the VCM decoder may not be present. In one example, the machines may run directly on the bitstream. In some other cases, the VCM decoder may comprise only a lossless decoding stage, and the lossless decoded data is provided as input to the machines. In yet some other cases, the VCM decoder may comprise a lossless decoding stage following by a dequantization operation, and the loss-decoded and dequantized data is provided as input to the machines.

[0113] When a conventional video encoder, such as a H.266 / VVC encoder, is used as a VCM encoder, one or more of the following approaches may be used to adapt the encoding to be suitable to machine analysis tasks:• One or more regions of interest (ROIs) may be detected. An ROI detection method may be used. For example, ROI detection may be performed using a task NN, such as an object detection NN. In some cases, ROI boundaries of a group of pictures or an intra period may be spatially overlaid and rectangular areas may be formed to cover the ROI boundaries. The detected ROIs (or rectangular areas, likewise) may be used in one or more of the following ways:o The quantization parameter (QP) may be adjusted spatially in a manner that ROIs are encoded using finer quantization step size(s) than other regions. For example, QP may be adjusted CTU-wise.o The video is preprocessed to contain only the ROIs, while the other areas are replaced by one or more constant values or removed.o The video is preprocessed so that the areas outside the ROIs are blurred or filtered.o A grid is formed in a manner that a single grid cell covers a ROI. Grid rows or grid columns that contain no ROIs are downsampled as preprocessing to encoding.Quantization parameter of the highest temporal sublayer(s) is increased (i.e. coarser quantization is used) when compared to practices for human watchable video.• The original video is temporally downsampled as preprocessing prior to encoding. A frame rate upsampling method may be used as postprocessing subsequent to decoding, if machine analysis at the original frame rate is desired.• A filter is used to preprocess the input to the conventional encoder. The filter may be a machine learning based filter, such as a convolutional neural network.

[0114] It is to be understood that, in the context of video coding for machines, the terms "machine vision", "machine vision task", "machine task", "machine analysis", "machine analysis task", "computer vision", "computer vision task", "task network" and "task" may be used interchangeably.

[0115] Also, it is to be understood that, in the context of video coding for machines, the terms "machine consumption" and "machine analysis" may be used interchangeably.Neural Network Based Filtering

[0116] A neural network may be used for filtering or processing input data. A neural network may be referred to as a neural network based filter, or simply NN filter. A NN filter may comprise one or more neural networks, and / or one or more components that may not be categorized as neural networks.

[0117] The purpose of a NN filter may comprise (but may not be limited to) visual enhancement, colourization, upsampling, super-resolution, inpainting, temporal extrapolation, generating content, and the like.

[0118] In some video codecs, a neural network may be used as a filter in the encoding and decoding loop (also referred to simply as coding loop), and it may be referred to as neural network loop filter, or neural network in-loop filter. The NN loop filter may replace all other loop filters of an existing video codec, or may represent an additional loop filter with respect to the already present loop filters in an existing video codec.

[0119] A neural network filter may be used as post-processing filter for a codec, e.g., may be applied to an output of an image or video decoder in order to remove or reduce coding artifacts.

[0120] In one example, a codec is a modified VVC / H.266 compliant codec (e.g., a VVC / H.266 compliant codec that has been modified and thus it may not be compliant to the VVC / H.266) that comprises one or more NN loop filters. An input to the one or more NN loop filters may comprise at least a reconstructed block or frames (simply referred to as reconstruction) or data derived from a reconstructed block or frame (e.g., the output of a conventional loop filter). The reconstruction may be obtained based on predicting a block or frame (e.g., by means of intra-frame prediction or inter-frame prediction) and performing residual compensation. The one or more NN loop filters may enhance the quality of at least one of their input, so that a rate-distortion loss is decreased. The rate may indicate a bitrate (estimate or real) of the encoded video. The distortion may indicate a pixel fidelity distortion such as the following:• Mean-squared error (MSE)• Mean absolute error (MAE)• Mean Average Precision (mAP) computed based on the output of a task NN (such as an object detection NN) when the input is the output of the post-processing NN. • Other machine task-related metric, for tasks such as object tracking, video activity classification, video anomaly detection, etc.

[0121] The enhancement may result into a coding gain, which can be expressed for example in terms of BD-rate or BD-PSNR.

[0122] A neural network filter may be used as post-processing filter for a codec, e.g., may be applied to an output of an image or video decoder in order to remove or reduce coding artifacts. In one example, the NN filter may be used as a post-processing filter where the input comprises data that is output by or is derived from an output of a traditional decoder, such as a decoder that is compliant with the VVC / H.266 standard. In another example, the NN filter may be used as a post-processing filter where the input comprises data that is output by or is derived from an output of a decoder of an end-to-end learned decoder.Input to a NN filter

[0123] In the case of filtering images, a filter may take as input at least one or more first images to be filtered and may output at least one or more second images, where the one or more second images are the filtered version of the one or more first images. In one example, the filter takes as input one image and outputs one image. In anotherexample, the filter takes as input more than one image and outputs one image. In another example, the filter takes as input more than one image and outputs more than one image.

[0124] It is to be understood that a filter may take as input also other data (also referred to as auxiliary data, or extra data) than the data that is to be filtered, such as data that can aid the filter to perform a better filtering than if no auxiliary data was provided as input. In one example, the auxiliary data comprises information about prediction data, and / or information about the picture type, and / or information about the slice type, and / or information about a Quantization Parameter (QP) used for encoding, and / or information about boundary strength, etc. In one example, the filter takes as input one image and other data associated to that image, such as information about the quantization parameter (QP) used for quantizing and / or dequantizing that image, and outputs one image.Overfitting a neural network filter

[0125] A NN filter may be adapted at test time based at least on part of the data to be encoded and / or decoded and / or post-processed. Although, for simplicity, the case of a NN filter is being considered herein, similar adaptations may be performed for other coding tools and / or post-processing tools that are based on neural network technology. For example, a neural network based intra-frame prediction, a neural network based inter-frame prediction, or the like.

[0126] Such operation may be referred to, for example, with one of the following terms, when their meaning is clear from the context: adaptation, content adaptation, overfitting, fine-tuning, optimization, specialization, and the like.

[0127] The NN filter that results from the adaptation process may be referred to, for example, with one of the following terms: adapted filter, content-adapted filter, overfitted filter, fine-tuned filter, optimized filter, specialized filter, or the like.

[0128] The overfitting process may be performed at encoder side based on a training process. The resulting overfitted filter is then used to derive an overfitting signal, or adaptation signal. The adaptation signal may be compressed and then signaled from encoder to decoder, in or along a bitstream that represents encoded data, such as an encoded image or video. FIG. 6A illustrates an example of such encoder-side operations in block diagram 600A.

[0129] In FIG. 6A, during overfitting process 602, x represents an input to the NN filter 604, x represents an output of the NN filter 604, x represents a ground-truth data associated with x, "Compute loss" 606 computes a training loss I in order to overfit the NN filter 604, "Overfit" 608 uses I to overfit the NN filter 604. As a result of the overfitting process 602, an overfitted NN filter 610 is obtained, which is used, together with the original NN filter 612, to derive 614 an adaptation signal 616. The adaptation signal 616 is compressed 618 into a compressed adaptation signal 620 and signaled 620 to a decoder or receiver.

[0130] FIG. 6B illustrates an example of decoder-side, receiver-side, operations in block diagram 600B. In FIG. 6B, at decoder side, the overfitting signal, compressed adaptation signal 620, is decompressed 624 into decompressed adaptation signal 626. Signal 626, or a signal derived therefrom, is used to update 628 the NN filter 630. The updated NN filter 632 is then used to filter one or more pictures, or one or more blocks.

[0131] The NN filter that is obtained from the overfitting process at encoder side may be different from the NN filter that is obtained from the updating process at decoder side. For example, one reason may be that the adaptation signal may be compressed in a lossy way. Thus, the former NN filter may be referred to as overfitted filter or adapted filter (or other similar terms, see above), and the latter NN filter may be referred to as updated filter.

[0132] The overfitting process performed at encoder side will now be described in more detail. The adaptation process may start with an initial NN filter. In one example, the initial NN filter is a pretrained NN filter, which was pretrained during an offline stage on a sufficientlylarge dataset. In another example, the initial NN filter may be a randomly initialized NN filter. In the adaptation, one or more parameters of the NN filter may be adapted. Examples of such parameters may include (but may not be limited to) the following:• The bias terms of a convolutional neural network.• Multiplier parameters, that multiply one or more tensors produced by the NN filter, such as one or more feature tensors that are output by respective one or more layers of the NN filter.• Parameters of the kernels of a convolutional neural network.• Parameters of an adapter layer.• One or more arrays or tensors that are used as input to respective one or more layers of the NN filter.

[0133] The adaptation may be performed by a training process, such as by minimizing a loss function until a stopping criterion is met. The data used for this training process may comprise of one or more pictures or blocks of input to the NN filter and associated respective one or more pictures or blocks of ground-truth data. In one example where the filter is an in-loop filter, the input to the NN filter is reconstruction data, after prediction and residual compensation; the ground-truth data is the uncompressed data that is given as input to the encoder. In one example where the filter is a post-processing filter, the input to the NN filter is decoded data (e.g., the output of a video decoder); the ground-truth data is the uncompressed data that is given as input to the encoder.

[0134] The loss function used during the training process may comprise one or more distortion loss functions (also referred to as reconstruction loss functions) and zero or more rate loss functions. A rate loss function may measure, for example, the cost in terms of bitrate of signaling any adaptation signal, such as updates to the parameters of the NN filter. A distortion loss function may comprise one of MSE, MS-SSIM, VMAF, etc.Deriving the adaptation signal

[0135] The adaptation signal may be derived based on the adapted NN filter and on the original NN filter (i.e., the NN filter before the overfitting process).

[0136] In one example, the adaptation signal may include an update to one or more parameters of the NN filter. Such an update may be referred to as a weight update, or parameter update. Such update may be computed, for example, by subtracting the values of the adapted parameters (i.e., the parameters of the adapted NN filter) from the corresponding values of the original parameters (i.e., the parameters of the original NN filter).

[0137] In another example, the adaptation signal comprises the parameters (of the NN filter) that were adapted, also referred to as updated parameters, or adapted parameters, or adapted weights, or overfitted parameters, and the like.Compression of adaptation signal

[0138] In order to keep the size of the adaptation signal low, the adaptation signal may go through one or more compression steps, such as sparsification, quantization and lossless coding.

[0139] In one example, an encoder that compresses the adaptation signal into a bitstream that is compliant with a neural network compression standard, such as MPEG NNC, may be used.Signaling

[0140] The compressed adaptation signal may be signaled from encoder to decoder in or along a bitstream that represents encoded image or video data.

[0141] In one example, the compressed adaptation signal is signaled in an Adaptation Parameter Set (APS) syntax structure of a video coding bitstream.

[0142] In another example, the compressed adaptation signal is signaled in a Supplemental Enhancement Information (SEI) message of a video coding bitstream.

[0143] Signaling may include other information which is associated with the adaptation signal and that may be required for correctly parsing and / or decompressing and / or using the adaptation signal, such as any quantization parameters.Decoder or Receiver Side Operations

[0144] At decoder side, the signaled compressed adaptation signal is received and decompressed. The decompressed adaptation signal may then be used to update the NN filter. In one example, where the adaptation signal comprises a weight update, where the weight update comprises one or more updates to respective one or more parameters of the NN filter, the one or more updates are added to the one or more parameters. In another example, where the adaptation signal comprises one or more updated or adapted parameters, the one or more updated or adapted parameters are used to replace respective one or more parameters of the NN filter. Once the NN filter has been updated based on the adaptation signal, the updated NN filter may be used for its purpose. For example, for filtering an input picture or an input block.Example including latent tensor overfitting

[0145] Phase 1: overfitting. Overfitting is performed for a certain number of iterations. At each iteration, the input image is input to the encoding pipeline, such as Encoder NN, and quantizer, and a latent tensor is obtained. The latent tensor is then input to the probability NN used for lossless coding, and to the decoding pipeline, such as dequantization and Decoder NN. In case of overfitting the latent tensor, the Encoder NN is used only once at the beginning in order to obtain an initial version of the latent tensor which would then be updated by the overfitting process. The output of the probability NN may be used for computing a rate loss. The output of the decoder may be used for computing a reconstruction loss. These losses are then used for overfitting the selected Overfitting Parameters, for example by using gradientdescent and backpropagation. The number of overfitting iterations may depend on the available amount of time for overfitting, on the learning rate used for overfitting, and on the target performance that is desired. Higher learning rate may provide faster overfitting, and thus fewer overfitting iterations may be needed to achieve a desired target performance.However, too high learning rate may cause divergence and thus not good final performance. The stopping criterion for the overfitting may be one of the following: the loss does not decrease anymore (or starts to increase), the available time for overfitting is over. In case of overfitting some of the Decoder NN's parameters, a compression strategy may be used for encouraging the updates to the overfitted Decoder NN's parameters to require low bitrate when sending them to the decoder side. This may comprise using an additional loss term during overfitting, or a quantization operation, which are applied on the updates to the overfitted Decoder NN's parameters. Examples of such loss terms are loss terms that encourage the updates to have low entropy, or high sparsity, or be robust to quantization, etc.

[0146] Phase 2: final encoding. The overfitted model from Phase 1 may be used for encoding the input image. In case the OPs were the Encoder NN's parameters only, the input image is input to the Encoder NN, quantized and arithmetic-encoded. In case the OPs were the latent tensor, the latent tensor obtained before stopping the overfitting may be quantized and then arithmetic encoded. In case the OPs were some of the Decoder NN's parameters, the latent tensor and the overfitted decoder's parameters (or the latent tensor and the updates to the overfitted decoder's parameters) would be both part of the encoded data to be sent to the decoder-side. The overfitted decoder's parameters or the updates to the overfitted decoder's parameters may be quantized and / or entropy encoded before being provided to the decoder side.Genetic Algorithms

[0147] Genetic Algorithms (GAs) are a class of optimization techniques inspired by the principles of natural selection and genetics. They belong to the broader category of evolutionary algorithms and are used to solve complex optimization and search problems by mimicking the process of natural evolution.Core Concepts

[0148] Population: A genetic algorithm begins with an initial population of potential solutions to the problem at hand. Each individual or instance in the population (e.g., eachpotential solution) is typically represented as a chromosome, which is a string of genes. A gene can be a binary digit, a real number, or other type of data such as a neural network weight, depending on the problem.

[0149] Fitness Function: The fitness function evaluates how well each individual in the population solves the problem. It assigns a fitness score to each individual, guiding the selection process for reproduction.

[0150] Selection: Individuals are selected from the current population based on their fitness scores. The higher the fitness score, the more likely an individual is to be selected for reproduction (e.g., for crossover). Common selection methods include roulette wheel selection, tournament selection, and rank-based selection.

[0151] Crossover (Recombination): Selected individuals (parents) undergo crossover to produce offspring. This process involves exchanging segments of their chromosomes to create new individuals. Crossover aims to combine the best traits of the parents, potentially leading to better solutions.

[0152] Mutation: To maintain genetic diversity within the population and avoid premature convergence, random mutations are introduced. Mutation alters one or more genes in a chromosome, ensuring that the algorithm explores a broader search space.

[0153] Generation: The new population of offspring replaces the old population, and the process repeats for a number of generations or until a stopping criterion is met, such as a satisfactory fitness level or a maximum number of generations.

[0154] In this disclosure, the terms frame, picture and image may be used interchangeably. For example, the input and output to an end-to-end learned codec may be pictures. The input and output of a NN filter may be pictures.Overfitting Neural Networks for Certain Input

[0155] As described herein, in some cases, optimizing one or more aspects of a signal to be decoded, one or more aspects of a decoder, or, even further, one or more aspects of a postprocessing operation (e.g., after a decoding operation) may be beneficial in terms of one ormore evaluation metrics, such as a rate-distortion metric. Optimization may be performed based on an input data item (e.g., an image) and may be referred to as overfitting.

[0156] In some embodiments of the disclosed technology, optimization of a neural network (NN) filter based on an input data item (e.g., an image) to be encoded is described. This optimization of a NN filter may be referred to as an overfitting process. Additionally, example methods are described for overfitting NNs or data signals that are input to NNs on input data. The example methods disclosed aim to enable a higher performance and / or faster adaptability of the NNs when evaluated on the input data.

[0157] While at least some of the herein described embodiments and examples refer to overfitting NNs, it is to be understood that the described embodiments and examples may also apply to or may be valid for overfitting signals that are input to NNs. For example, overfitting a latent tensor that is input to a NN-based decoder.

[0158] In one example embodiment, at training phase, input pictures (or regions from the input pictures) or their features may first be clustered, or grouped, into one or more clusters to have pictures (or regions) with similar characteristics in the same cluster. Any clustering or grouping method is applicable.

[0159] Once the input pictures are grouped into one or more clusters, different versions of one or more aspects of an overfitting process may be used for overfitting on data comprised in different clusters and the best options for each picture (or cluster) may be chosen. One or more aspects of the overfitting process may include but is not limited to the following: optimizer, optimizer's parameters, loss function, loss weights, and layer-wise learning rates. In one non-limiting example, regions comprised in training images may be grouped into three clusters based on a similarity metric, and respective two loss functions (e.g., LI norm and L2 norm) may be used for overfitting one or more parameters of a neural network on regions comprised in the three clusters, to obtain respective two sets of overfitted parameters for each cluster, where the two sets of overfitted parameters are used by the neural network for processing regions comprised in each cluster. The performance of the two sets of overfitted parameters for each cluster may be evaluated and the loss function associated to the set of overfitted parameters with higher performance may be selected as the loss function to be used at inference time for regions that are similar to regions in that cluster. Additionally, oralternatively, when possible with the loss weights, the best option for each cluster may be defined by an average of the best options for each input picture that belongs to the specific cluster.

[0160] Continuing with the example, in a scenario where average cannot be calculated, such as with the optimizers, the best option may be chosen by voting. Each input picture may give a single vote for its best option and after all the pictures are processed, the best option for a specific cluster may be defined as the option that had the most votes.

[0161] At inference, the encoder may measure the similarity between the input clusters and input picture and retrieve the options for overfitting from the best matching cluster. The similarity may be measured according to the mean square error (MSE) or peak signal-to-noise ratio (PSNR) metric between the centroid or center of the input clusters or the input picture. In another example, the similarity may be based on the MSE or PSNR metric between the input picture and the samples or a sub set of samples of each input cluster.

[0162] In some embodiments, the grouping process may be performed based on a quality metric. For example, data may be grouped according to the MSE or PSNR metric, so that data in a same cluster have same or similar quality and data in different clusters have different quality (or more different than in a same cluster).

[0163] In some embodiments, the grouping process may be performed based on one or more encoding parameters, such as according to a quantization parameter (QP).

[0164] In some embodiments, the grouping process may be performed in two stages. At the first stage, the best option of the overfitting parameters for each input picture may be derived from the overfitting process. In a second stage, the input pictures may be grouped according to the similarity between the derived best option. The similarity between the best options may be a weight sum of the distance between each pair of corresponding overfitting parameters.

[0165] In some embodiments, during the overfitting process, one or more additional loss functions or one or more different loss functions may be used to improve the performance of the overfitted neural network.

[0166] In some embodiments, edge related loss may be assigned as an extra loss component for low quality input data. The quality threshold may be determined for example by QP or MSE against the uncompressed picture. Additionally, the edge loss may be extracted using an edge detection operator, such as Sobel, Laplacian, or Canny.

[0167] In some embodiments, regions, areas, orfeatures of the input may be assigned more weight than others. Weights may be assigned using, for example, with a weight mask when calculating the distortion loss. In one example, an output of a NN may be used to compute a loss, where the loss is a 2-dimensional tensor (e.g., a matrix), and the loss is multiplied by a mask, where the mask is a matrix with values in the range [0, 1], to obtain a masked or weighted loss. The masked or weighted loss may then be used for overfitting one or more parameters of the neural network, e.g., for computing one of more gradients with respect to one or more parameters of the neural network.

[0168] In a related embodiment, the low loss values may be suppressed by multiplying the loss tensor by a binary mask, where the mask has Os for small errors and Is for high errors. The classification into small or high error may be based on a threshold or on a percentage (e.g., 50% or 70% of the highest errors). This embodiment may be applied with or without an overfitting process.

[0169] In a related embodiment, based on a frequency analysis of an input picture, more emphasis may be given to a certain frequency band. For example, the input, or the output, may be converted to frequency domain via Fast Fourier Transform (FFT) and zero-frequency shifted to the center. Alternatively, the input, or the output, may be converted to frequency domain via Discrete Cosine Transform (DCT). Different frequency bands may be extracted from different areas of the spectrum. In a case of three frequency bands, the low-frequency band may be presented by a circle around the spectrum center, medium frequency band as a ring around the low-frequency band and the high frequency band as outer regions of the frequency spectrum. Different frequency-bands may be assigned different weights via a weight mask during the loss calculation, which may also be carried out in frequency domain. This embodiment may be applied with or without an overfitting process.

[0170] In yet another embodiment, the used loss function may be determined based on QP. For example, for low QP (high quality), there will be many samples with high quality and fewsamples with low quality, which represent outliers, thus, L2 loss may be used. Instead, for high QP (low quality), all samples would be low quality, as there are no outliers, LI loss may be used. This embodiment may be applied with or without an overfitting process.

[0171] In another embodiment, an auxiliary NN may be trained to output the auxiliary information which is used in the overfitting process. The auxiliary information may be different for each iteration (one or more batches or epochs) of the overfitting process. The auxiliary information may be for example:• Weight tensor (to weight the training loss or parts of it, can comprise binary values or integer values, real values, quantized values, floating-point values, either on pixel or frequency domain)• Optimizer index, indicating a particular optimizer• Optimizer parameters such as learning rate and momentum, or indices mapped to certain optimizer parameter values• Learning rates for different layers of the network• Loss function index or map• Loss weights, or indices that indicate certain loss weight values• Indications of a particular neural network or of a particular portion of a neural network • Indications of one or more inputs and / or one or more auxiliary inputs to be used as input to the neural network, such as a particular previously decoded picture, or a particular QP value.• Indications of one or more modifications to respective one or more inputs or to respective one or more auxiliary inputs to the neural network, such as adding 1 to a QP value that is provided as auxiliary input.• Indications of one or more modifications to an output of the neural network.• Indications of one or more combination parameters for a combination operation that combines an output of the neural network with another signal, where the one or morecombination parameters may comprise, for example, a combination method or a combination weight value or coefficient.• Indications of one or more data items to be used to derive a ground-truth for computing a loss in the overfitting process. Optionally, also one or more ground-truth derivation parameters may be indicated, to be used for deriving the ground-truth, such as one or more combination parameters for combining two or more ground-truth data items into one ground-truth. In one example, the indications of one or more data items comprise an indication of using a particular ground-truth data item that has been filtered according to a particular filtering process, such as a motion-compensated temporal filter.• Combination of one or more of the forementioned auxiliary information.

[0172] Turning to FIG. 7, possible input variations are illustrated by variations 702, 704, 706, 708, and 710. As shown, the auxiliary NN 714 may take as its input the reconstructed picture 712, features 724 from the reconstructed picture 712, a filtered picture 726, an uncompressed picture 720 or a combination of them ortheirfeatures. In some embodiments, features 724 may be extracted from reconstructed picture 712 using NN filter 722. Additionally, or alternatively, filtered picture 726 may be generated from reconstructed picture 712 using NN filter 722. Auxiliary info 716 that is generated by Auxiliary NN 714 may be provided as input to Overfit NN filter 718.

[0173] In some embodiments, the Auxiliary NN may operate on pixel domain or frequency domain. For example, in the case of doing frequency analysis with the auxiliary NN, where auxiliary info contains the weight matrix on frequency domain, the inputs of the auxiliary NN may be converted to frequency domain for easier training.

[0174] Training of the auxiliary NN will now be described. In at least one embodiments, ground truth options for each input picture may be generated by running the overfitting process for each input picture and choosing the best combination of options based on the training loss. After the ground truth is generated, the auxiliary NN may be trained for classification tasks, regression tasks, or both. This depends on the format of generated ground truths, which on the other hand depends on what auxiliary information is outputted.

[0175] In another embodiment, meta-learning, for example according to the Model-Agnostic Meta Learning (MAML) algorithm, may be used to train the auxiliary NN. In one example, a training process comprises a nested training process, where an external or main training loop contains a nested or internal training loop, and where at each iteration of the main training loop, two or more internal training loops are performed for overfitting the neural network according to the auxiliary NN's outputs, to obtain respective two or more overfitted auxiliary NNs. In this example, the two or more overfitted auxiliary NNs may be evaluated to obtain respective two or more evaluation scores. Subsequently, in the main training loop, the neural network may be updated based on a training loss that is derived based at least on the two or more evaluation scores.

[0176] In some embodiments, the auxiliary NN may be located on the encoder side or the decoder side.

[0177] In some embodiments, a genetic algorithm (GA) may be used to generate ground truth info for each input picture. For example, with the case of auxiliary info containing multiple options that are mapped to certain indices, such as optimizer index and a loss function index, a chromosome may be represented as a list, where each element is an index corresponding to a specific option. The initial population of the GA may be randomly chosen or defined by using a grouping method similar to what was described in an earlier embodiment. Alternatively, a chromosome may be represented as a tensor, where each channel corresponds to a certain option.

[0178] The fitness function for the GA may be the same as training loss and it is used to determine the individuals that will go to the next iteration. The determination may be done by choosing some proportion of the individuals with the best fitness score.

[0179] In the crossover, one or more of the selected individuals may be combined to a new individual. In the case of the options in the individuals being floating point numbers, the new option may be determined by a mean of the option values or mean of the option values weighted with the fitness score. In the case of options in the individuals being index values, the new value may be chosen by random selection or voting.

[0180] In the mutation part, for one or more element in chromosome, the value of the element may be changed. For example, in the case the element values are indices, an index may be changed to another possible index value. If the element values are floating-point numbers, the mutation may be achieved by adding to or subtracting a small value from the element. Crossover, mutation or both of them may be skipped in certain iterations of GA.

[0181] After one or more iterations of the GA, the individual with the best performance / fitness may be chosen as the ground truth corresponding to a certain input picture. This ground truth may be used to train the Auxiliary NN.

[0182] Regarding all the embodiments that use ground truth-based training: If the auxiliary info contains options that directly affect to the amount of loss, making the comparison of a set of two different options hard, the "goodness" of the set of options may be measured using some predetermined evaluation options.

[0183] FIGs. 8A and 8B illustrate example implementations 800A and 800B of an encoder 802 and a decoder 804. In a first example embodiment shown in 800A, at inference time, the parameters (or updates to the parameters) 806 of the NN filter 718 that is overfitted with the help of auxiliary information 716, may be signaled to NN filter 722A of the decoder 804. At the decoder, only NN filtering is applied by NN filter 722A.

[0184] In a second example embodiment shown in 800B, auxiliary information 716 or data derived from it can be used as an auxiliary input to the NN filter 722A. The auxiliary input is signaled to the decoder 804, where it is used similarly as auxiliary input 808 to the NN filter 722A.

[0185] For example, if auxiliary info contains a weight matrix, in addition to using it to weight the loss in the overfitting phase, it may be concatenated to the input to the NN in the overfitting phase as well as on the decoder side.

[0186] In another example, a weight tensor may be used to process an output of a NN (e.g., an output of the NN filter), such as in a linear combination of an output of the NN with an input to the NN, where the weight tensor may be used to derive weighting coefficients for the linear combination.

[0187] In some embodiments, an overfit loss term may be used during the training of the NN in order to make the inference-time overfitting easier by taking the overfitting process into account already at the offline training stage. For example, to improve one or more aspects of the overfitting process happening at encoding time, the training of the NN may perform overfitting operations based on an overfit loss.

[0188] In one embodiment the total loss used for training the NN consists of main loss and overfit progress loss. In this example, the overfit progress loss represents the overfit loss. Additionally, or alternatively, the overfit progress loss may be determined based at least on two or more overfitting losses, such as based on an initial overfitting loss and a final overfitting loss.

[0189] The total loss may be defined for example as:totaljoss = mainjoss * alpha*(l / overfit_progressjoss), where mainjoss corresponds to normal NN training loss e.g. a distortion loss, alpha is a scalar for weighing, and overfit progress loss, which is also a scalar, is defined as:overfit_progress_loss = initial_overfit_loss - final_overfit_loss, where initial_overfit_loss is equal to the scalar value of mainjoss, and final_overfit_loss is the scalar value of the loss after one or more iterations of overfitting some parts of the network. Iteration in this case may refer for example to one or more epochs or one or more batches. It is to be noted that the process of overfitting does not directly update the final parameters of the trained network. Therefore, the network is either copied for the overfitting process or the network's state is copied and the state restored after the overfitting process. The parts to overfit may be one or more weights, biases, multipliers or trainable rank decomposition matrices.

[0190] Embodiments described herein may be applicable to be included in the overfittable network training pipeline, in order to benefit more from them in the inference.

[0191] In some embodiments, the learning rate of the training process may be adjusted using the overfit progress loss term. For example, the learning rate may be inverse scaled by the overfit progress loss term: I r_(t+l)= lr_t*(a / (overfit_ progressjoss)), where a is a scalar; to attenuate the parameter updates when the overfitting iteration significantly improves theperformance of the NN. Conversely, the parameter updates may be signified when the overfitting iteration does not provide gains in performance.

[0192] In some embodiments a NN (referred to as NN info extractor) may be trained to extract information from an input picture, which is then used to construct another NN that can be overfitted better. This extracted NN information may be, for example, parameter indices, layer indices, NN indices or indices mapped to certain subnetworks inside one main network, identifiers of layer types, identifiers of neural network architecture types, etc. The input to the NN info extractor, for example, may be the reconstructed picture, features of a reconstructed picture, an uncompressed picture, features of an uncompressed picture or any combination of them.

[0193] In one embodiment, the NN information may include information about an architecture of the NN. In another embodiment, the NN information may include information about one or more modifications to an architecture of the NN. In another embodiment, where the NN comprises two or more portions, the NN information may comprise information about an architecture or a modification of an architecture for one or more of the two or more portions of the NN.

[0194] It is to be noted that the pretrained neural networks in the following embodiments may be trained by using one or more methods from previous embodiments described herein.

[0195] FIG. 9 illustrates the general idea of better overfittable NN creation. Depending on the embodiment, the granularity of the NN info changes, but the high-level idea remains the same. The NN info may be on different granularities, depending on how much overhead via NN info is allowed / desired. In one embodiment, a NN info extractor 906 may be comprised in a decoder, and an output of the NN info extractor 908 (e.g., NN info) may be used to obtain an overfitted NN 918.

[0196] In one embodiment, illustrated by training diagram 902, input 904 is used as input to the NN info extractor 906 which outputs information 908 which indicates the NN structure to be selected. The last layer of NN info extractor can be for example a softmax function or any other nonlinearity from which a one-hot-coded vector can be derived. In this case, the amount of output channels should be same as the amount of pretrained NN structures 912.The one or more pretrained NN structures 912 may be available in the encoder and decoder. The one or more pretrained NN structures may be different with respect to one or more aspects, including (but not limited to) complexity, training loss used for training, training algorithm used for training, types of layers (e.g., convolutional, recurrent, fully-connected, Transformer, etc.). The pretrained NN structures may be of different complexity and shape, e.g. some could be shallow but wide and others deep but narrow, in order to have variety for processing different types of inputs. NN constructor 910 constructs an overfittable NN 914, which may have one or more parameters overfit 916 resulting in overfitted NN 918 as output.

[0197] An example of training and inference of an NN info extractor is illustrated in diagram 1000 of FIG. 10. In this non-limiting example, given N different NNs, they are first all overfitted 1012 to each picture in a dataset and a total distortion loss 1016 is measured 1014. A ground truth is generated 1010 for each picture 1004 as one-hot-coded vectors 1008, where the 1 represents the network with the lowest total loss and 0s represent the other available networks. Each picture 1004 is assigned 1006 its generated one-hot-coded GT and provided as output 1012. Note that the overfitting process is included in the ground truth preparation stage as the total loss may contain a loss component derived from overfitting. Next, in the training phase 1020, only the NN info extractor 1024 is trained as classifier using the input data 1004 and the one-hot-coded ground truth vectors 1012. A classification loss 1028 may be determined based on input data 1004 and its corresponding GT data 1012.

[0198] Resulting NN info extractor outputs a one-hot-coded vector at inference, which corresponds to the best available network structure for the input picture. The one-hot-coded vector may be signaled to the decoder along with the weight updates for the pretrained NN filter, which may then be used to construct the overfitted NN filter, which may then be used to filter the reconstructed picture.

[0199] In another embodiment, the previous embodiment may be run without overfitting aspects, i.e. a classifier is simply trained to choose the best neural network from N available neural networks.

[0200] In another embodiment, a term for complexity is added into the performance measurement, it acts as a weighted regularization to prevent the NN constructorto favor high complexity structures. The complexity may be calculated for example from:• Output of the NN constructor (e.g. summation if applicable)• Number of MACs in the resulting NN structure• Number of layers in the resulting NN structure• Estimated latency of the resulting NN structure• Memory consumption of the resulting NN structure• Runtime of the resulting structure for the input picture• Overfit speed (Time it takes to reach some performance threshold)• Combination of the forementioned complexity measures

[0201] In some embodiments, the complexity term may be used as a scalar that scales the total loss. In some embodiments, the complexity of the constructed NN may be estimated by another neural network, using the data that may be derived from the output of the NN constructor, the descriptors for the NN architecture, input and output of the constructed NN.

[0202] In yet another embodiment, the same training may be applied to more granular NN info, such as layer indices or parameter indices. In this case, the NN info may be for example a binary tensor.

[0203] In another embodiment, the NN structures may not be pretrained. For example, they may include different architectures and randomly-initialized weights.

[0204] In another embodiment, the NN info comprises one or more sequences of symbols that represents an architecture of the NN or of a portion of the NN or a modification of an architecture of the NN or a modification of a portion of the NN, and where the symbols describe or indicate operations such as neural network layers and computational operations.

[0205] In another embodiment, a single pretrained NN may be stored in an encoder and a decoder. The NN info in this case may be, for example, the layer indices, parameter indices or sub-NN indices, such as some predetermined NN structures within the structure of the single pretrained NN. Essentially, anything that may be used to build or choose the structure based on the pretrained NN is applicable. In one example, the pretrained network may be ofmaximum allowed size that can be stored in a decoder. Its size will always be at least as large aS the Size Of NNoverfittable-

[0206] In another related embodiment, the training of the NN may follow training similar to what was described in the case of NN info being NN structure indicator. With this embodiment, the NN info may preferably be sub-NN indices due to the need of ground truth generation but using parameter indices or layer indices are also applicable.

[0207] Turning to FIG. 11, a genetic algorithm (GA) may be used in the ground truth generation. This will be particularly useful if the search space is large, for example, when the search space is a predetermined NN structure and NN info contains the mapping to this structure on a parameter or layer level.

[0208] The GA for ground truth generation may work in a following way: A chromosome can be represented as a tensor, where each element represents whether a parameter shall be chosen from the pretrained structure. An element value of 1 indicates that the corresponding parameter will be chosen, while a value of 0 indicates that it will not be chosen. The initial population of the GA may be for example NN infos that are defined randomly or defined by using a grouping method similar to what was used in earlier embodiments.

[0209] The fitness function for the GA may be defined as f=aL+|3n, where a and |3 are scalars for weighting, L may be for example the overfitting loss, and n may be the number of parameters in the network. In another example, n may be a measurement that reflects the complexity of a NN such as the number of multiply-accumulate operations. In yet another example, n may be an estimate of (or may be related to) an entropy of a signal to be provided to the decoder, such as an estimate of an entropy of the NN info or an estimate of an entropy of a weight update for the NN. In yet another example, n may be an estimate of a bitrate of an encoded signal to be provided to the decoder, such as an estimate of a bitrate of an encoded NN info or an estimate of a bitrate of an encoded weight update forthe NN. Selection of the best individuals may be based on a predetermined value for the fitness score, or by choosing one or more candidates derived from different NN infos that have the best fitness scores.

[0210] In the crossover, one or more of the selected individuals may be combined to a new individual. The process can emphasize combining the individuals that have the same or different heritage or combine randomly based on the iteration. For example, combining individuals that are derived originally from the same NN, gives more emphasis to the optimization aspects from gradient based joint training.

[0211] In the mutation part, for one or more selected NN info, elements in a chromosome tensor with value 1 (retained param) may be set to Os (pruned param) and vice-versa.

[0212] Crossover, mutation or both of them may also be skipped in certain iterations of GA, which further emphasizes the effect of the optimization aspects from gradient based training.

[0213] After certain number of iterations, the individual with the best performance / fitness will be chosen as the ground truth corresponding to a certain input picture. This ground truth may be used to train the NN info extractor similarly as depicted in previous embodiments.

[0214] In another embodiment, a genetic algorithm 1206 may be used in the inference to further enhance the NN info generated by a trained NN info extractor, which is used in the NN constructor at encoder 1202 and sent to the decoder 1204, as shown in the FIG. 12.

[0215] The initial population may be initialized from the NN info by adding different mutations to it. However, the original NN info may also be part of the initial population. Other parts of the genetic algorithm may work similarly as with the ground-truth creation with GA embodiment.

[0216] In a first variation of the inference-time genetic algorithm, illustrated by diagram 1302 of FIG. 13, the genetic algorithm includes the overfit loss to the fitness scores.

[0217] In a second variation, illustrated by diagram 1304 of FIG. 13, the overfit loss and therefore the overfitting process is removed from the genetic algorithm.

[0218] In another embodiment, NN info extractor may be combined with the auxiliary NN from previous embodiments. Hence, the NN info extractor may additionally output the optimal parameters for the overfitting process of the NN0Verfittabie, as well as possible auxiliary inputs for the NNoverfittable-

[0219] In some embodiments of the disclosed technology, an input content selection process may be performed before the overfitting is performed, or while the overfitting is performed, where the content selection process may select a subset of content that is used by the overfitting, and where the subset of content may be representative of or correlated with or similar to (according to one or more correlation or similarity metrics) content in the original image or video. In one example, the content selection process may analyze motion information in the video and may choose not to include into the selected subset any content which is similar to content that is already included in the subset. In another example, the selected subset may include one or more foreground regions or objects, and data representative of background regions or background objects. In yet another example, the content selection process may perform a grouping algorithm on the original content in order to group one or more similar regions into a single cluster, and where content corresponding to a centroid of that single cluster may be included into the selected subset of content.Additional Considerations

[0220] The following examples are provided and described herein.

[0221] In accordance with one aspect, an apparatus including at least one processor in communication with a memory device may be provided. The at least one processor may be configured to cause the apparatus to perform: grouping received input data into one or more clusters; for at least one of the one or more clusters, selecting one or more aspects of an overfitting process, where the one or more aspects are associated to the at least one of the one or more clusters; and for at least one of the one or more clusters, adapting one or more parameters of a neural network using the selected one or more aspects of the overfitting process associated to the at least one of the one or more clusters and using data in the at least one of the one or more clusters.

[0222] The apparatus may include wherein the one or more aspects include one or more of optimizer, optimizer's parameters, loss function, loss weights, and layer-wise learning rates. The apparatus may include wherein the input data include training images and the training images are grouped into clusters based on a similarity metric. The apparatus may include wherein one or more loss functions are used for adapting one or more parameters of theneural network for each cluster. The apparatus may include wherein the instructions further cause the apparatus to perform: evaluating performance of the adapting one or more parameters of the neural network; selecting a loss function associated with a set of the one or more parameters with higher performance; and using the selected loss function at inference time for regions of input data that are similar to regions of corresponding one or more clusters.

[0223] Turning to FIG. 14A, in accordance with at least one embodiment, an example method is provided including: grouping 1402 received input data into one or more clusters; for at least one of the one or more clusters, selecting 1404 one or more aspects of an overfitting process, where the one or more aspects are associated to the at least one of the one or more clusters; and for at least one of the one or more clusters, adapting 1406 one or more parameters of a neural network using the selected one or more aspects of the overfitting process associated to the at least one of the one or more clusters and using data in the at least one of the one or more clusters.

[0224] The method may include wherein the one or more aspects include one or more of optimizer, optimizer's parameters, loss function, loss weights, and layer-wise learning rates. The method may include wherein the input data include training images and the training images are grouped into clusters based on a similarity metric. The method may include wherein one or more loss functions are used for adapting one or more parameters of the neural network for each cluster. The method may include wherein the method further comprises: evaluating 1408 performance of the adapting one or more parameters of the neural network; selecting 1410 a loss function associated with a set of the one or more parameters with higher performance; and 1412 using the selected loss function at inference time for regions of input data that are similar to regions of corresponding one or more clusters.

[0225] In accordance with at least one embodiment, an example apparatus is provided including means for grouping received input data into one or more clusters; for at least one of the one or more clusters, means for selecting one or more aspects of an overfitting process, where the one or more aspects are associated to the at least one of the one or more clusters; and for at least one of the one or more clusters, means for adapting one or more parametersof a neural network using the selected one or more aspects of the overfitting process associated to the at least one of the one or more clusters and using data in the at least one of the one or more clusters.

[0226] In accordance with at least one embodiment, an example non-transitory computer readable medium comprising program instructions may be provided. The instructions, when executed by an apparatus, cause the apparatus to perform at least the following: grouping received input data into one or more clusters; for at least one of the one or more clusters, selecting one or more aspects of an overfitting process, where the one or more aspects are associated to the at least one of the one or more clusters; and for at least one of the one or more clusters, adapting one or more parameters of a neural network using the selected one or more aspects of the overfitting process associated to the at least one of the one or more clusters and using data in the at least one of the one or more clusters.

[0227] In accordance with one aspect, an apparatus including at least one processor in communication with a memory device may be provided. The at least one processor may be configured to cause the apparatus to perform: receiving reconstruction data and uncompressed data; applying the reconstruction data to a neural network filter to obtain first filtered data; computing a main loss based on the first filtered data and the uncompressed data; storing a copy of at least a portion of the neural network filter; applying the reconstruction data to the neural network filter to obtain second filtered data; computing an overfitting loss based on the second filtered data and the uncompressed data; computing one or more gradients of the overfitting loss with respect to one or more overfittable parameters of the neural network filter; using the one or more gradients to update the one or more overfittable parameters of the neural network filter, obtaining an adapted neural network filter; computing an overfit progress loss based on the adapted neural network filter; restoring the neural network filter after the neural network filter had been adapted with overfitting, based on the copy of at least a portion of the neural network filter; and determining a total loss based on a combination of the main loss and the overfit progress loss and updating the neural network filter.

[0228] Turning to FIG. 14B, in accordance with at least one embodiment, an example method is provided including: receiving 1414 reconstruction data and uncompressed data;applying 1416 the reconstruction data to a neural network filter to obtain first filtered data; computing 1418 a main loss based on the first filtered data and the uncompressed data; storing 1420 a copy of at least a portion of the neural network filter; applying 1422 the reconstruction data to the neural network filter to obtain second filtered data; computing 1424 an overfitting loss based on the second filtered data and the uncompressed data; computing 1426 one or more gradients of the overfitting loss with respect to one or more overfittable parameters of the neural network filter; using 1428 the one or more gradients to update the one or more overfittable parameters of the neural network filter, obtaining an adapted neural network filter; computing 1430 an overfit progress loss based on the adapted neural network filter; restoring 1432 the neural network filter after the neural network filter had been adapted with overfitting, based on the copy of at least a portion of the neural network filter; and determining 1434 a total loss based on a combination of the main loss and the overfit progress loss and updating the neural network filter.

[0229] In accordance with at least one embodiment, an example apparatus is provided including means for receiving reconstruction data and uncompressed data; means for applying the reconstruction data to a neural network filter to obtain first filtered data; means for computing a main loss based on the first filtered data and the uncompressed data; means for storing a copy of at least a portion of the neural network filter; means for applying the reconstruction data to the neural network filter to obtain second filtered data; means for computing an overfitting loss based on the second filtered data and the uncompressed data; means for computing one or more gradients of the overfitting loss with respect to one or more overfittable parameters of the neural network filter; means for using the one or more gradients to update the one or more overfittable parameters of the neural network filter, obtaining an adapted neural network filter; means for computing an overfit progress loss based on the adapted neural network filter; means for restoring the neural network filter after the neural network filter had been adapted with overfitting, based on the copy of at least a portion of the neural network filter; and means for determining a total loss based on a combination of the main loss and the overfit progress loss and updating the neural network filter.

[0230] In accordance with at least one embodiment, an example non-transitory computer readable medium comprising program instructions may be provided. The instructions, when executed by an apparatus, cause the apparatus to perform at least the following: receiving reconstruction data and uncompressed data; applying the reconstruction data to a neural network filter to obtain first filtered data; computing a main loss based on the first filtered data and the uncompressed data; storing a copy of at least a portion of the neural network filter; applying the reconstruction data to the neural network filter to obtain second filtered data; computing an overfitting loss based on the second filtered data and the uncompressed data; computing one or more gradients of the overfitting loss with respect to one or more overfittable parameters of the neural network filter; using the one or more gradients to update the one or more overfittable parameters of the neural network filter, obtaining an adapted neural network filter; computing an overfit progress loss based on the adapted neural network filter; restoring the neural network filter after the neural network filter had been adapted with overfitting, based on the copy of at least a portion of the neural network filter; and determining a total loss based on a combination of the main loss and the overfit progress loss and updating the neural network filter.

[0231] In accordance with one aspect, an apparatus including at least one processor in communication with a memory device may be provided. The at least one processor may be configured to cause the apparatus to perform: determining one or more ground truth options for an input picture; training an auxiliary neural network for one or more tasks based at least in part on the one or more ground truth options; and adapting a neural network based at least in part on one or more outputs of the auxiliary neural network.

[0232] The apparatus may include wherein the one or more outputs of the auxiliary neural network includes one or more of a weight tensor, an optimizer index, one or more optimizer parameters, one or more learning rates for differing neural network layers, a loss function index, a loss function map, one or more loss weights, one or more indices to indicate loss weight values, one or more output modifications, one or more combination parameters, and one or more ground-truth data items. The apparatus may include wherein the one or more tasks include either classification tasks, regression tasks, or both. The apparatus may include wherein the auxiliary neural network is trained using a Model-Agnostic Meta Learningalgorithm. The apparatus may include wherein the instructions further cause the apparatus to perform: extracting, using the auxiliary network, information based on the input picture; selecting, based on the extracted information, at least one pretrained neural network from a plurality of pretrained neural networks; and adapting the neural network based at least in part on the selected pretrained neural network. The apparatus may include wherein the extracted information includes one or more of parameter indices, network layer indices, one or more identifiers of layer types, and one or more identifiers of neural network architecture types. The apparatus may include wherein the extracted information includes one or more modifications to an architecture of the neural network. The apparatus may include wherein the input picture is one or more of a reconstructed picture, one or more features of a reconstructed picture, an uncompressed picture, one or more features of an uncompressed picture, or a combination thereof. The apparatus may include wherein two or more pretrained neural networks in the plurality of pretrained neural networks are different with respect to one or more aspects. The apparatus may include wherein the one or more aspects include complexity, training loss used during training, training algorithm used for training, shape, or a combination thereof. The apparatus may include wherein the neural network comprises an adaptable neural network.

[0233] Turning to FIG. 14C, in accordance with at least one embodiment, an example method is provided including: determining 1436 one or more ground truth options for an input picture; training 1438 an auxiliary neural network for one or more tasks based at least in part on the one or more ground truth options; and adapting 1440 a neural network based at least in part on one or more outputs of the auxiliary neural network.

[0234] The method may include wherein the one or more outputs of the auxiliary neural network includes one or more of a weight tensor, an optimizer index, one or more optimizer parameters, one or more learning rates for differing neural network layers, a loss function index, a loss function map, one or more loss weights, one or more indices to indicate loss weight values, one or more output modifications, one or more combination parameters, and one or more ground-truth data items. The method may include wherein the one or more tasks include either classification tasks, regression tasks, or both. The method may include wherein the auxiliary neural network is trained using a Model-Agnostic Meta Learning algorithm. Themethod may include: extracting 1442, using the auxiliary network, information based on the input picture; selecting 1444, based on the extracted information, at least one pretrained neural network from a plurality of pretrained neural networks; and adapting 1446 the neural network based at least in part on the selected pretrained neural network. The method may include wherein the extracted information includes one or more of parameter indices, network layer indices, one or more identifiers of layer types, and one or more identifiers of neural network architecture types. The method may include wherein the extracted information includes one or more modifications to an architecture of the neural network. The method may include wherein the input picture is one or more of a reconstructed picture, one or more features of a reconstructed picture, an uncompressed picture, one or more features of an uncompressed picture, or a combination thereof. The method may include wherein two or more pretrained neural networks in the plurality of pretrained neural networks are different with respect to one or more aspects. The method may include wherein the one or more aspects include complexity, training loss used during training, training algorithm used for training, shape, or a combination thereof. The method may include wherein the neural network comprises an adaptable neural network.

[0235] In accordance with at least one embodiment, an example apparatus is provided including means for determining one or more ground truth options for an input picture; means for training an auxiliary neural network for one or more tasks based at least in part on the one or more ground truth options; and means for adapting a neural network based at least in part on one or more outputs of the auxiliary neural network.

[0236] In accordance with at least one embodiment, an example non-transitory computer readable medium comprising program instructions may be provided. The instructions, when executed by an apparatus, cause the apparatus to perform at least the following: determining one or more ground truth options for an input picture; training an auxiliary neural network for one or more tasks based at least in part on the one or more ground truth options; and adapting a neural network based at least in part on one or more outputs of the auxiliary neural network.

[0237] In accordance with one aspect, an apparatus including at least one processor in communication with a memory device may be provided. The at least one processor may beconfigured to cause the apparatus to perform: obtaining auxiliary information from an auxiliary neural network trained using one or more ground truth options of an input picture; adapting a neural network using the auxiliary information; and training the neural network based on an overfit loss determined during adapting of the neural network using the auxiliary information.

[0238] The apparatus may include wherein the auxiliary information includes a weight matrix. The apparatus may include instructions that further cause the apparatus to perform: computing, using the weight matrix, the overfit loss during adapting of the neural network using the auxiliary information. The apparatus may include instructions that further cause the apparatus to perform: concatenating the weight matrix with at least one input during adapting of the neural network using the auxiliary information.

[0239] Turning to FIG. 14D, in accordance with at least one embodiment, an example method is provided including: obtaining 1448 auxiliary information from an auxiliary neural network trained using one or more ground truth options of an input picture; adapting 1450 a neural network using the auxiliary information; and training 1452 the neural network based on an overfit loss determined during adapting of the neural network using the auxiliary information.

[0240] The method may include wherein the auxiliary information includes a weight matrix. The method may further comprise: computing 1454, using the weight matrix, the overfit loss during adapting of the neural network using the auxiliary information. The method may further comprise: concatenating 1456 the weight matrix with at least one input during adapting of the neural network using the auxiliary information.

[0241] In accordance with at least one embodiment, an example apparatus is provided including means for obtaining auxiliary information from an auxiliary neural network trained using one or more ground truth options of an input picture; means for adapting a neural network using the auxiliary information; and means for training the neural network based on an overfit loss determined during adapting of the neural network using the auxiliary information.

[0242] In accordance with at least one embodiment, an example non-transitory computer readable medium comprising program instructions may be provided. The instructions, when executed by an apparatus, cause the apparatus to perform at least the following: obtaining auxiliary information from an auxiliary neural network trained using one or more ground truth options of an input picture; adapting a neural network using the auxiliary information; and training the neural network based on an overfit loss determined during adapting of the neural network using the auxiliary information.

[0243] In accordance with one aspect, an apparatus including at least one processor in communication with a memory device may be provided. The at least one processor may be configured to cause the apparatus to perform: determining one or more ground truth options for an input picture; training an auxiliary neural network for one or more tasks based at least in part on the one or more ground truth options; and adapting a neural network based at least in part on one or more outputs of the auxiliary neural network.

[0244] The apparatus may include wherein the one or more outputs of the auxiliary neural network includes one or more of a weight tensor, an optimizer index, one or more optimizer parameters, one or more learning rates for differing neural network layers, a loss function index, a loss function map, one or more loss weights, one or more indices to indicate loss weight values, one or more output modifications, one or more combination parameters, and one or more ground-truth data items. The apparatus may include wherein the one or more tasks include classification tasks, regression tasks, or both. The apparatus may include wherein the auxiliary neural network is trained using a Model-Agnostic Meta Learning algorithm.

[0245] Turning to FIG. 14E, in accordance with at least one embodiment, an example method is provided including: determining 1458 one or more ground truth options for an input picture; training 1460 an auxiliary neural network for one or more tasks based at least in part on the one or more ground truth options; and adapting 1462 a neural network based at least in part on one or more outputs of the auxiliary neural network.

[0246] The method may include wherein the one or more outputs of the auxiliary neural network includes one or more of a weight tensor, an optimizer index, one or more optimizer parameters, one or more learning rates for differing neural network layers, a loss functionindex, a loss function map, one or more loss weights, one or more indices to indicate loss weight values, one or more output modifications, one or more combination parameters, and one or more ground-truth data items. The method may include wherein the one or more tasks include classification tasks, regression tasks, or both. The method may include wherein the auxiliary neural network is trained using a Model-Agnostic Meta Learning algorithm.

[0247] In accordance with at least one embodiment, an example apparatus is provided including means for determining one or more ground truth options for an input picture; means for training an auxiliary neural network for one or more tasks based at least in part on the one or more ground truth options; and means for adapting a neural network based at least in part on one or more outputs of the auxiliary neural network.

[0248] In accordance with at least one embodiment, an example non-transitory computer readable medium comprising program instructions may be provided. The instructions, when executed by an apparatus, cause the apparatus to perform at least the following: determining one or more ground truth options for an input picture; training an auxiliary neural network for one or more tasks based at least in part on the one or more ground truth options; and adapting a neural network based at least in part on one or more outputs of the auxiliary neural network.

[0249] In accordance with one aspect, an apparatus including at least one processor in communication with a memory device may be provided. The at least one processor may be configured to cause the apparatus to perform: extracting, using a first neural network, information based on an input picture; selecting, based on the extracted information, at least one pretrained neural network from a plurality of pretrained neural networks; and constructing a second neural network based at least in part on the selected pretrained neural network.

[0250] The apparatus may include wherein the extracted information includes one or more of parameter indices, network layer indices, one or more identifiers of layer types, and one or more identifiers of neural network architecture types. The apparatus may include wherein the extracted information includes one or more modifications to an architecture of the second neural network. The apparatus may include wherein the input picture is one or more of a reconstructed picture, one or more features of a reconstructed picture, an uncompressedpicture, one or more features of an uncompressed picture, or a combination thereof. The apparatus may include wherein two or more pretrained neural networks in the plurality of pretrained neural networks are different with respect to one or more aspects. The apparatus may include wherein the one or more aspects include complexity, training loss used during training, training algorithm used for training, shape, or a combination thereof. The apparatus may include wherein the second neural network comprises an adaptable neural network. The apparatus may include wherein the second neural network is constructed based on one or more complexity terms. The apparatus may include wherein the one or more complexity terms include one or more of output of the first neural network, number of multiply-accumulate operations in the constructed second neural network, number of layers in the constructed second neural network, estimated latency of the constructed second neural network, memory consumption of the constructed second neural network, runtime of the constructed second neural network forthe input picture, and an overfit speed. The apparatus may include wherein the one or more complexity terms is used as a scalar that scales total loss of the constructed second neural network. The apparatus may include wherein the one or more complexity terms are determined using a third neural network.

[0251] Turning to FIG. 14F, in accordance with at least one embodiment, an example method is provided including: extracting 1464, using a first neural network, information based on an input picture; selecting 1468, based on the extracted information, at least one pretrained neural network from a plurality of pretrained neural networks; and constructing 1470 a second neural network based at least in part on the selected pretrained neural network.

[0252] The method may include wherein the extracted information includes one or more of parameter indices, network layer indices, one or more identifiers of layer types, and one or more identifiers of neural network architecture types. The method may include wherein the extracted information includes one or more modifications to an architecture of the second neural network. The method may include wherein the input picture is one or more of a reconstructed picture, one or more features of a reconstructed picture, an uncompressed picture, one or more features of an uncompressed picture, or a combination thereof. The method may include wherein two or more pretrained neural networks in the plurality ofpretrained neural networks are different with respect to one or more aspects. The method may include wherein the one or more aspects include complexity, training loss used during training, training algorithm used for training, shape, or a combination thereof. The method may include wherein the second neural network comprises an adaptable neural network. The method may include wherein the second neural network is constructed based on one or more complexity terms. The method may include wherein the one or more complexity terms include one or more of output of the first neural network, number of multiply-accumulate operations in the constructed second neural network, number of layers in the constructed second neural network, estimated latency of the constructed second neural network, memory consumption of the constructed second neural network, runtime of the constructed second neural network for the input picture, and an overfit speed. The method may include wherein the one or more complexity terms is used as a scalar that scales total loss of the constructed second neural network. The method may include wherein the one or more complexity terms are determined using a third neural network.

[0253] In accordance with at least one embodiment, an example apparatus is provided including means for extracting, using a first neural network, information based on an input picture; means for selecting, based on the extracted information, at least one pretrained neural network from a plurality of pretrained neural networks; and means for constructing a second neural network based at least in part on the selected pretrained neural network.

[0254] In accordance with at least one embodiment, an example non-transitory computer readable medium comprising program instructions may be provided. The instructions, when executed by an apparatus, cause the apparatus to perform at least the following: extracting, using a first neural network, information based on an input picture; selecting, based on the extracted information, at least one pretrained neural network from a plurality of pretrained neural networks; and constructing a second neural network based at least in part on the selected pretrained neural network.

[0255] In accordance with one aspect, an apparatus including at least one processor in communication with a memory device may be provided. The at least one processor may be configured to cause the apparatus to perform: extracting, using a first neural network, information based on an input picture; generating ground-truth information of the extractedinformation using a genetic algorithm; selecting, based on the ground-truth information, at least one pretrained neural network from a plurality of pretrained neural networks; and constructing a second neural network based at least in part on the selected pretrained neural network.

[0256] The apparatus may include wherein the second neural network is an adaptable neural network. The apparatus may include wherein the second neural network is adapted based at least in part on the ground-truth information of the extracted information. The apparatus may include wherein the extracted information comprises a plurality of chromosomes each represented by a tensor and generating ground-truth information comprises: assigning an element value to each tensor of the plurality of chromosomes, wherein each element value indicates a parameter for constructing the second neural network. The apparatus may include wherein the element value is 0 or 1. The apparatus may include wherein the element value is changed during a subsequent iteration of the genetic algorithm. The apparatus may include wherein the instructions further cause the apparatus to perform: training the first neural network using the ground-truth information of the extracted information.

[0257] Turning to FIG. 14G, in accordance with at least one embodiment, an example method is provided including: extracting 1472, using a first neural network, information based on an input picture; generating 1474 ground-truth information of the extracted information using a genetic algorithm; selecting 1476, based on the ground-truth information, at least one pretrained neural network from a plurality of pretrained neural networks; and constructing 1478 a second neural network based at least in part on the selected pretrained neural network.

[0258] The method may include wherein the second neural network is an adaptable neural network. The method may include wherein the second neural network is adapted based at least in part on the ground-truth information of the extracted information. The method may include wherein the extracted information comprises a plurality of chromosomes each represented by a tensor and generating ground-truth information comprises: assigning 1478 an element value to each tensor of the plurality of chromosomes, wherein each element value indicates a parameter for constructing the second neural network. The method may includewherein the element value is 0 or 1. The method may include wherein the element value is changed during a subsequent iteration of the genetic algorithm. The method may further comprise: training 1480 the first neural network using the ground-truth information of the extracted information.

[0259] In accordance with at least one embodiment, an example apparatus is provided including means for extracting, using a first neural network, information based on an input picture; means for generating ground-truth information of the extracted information using a genetic algorithm; means for selecting, based on the ground-truth information, at least one pretrained neural network from a plurality of pretrained neural networks; and means for constructing a second neural network based at least in part on the selected pretrained neural network.

[0260] In accordance with at least one embodiment, an example non-transitory computer readable medium comprising program instructions may be provided. The instructions, when executed by an apparatus, cause the apparatus to perform at least the following: extracting, using a first neural network, information based on an input picture; generating ground-truth information of the extracted information using a genetic algorithm; selecting, based on the ground-truth information, at least one pretrained neural network from a plurality of pretrained neural networks; and constructing a second neural network based at least in part on the selected pretrained neural network.

[0261] The term "non-transitory," as used herein, is a limitation of the medium itself (i.e., tangible, not a signal) as opposed to a limitation on data storage persistency (e.g., RAM vs. ROM).

[0262] As used in this application, the term "circuitry" may refer to one or more or all of the following:(a) hardware-only circuit implementations (such as implementations in only analog and / or digital circuitry) and(b) combinations of hardware circuits and software, such as (as applicable):(i) a combination of analog and / or digital hardware circuit(s) with software / firmware and(ii) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions) and(iii) hardware circuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation."

[0263] This definition of circuitry applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.

[0264] It should be understood that the foregoing description is only illustrative. Various alternativesand modificationscan be devised by those skilled intheart. For example, features recited in the various dependent claims could be combined with each other in any suitable combination(s). In addition, features from different embodiments described above could be selectively combined into a new embodiment. Accordingly, the description is intended to embrace all such alternatives, modifications and variances which fall within the scope of the appended claims.

Claims

CLAIMS1. An apparatus comprising:at least one processor; andat least one memory storing instructions that, when executed with the at least one processor, cause the apparatus to perform:grouping received input data into one or more clusters;for at least one of the one or more clusters, selecting one or more aspects of an overfitting process, where the one or more aspects are associated to the at least one of the one or more clusters; andfor at least one of the one or more clusters, adapting one or more parameters of a neural network using the selected one or more aspects of the overfitting process associated to the at least one of the one or more clusters and using data in the at least one of the one or more clusters.

2. The apparatus of claim 1, wherein the one or more aspects include one or more of optimizer, optimizer's parameters, loss function, loss weights, and layer-wise learning rates.

3. The apparatus of claim 1 or 2, wherein the input data include training images and the training images are grouped into clusters based on a similarity metric.

4. The apparatus of claim 1 or 2, wherein one or more loss functions are used for adapting one or more parameters of the neural network for each cluster.

5. The apparatus of any of claims 1-4, wherein the instructions further cause the apparatus to perform:evaluating performance of the adapting one or more parameters of the neural network;selecting a loss function associated with a set of the one or more parameters with higher performance; andusing the selected loss function at a subsequent inference time of the neural network for regions of input data that are similar to regions of corresponding one or more clusters.

6. A method comprising:grouping received input data into one or more clusters;for at least one of the one or more clusters, selecting one or more aspects of an overfitting process, where the one or more aspects are associated to the at least one of the one or more clusters; andfor at least one of the one or more clusters, adapting one or more parameters of a neural network using the selected one or more aspects of the overfitting process associated to the at least one of the one or more clusters and using data in the at least one of the one or more clusters.

7. An apparatus comprising:at least one processor; andat least one memory storing instructions that, when executed with the at least one processor, cause the apparatus to perform:receiving reconstruction data and uncompressed data;applying the reconstruction data to a neural network filter to obtain first filtered data;computing a main loss based on the first filtered data and the uncompressed data;storing a copy of at least a portion of the neural network filter;applying the reconstruction data to the neural network filter to obtain second filtered data;computing an overfitting loss based on the second filtered data and the uncompressed data;computing one or more gradients of the overfitting loss with respect to one or more overfittable parameters of the neural network filter;using the one or more gradients to update the one or more overfittable parameters of the neural network filter, obtaining an adapted neural network filter;computing an overfit progress loss based on the adapted neural network filter;restoring the neural network filter after the neural network filter had been adapted with overfitting, based on the copy of at least a portion of the neural network filter; anddetermining a total loss based on a combination of the main loss and the overfit progress loss and updating the neural network filter.

8. A method comprising:receiving reconstruction data and uncompressed data;applying the reconstruction data to a neural network filter to obtain first filtered data;computing a main loss based on the first filtered data and the uncompressed data;storing a copy of at least a portion of the neural network filter;applying the reconstruction data to the neural network filter to obtain second filtered data;computing an overfitting loss based on the second filtered data and the uncompressed data;computing one or more gradients of the overfitting loss with respect to one or more overfittable parameters of the neural network filter;using the one or more gradients to update the one or more overfittable parameters of the neural network filter, obtaining an adapted neural network filter;computing an overfit progress loss based on the adapted neural network filter;restoring the neural network filter after the neural network filter had been adapted with overfitting, based on the copy of at least a portion of the neural network filter; anddetermining a total loss based on a combination of the main loss and the overfit progress loss and updating the neural network filter.

9. An apparatus comprising:at least one processor; andat least one memory storing instructions that, when executed with the at least one processor, cause the apparatus to perform:determining one or more ground truth options for an input picture;training an auxiliary neural network for one or more tasks based at least in part on the one or more ground truth options; andoverfitting a neural network based at least in part on one or more outputs of the auxiliary neural network.

10. The apparatus of claim 9, wherein the one or more outputs of the auxiliary neural network includes one or more of a weight tensor, an optimizer index, one or more optimizer parameters, one or more learning rates for differing neural network layers, a loss function index, a loss function map, one or more loss weights, one or more indices to indicate loss weight values, one or more output modifications, one or more combination parameters, and one or more ground-truth data items.

11. The apparatus of claim 9 or 10, wherein the one or more tasks include either classification tasks, regression tasks, or both.

12. The apparatus of any of claims 9-11, wherein the auxiliary neural network is trained using a Model-Agnostic Meta Learning algorithm.

13. The apparatus of any of claims 9-12, wherein the instructions further cause the apparatus to perform:extracting, using the auxiliary network, information based on the input picture;selecting, based on the extracted information, at least one pretrained neural network from a plurality of pretrained neural networks; andadapting the neural network based at least in part on the selected pretrained neural network.

14. The apparatus of claim 13, wherein the extracted information includes one or more of parameter indices, network layer indices, one or more identifiers of layer types, and one or more identifiers of neural network architecture types.

15. The apparatus of claim 13 or 14, wherein the extracted information includes one or more modifications to an architecture of the neural network.

16. The apparatus of any of claims 13-15, wherein the input picture is one or more of a reconstructed picture, one or more features of a reconstructed picture, an uncompressed picture, one or more features of an uncompressed picture, or a combination thereof.

17. A method, comprising:determining one or more ground truth options for an input picture;training an auxiliary neural network for one or more tasks based at least in part on the one or more ground truth options; andadapting a neural network based at least in part on one or more outputs of the auxiliary neural network.

18. An apparatus comprising:at least one processor; andat least one memory storing instructions that, when executed with the at least one processor, cause the apparatus to perform:obtaining auxiliary information from an auxiliary neural network trained using one or more ground truth options of an input picture;adapting a neural network using the auxiliary information; andtraining the neural network based on an overfit loss determined during adapting of the neural network using the auxiliary information.

19. The apparatus of claim 18, wherein the auxiliary information includes a weight matrix.

20. The apparatus of claim 19, the instructions further cause the apparatus to perform:computing, using the weight matrix, the overfit loss during adapting of the neural network using the auxiliary information.

21. The apparatus of claim 19, the instructions further cause the apparatus to perform:concatenating the weight matrix with at least one input during adapting of the neural network using the auxiliary information.

22. A method comprising:obtaining auxiliary information from an auxiliary neural network trained using one or more ground truth options of an input picture;adapting a neural network using the auxiliary information; andtraining the neural network based on an overfit loss determined during adapting of the neural network using the auxiliary information.

23. An apparatus comprising:at least one processor; andat least one memory storing instructions that, when executed with the at least one processor, cause the apparatus to perform:extracting, using a first neural network, information based on an input picture;selecting, based on the extracted information, at least one pretrained neural network from a plurality of pretrained neural networks; andconstructing a second neural network based at least in part on the selected pretrained neural network.

24. The apparatus of claim 23, wherein the extracted information includes one or more of parameter indices, network layer indices, one or more identifiers of layer types, and one or more identifiers of neural network architecture types.

25. The apparatus of claim 23 or 24, wherein the extracted information includes one or more modifications to an architecture of the second neural network.

26. The apparatus of any of claims 23-25, wherein the input picture is one or more of a reconstructed picture, one or more features of a reconstructed picture, an uncompressed picture, one or more features of an uncompressed picture, or a combination thereof.

27. A method, comprising:extracting, using a first neural network, information based on an input picture;selecting, based on the extracted information, at least one pretrained neural network from a plurality of pretrained neural networks; andconstructing a second neural network based at least in part on the selected pretrained neural network.

28. An apparatus comprising:at least one processor; andat least one memory storing instructions that, when executed with the at least one processor, cause the apparatus to perform:extracting, using a first neural network, information based on an input picture;generating ground-truth information of the extracted information using a genetic algorithm;selecting, based on the ground-truth information, at least one pretrained neural network from a plurality of pretrained neural networks; andconstructing a second neural network based at least in part on the selected pretrained neural network.

29. The apparatus of claim 28, wherein the second neural network is an overfittable neural network.

30. The apparatus of claim 29, wherein the second neural network is overfit based at least in part on the ground-truth information of the extracted information.

31. The apparatus of claim 28, wherein the instructions further cause the apparatus to perform:training the first neural network using the ground-truth information of the extracted information.

32. A method, comprising:extracting, using a first neural network, information based on an input picture;generating ground-truth information of the extracted information using a genetic algorithm;selecting, based on the ground-truth information, at least one pretrained neural network from a plurality of pretrained neural networks; andconstructing a second neural network based at least in part on the selected pretrained neural network.

Images