Multi-picture post-filtering of a video

Abstract

An apparatus for reconstructing a video from a data stream reconstructs the picture of the video and filters the reconstructed picture using a neural network. Filter input data is obtained based on the reconstructed picture and based on one or more auxiliary sample arrays. The picture and the one or more auxiliary sample arrays belong to the same time frame of the video.

Description

[0001] Multi-picture post -filtering of a video

[0002] Description

[0003] Embodiments of the invention relate to an apparatus, e.g., a video decoder, for reconstructing a video from a data stream, an apparatus, e.g., a video encoder, for encoding a video into a data stream, a method for reconstructing a video from a data stream, a method for encoding a video into a data stream, and / or a video data stream.

[0004] Neural Network Post Filters (NN PF) can be used for enhancing the quality or resolution of decoded pictures of a video. They can also enhance the user experience by taking decoded pictures as input and generating an output that is richer, e.g., an immersive experience generating 3D scenes from a 2D video. Currently, it is possible to include an NNPF into SEI messages. The current design includes two SEI messages: namely the NNPFC SEI message, which includes parameters that describe the neural network and parameters that describe the input to the network, and NNPFA SEI messages that activate a NNPF (as described by the NNPFC SEI message) for a particular picture or set of pictures. NNPF can also take several pictures of different time instances as input, as for instance, when picture rate upsampling is desired (i.e., increasing the framerate of the video by inserting interpolated pictures).

[0005] However, it would be desirable to provide a concept for a post-filtering of a reconstructed picture, which provides for an improved trade-off between a high quality of the picture generated by the post-filtering, a low computational load, and a low signaling overhead.

[0006] This objective is achieved by the subject-matter of the independent claims.

[0007] Embodiments of the present invention rely on the idea of using, in a neural network filtering of a picture reconstructed from a video data stream, one or more auxiliary sample arrays of the video data stream, which belong to the same time frame as the reconstructed picture to be filtered. For example, the one or more auxiliary sample arrays may include a picture, which represents the time frame of the video at a different resolution compared to the reconstructed picture, or from a different view position or view direction, or auxiliary data such as a depth map, or a mask, which assigns samples of a picture to different subpictures, e.g., pictures of different objects. In other words, the video data stream may comprise one or more auxiliary sample array, e.g., out of the just mentioned examples. Embodiments of

[0008] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmthe invention exploit the information carried by these auxiliary sample arrays for the filtering process in order to increase the quality of the filter output.

[0009] In other words, embodiments of the invention rely on the finding that taking multiple sample arrays of the same time frame, e.g., pictures from several layers, can be beneficial. For example, different use-cases can be fulfilled that could benefit from using pictures from more than one layer as input to the neural network. For instance, a layer could include an object mask indicating that different samples of a related picture correspond to different objects. Such an information could improve the performance of NNPF by for instance selectively filtering specific areas of an image or employing some enhancement only to some of the objects in the image based on such object masks. One could also think of a layer including depth in addition to the texture, which could be very useful in enhancing spatial understanding, guiding the filtering process for 3D content or even for an NNPF that would be used for view synthesis based on both a texture picture and depth picture.

[0010] For example, the multiple sample arrays, e.g., the one or more sample arrays representing the picture to be reconstructed and the one or more auxiliary sample arrays may be assigned to different layers of the video data stream, or may be referred to as belonging to different layers of the video data stream. For example, each of the layers comprises, for each time frame of the video, or for each of a subset of time frames of the video, one or more sample arrays. In other words, the video data stream may be a multi-layer video data stream.

[0011] For example, in contrast to the current design of NNPF, which cannot take several pictures from different layers, embodiments of the invention may use pictures of different layers, as input for a post-filtering.

[0012] Embodiments of the invention provide an apparatus for reconstructing a video from a data stream. The apparatus is configured for reconstructing a picture of the video. The apparatus is further configured for filtering the reconstructed picture using a neural network by obtaining (e.g., deriving, forming) filter input data (e.g., an input tensor) based on the reconstructed picture and based on one or more auxiliary sample arrays (e.g., two-dimensional sample arrays, e.g., representing spatially sampled information), and subjecting (or inputting) the filter input data to the neural network. The picture and the one or more auxiliary sample arrays belong to the same time frame (e.g., access unit) of the video.

[0013] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmFor example, the picture comprises or consists of one or more sample arrays, e.g., the picture comprises or consists of a luma sample array and two chroma sample arrays. E.g., the picture belongs to one of a sequence of time frames of the video; E.g., the data stream has encoded thereinto, for each of a sequence of time frames, one or more pictures, and optionally, in addition to the one or more pictures, one or more sample arrays carrying spatially sampled information.

[0014] For example, the filter input data comprises the reconstructed picture or a portion thereof and further comprises the one or more auxiliary sample arrays or respective portions of the one or more auxiliary sample arrays. For example, deriving or obtaining the filter input data may optionally be implemented by the apparatus by including, into the filter input data, the reconstructed picture, or a portion thereof (e.g. a patch thereof), and the one or more auxiliary sample arrays, or respective portions thereof (e.g., respective patches thereof).

[0015] Further embodiments of the invention provide an apparatus for encoding a video into a data stream. The apparatus is configured for encoding a picture of the video into the data stream. The apparatus is further configured for inserting, into the data stream, an indication (e.g., a syntax structure) of one or more auxiliary sample arrays, the one or more auxiliary sample arrays being to be used for obtaining filter input data for filtering a reconstructed version of the picture, which filter input data is to be obtained based on the reconstructed picture and based on the one or more auxiliary sample arrays, and which filter input data is to be subjected to the neural network. The picture and the one or more auxiliary sample arrays belong to the same time frame (e.g., access unit) of the video.

[0016] Further embodiments of the invention provide a method for reconstructing a video from a data stream, the method comprising: reconstructing a picture of the video; and filtering the reconstructed picture using a neural network by obtaining filter input data based on the reconstructed picture and based on one or more auxiliary sample arrays, and subjecting the filter input data to the neural network. The picture and the one or more auxiliary sample arrays belong to the same time frame of the video.

[0017] Further embodiments of the invention provide a method for encoding a video into a data stream, the method comprising: encoding a picture of the video into the data stream; and inserting, into the data stream, an indication of one or more auxiliary sample arrays, the one or more auxiliary sample arrays being to be used for obtaining filter input data for filtering a

[0018] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmreconstructed version of the picture, which filter input data is to be obtained based on the reconstructed picture and based on the one or more auxiliary sample arrays, and which filter input data is to be subjected to the neural network. The picture and the one or more auxiliary sample arrays belong to the same time frame of the video.

[0019] Advantageous implementations are defined by the subject-matter of the dependent claims.

[0020] Embodiments of the present disclosure are described in more detail below with respect to the figures, among which:

[0021] Fig. 1 illustrates an apparatus for decoding a video according to an embodiment,

[0022] Fig. 2 illustrates an embodiment of the filter input determinator,

[0023] Fig. 3 illustrates a derivation of luma channels according to an embodiment,

[0024] Fig. 4 illustrates an embodiment for encoding a video according to an embodiment,

[0025] Fig. 5 illustrates an encoder according to an embodiment;

[0026] Fig. 6 illustrates a decoder according to an embodiment;

[0027] Fig. 7 illustrates sub-divisions of a picture according to an embodiment.

[0028] Embodiments of the present invention are now described in more detail with reference to the accompanying drawings, in which the same or similar elements or elements that have the same or similar functionality have the same reference signs assigned or are identified with the same name. In the following description, a plurality of details is set forth to provide a thorough explanation of embodiments of the disclosure. However, it will be apparent to one skilled in the art that other embodiments may be implemented without these specific details. In addition, features of the different embodiments described herein may be combined with each other, unless specifically noted otherwise.

[0029] Further, it is noted, that details and features described with respect to a decoder, may equivalently apply to corresponding features of a corresponding encoder and vice versa.

[0030] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmEmbodiments of the invention described in the following, which may optionally be implemented in, or combined with, the framework described in the section “Video coding schemes”, in particular, with respect to Fig. 5 to Fig. 7.

[0031] Fig. 1 illustrates an apparatus 20 for reconstructing a video from a data stream 14 according to an embodiment. Apparatus 20 may be referred to as decoder 20. For example, apparatus 20 may reconstruct the video by decoding an encoded representation of the video from the data stream 14. In other words, data stream 14 may carry an encoded representation of the video. Apparatus 20 comprises a reconstructor 21, which reconstructs a picture 12’ of the video 9. The apostrophe is used to indicate that picture 12’ as reconstructed by apparatus 20 may deviate from a picture 12 encoded by an encoder into the data stream 14, e.g., in terms of quantization loss, e.g., as it is described with respect to encoder 10 below, and in particular, with respect to the embodiments described in the section “video coding schemes”. Apparatus 20 further comprises a filtering module 61 , which may also be referred to as post filter. Filtering module 61 filters the reconstructed picture 12’ using a neural network, e.g., a neural network post filter. Filter input data 65, which is subjected to the filtering 61, i.e. , which provides the input data for the filtering module 61, is provided by filter input determinator 63 which provides the filter input data 63 based on the reconstructed picture 12’ and based on one or more auxiliary sample arrays 15. Filtering module 61 subjects the filter input data 65 to the neural network.

[0032] For example, the data stream 14 may have data for a plurality of time frames arranged along a temporal order 11 of the video encoded thereinto. For example, the data of one time frame may include a plurality of sample arrays. For example, a sample array may refer to an array carrying spatially sampled information. The data of one time frame may include one or more pictures, each of which may be represented by one or more of the sample arrays, e.g., in case of a color picture, a respective sample array for each of a plurality of components of the picture, e.g., a luma component and two chroma components. For example, the data of one time frame may include multiple representations of the same picture, e.g., at different resolutions, and / or may include pictures associated with different views. Additionally, or alternatively, the data of one time frame may include an auxiliary map and / or a mask, which indicates, for each sample position of a picture of the time frame, a subpicture to which the respective sample position is assigned. For example, a mask may provide a means for assigning a portion of a picture to different subpictures, each of which may represent an object, for example.

[0033] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmIn other words, a time frame, e.g., time frame 11* indicated in Fig. 1 , of the video encoded into data stream 14 may comprise a picture 12’ to be reconstructed by decoder 20, and in addition to a representation of picture 12’, one of more auxiliary sample arrays 15. The auxiliary sample arrays 15 may represent a further picture, such as of a further view, or a depth map or a mask, for example.

[0034] For example, all data of video data stream 14 belonging to one time frame may be referred to as one access unit of the data stream 14.

[0035] For example, the video data stream 14 may be a multilayer video data stream comprising a plurality of layers. For example, the picture 12’ is encoded into a first layer of data stream 14 and the one or more auxiliary sample arrays 15 are encoded on one or more further layers of the data stream 14. For example, each of the layers of the multiple layers of the data stream 14 may comprise one or more sample arrays for each of the time frames of the video, or for each of a subset of the time frames of the video.

[0036] Alternatively, in examples in which the data stream 14 does not have a layered structure, the auxiliary sample arrays 15 may, for example, be marked as auxiliary pictures or auxiliary sample arrays associated with the respective time frame.

[0037] It is noted that even further embodiments of the invention include implementations in which the auxiliary sample array 15 are not included in data stream 14 itself but may be provided by separate means to apparatus 20, e.g., in providing the auxiliary sample arrays 15 along with an association to the time frames of the data stream 14.

[0038] According to an embodiment, the filter input data may have the form of a tensor. For example, the tensor has the same dimension as in input, e.g., an input layer, of the neural network.

[0039] Accordingly, the filter input data 65 may be adapted to the neural network used by the filtering module 61. For example, depending on a mode used for the filtering by filtering module 61, e.g., one of the modes as will be described below, the filtering module 61 may select a respective neural network, and filter input determinator 63 may determine the filter input data 65 in accordance with the dimension of the selected neural network.

[0040] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmFor example, apparatus 20 may derive an indication of the neural network used by filtering module 61 for filtering the filter input data 65 from data stream 14. For example, the indication may be signaled in form of a selector, which selects on out of a set of neural networks.

[0041] For example, neural network parameters, such as weights and / or bias and / or offsets, may be derived by apparatus 20 from data stream 14.

[0042] For example, filter input determinator 63 includes, in the filter input data 65, the reconstructed picture 12’, or a portion, e.g., a patch, of the reconstructed picture 12’. Additionally, filter input determinator 63 may include in the filter input data 65 the one or more auxiliary sample arrays 15, or respective portions, e.g., patches, thereof, e.g., portions, which are co-located to the portion of the reconstructed picture 12’ included in the input filter data 65 within their respective sample arrays. As already mentioned, picture 12’ may comprise, or consist of, a plurality of sample arrays for respective components of the picture 12’.

[0043] For example, filtering module 61 may provide one or more filtered pictures 12” generated by subjecting the filter input data 65 to the neural network. The one or more filtered pictures 12” may represent the entire content of the picture 12’, or only a portion thereof, e.g., in cases in which the reconstructed picture 12’ is corrupt such that only a patch of the reconstructed picture 12’ is used as input for the filtering module 61.

[0044] For example, the one or more filtered pictures 12” may have a different, e.g., a higher, resolution compared to the reconstructed picture 12’, e.g., a spatial resolution. In other words, the filtering of filtering module 61 may include an upsampling.

[0045] In examples, filtering module 61 may perform a frame rate upsampling, such that a frame rate of the video is increased by determining additional pictures for time frames, which are located between time frames included in the video data stream 14. In these cases, for example, the filter input data may include a further picture of the video, which belongs to a further time frame preceding or succeeding the time frame, to which picture 12’ belongs, in the temporal order 11 of time frames of the video.

[0046] According to an embodiment, the video may be encoded into the data stream 14 in units of sequences, e.g., coded video sequences, CVS, or coded layer video sequences, CLVS.

[0047] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmFor example, each of the sequences may be decodable independently, i.e., independent of the other sequences of data stream 14. In case of a multilayered data stream, the segmentation of the data stream into the sequences may be identical for each of the layers, or for a subset of the layers.

[0048] According to an embodiment, coded video sequences of the first layer, i.e., the layer to which the picture 12’ belongs, and of the one or more further layers, i.e., the layers to which the one or more auxiliary pictures 15 belong, start at the same time frame and / or end at the same time frame.

[0049] In other words, a coded video sequence, to which the picture 12’ belongs, and one or more coded video sequences, to which the one or more auxiliary pictures 15 belong, may start at the same time frame and / or end at the same time frame.

[0050] However, this restriction is optional. According to an embodiment, apparatus 20 infers, in dependence on an indication in the data stream 14, that the coded video sequence of the first layer and coded video sequences of the one or more further layers start at the same time frame. For example, apparatus 20 derives from data stream 14 an indication, e.g., syntax element nnpfa_no_prev_clvs_flag, which indicates where the pictures of a coded video sequence preceding the coded video sequence to which the picture 12’ belongs, can be used for deriving the filter input data 65. If the indication indicates that pictures of the preceding coded video sequence cannot be used for deriving the filter input data 65, apparatus 20 may infer that the coded video sequence of the first layer and the coded video sequences of the one or more further layers start at the same time frame.

[0051] In a similar manner, regarding the end point of the coded video sequences, apparatus 20 may infer, in dependence on an indication in data stream 14, that the coded video sequence of the first layer and coded video sequences of the one or more further layers and at the same time frame. For example, apparatus 20 may derive, from data stream 14, an indication, e.g., syntax element nnpfa_no_foll_clvs_flag, which indicates where the pictures of a coded video sequence succeeding the coded video sequence, to which the picture 12’ belongs, can be used for deriving the filter input data 65. If the indication indicates that pictures of the succeeding coded video sequence cannot be used for deriving the filter input data 65, apparatus 20 may infer that the coded video sequences of the first layer and coded video sequences of the one or more further layers end at the same time frame.

[0052] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmAs already mentioned, an alternative to the picture 12’ and the one or more auxiliary pictures 15 of one time frame being encoded in different layers of data stream 14, other means of encoding multiple pictures or sample arrays of one time frame into data stream 14 may be used. For example, according to an embodiment, data stream 14 has encoded therein to a sequence of primary pictures including the picture 12’, the sequence of primary pictures representing the video, e.g., the sequence of primary pictures providing a representation of the video, e.g., from which the video can be reconstructed, e.g., without further pictures or sample arrays included in data stream 14. According to this embodiment, data stream 14 further has encoded into the one or more auxiliary sample arrays, e.g., as non-primary sample arrays. For example, the primary pictures or primary sample arrays may be encoded in first payload packets and the one or more auxiliary sample arrays may be encoded in second payload packets, which are interspersed between the first payload packets.

[0053] According to an embodiment, the sequence of primary pictures is encoded in payload packets of a first type, e.g., video coded layer (VCL), network abstraction layer (NAL) units, and data stream 14 has encoded thereinto the one or more auxiliary sample arrays 15 in payload packets, e.g., NAL units, of a second type, e.g., which are interspersed between the payload packets of the first type.

[0054] According to an embodiment, the filter input data 65 comprises pictures and / or auxiliary sample arrays of multiple time frames of the video.

[0055] In other words, according to an embodiment, the picture 12’ belongs to a first time frame 11* of the video. According to this embodiment, a further picture of the video, which belongs to a second time frame of the video, is reconstructed by apparatus 20 from the data stream 14. According to this embodiment, filter input determinator 63 obtains the filter input data 65 further based on the reconstructed further picture and based on one or more further auxiliary sample arrays, which belong to the second time frame. For example, the further picture may belong to the same layer as the picture 12’, and the one or more further auxiliary sample arrays may belong to the one or more further layers, to which the auxiliary sample arrays 15 belong.

[0056] According to an embodiment, apparatus 20 derives, from data stream 14, an indication of the neural network to be used by filtering unit 61 for filtering the filter input data 65. In other words, the neural network to be used for filtering the picture 12’ may be signaled in data stream 14. For example, data stream 14 may include, for each picture of the video, an

[0057] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmindication, which indicates if neural network post-filtering is to be applied to the picture, and if the neural network post-filtering is to be applied to the picture, the neural network to be used for post-filtering the picture.

[0058] According to an embodiment, apparatus 20 derives, from data stream 14, a syntax element, e.g., nnpfc_auxiliary_inp_idc, which differentiates between a plurality of modes of obtaining the filter input data 65. The plurality of modes includes a mode according to which the filter input data for the picture is to be obtained based on one or more auxiliary sample arrays 15 belonging to the same time frame as the picture. In other words, picture 12’ may be a picture, for which the mode, according to which auxiliary sample arrays of the same time frame, are to be used for filtering the picture.

[0059] In other words, according to an embodiment, apparatus 20 derives for a currently reconstructed picture of the video, which is to be subjected to neural network post-filtering, a syntax element from the data stream 14, which syntax element differentiates between the plurality of modes. If the syntax element indicates that the filter input data for the picture is to be obtained based on one or more auxiliary sample arrays belonging to the same time frame as the picture, filtered input determinator 63 obtains the filter input data 65 based on one or more auxiliary sample arrays, which, e.g., may be determined according to the selected mode.

[0060] For example, the plurality of modes may additionally include one o both of:

[0061] a mode according to which no auxiliary information is used for obtaining the filter input data 65, e.g., the filter input data is obtained based on the reconstructed picture only, or based on a plurality of pictures belonging to different time frames of the video only,

[0062] a mode according to which a quantization parameter for reconstructing the picture is to be used for obtaining the filter input data, and a mode according to which a text string is to be used for obtaining the filter input data.

[0063] According to an embodiment, filter input determinator 63 obtains the filter input data 65 in form of a tensor, which comprises, or consists of, a plurality of channels. Each of the channels provides an input channel for the neural network. For example, the channels represent one dimension of the tensor. For example, each of the channels may be one or multi-dimensional.

[0064] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmIn other words, a dimension of the tensor may be the count of the channels x the dimension of the channels.

[0065] For example, deriving or obtaining a channel of the filter input data may be performed by including, into the channel, the reconstructed picture, or a portion thereof, or one of the one or more auxiliary sample arrays, or a respective portion thereof. Optionally, this may apply to each of the channels.

[0066] In the following, embodiments for determining the one or more auxiliary sample arrays are described.

[0067] According to an embodiment, apparatus 20 derives, from the data stream, e.g., from or more syntax elements of the data stream, an indication of a set of auxiliary pictures, each of the auxiliary pictures being represented by one or more of the one or more auxiliary sample arrays. For example, the indication indicates a count or a quantity of the auxiliary pictures.

[0068] In other words, for example, for each of the set of auxiliary pictures indicated by the indication, the one or more auxiliary sample arrays, which represent the respective auxiliary picture, may be used for determining the filter input data 65, e.g., by including at least a portion of the respective auxiliary sample array into the filter input data 65.

[0069] According to an embodiment, the set of auxiliary pictures is indicated by indicating a set of layers out of a plurality of layers of the data stream 14. Each of the layers of the set of layers carries one of the auxiliary pictures.

[0070] In other words, for example, the set of auxiliary pictures may be indicated by means of indicating layer identifiers associated with layers, to which the respective pictures belong.

[0071] In other words, according to an embodiment, apparatus 20 derives, from the data stream, an indication of a set of layers out of a plurality of layers of the data stream, each of the layers of the set of layers carrying an auxiliary picture, which is represented by one or more of the one or more auxiliary sample arrays. For example, the auxiliary pictures carried by the set of layers may be referred to as set of auxiliary pictures. The set of layers may comprise or consist of one or more layers. For example, the set of layers may correspond

[0072] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmto the above mentioned one or more further layers carrying the one or more auxiliary sample arrays.

[0073] For example, the indication of the set of auxiliary pictures, or the indication of the set of layers, may correspond to the syntax element nnpfc_num_aux_layers_minus1 described below.

[0074] According to an embodiment, apparatus 20 derives, from data stream 14, an indication of the count of auxiliary pictures of the set of auxiliary pictures.

[0075] For example, in case that the set of auxiliary pictures is indicated by indicating a set of layers, the indication of the count of auxiliary pictures may indicate a count of layers of the set of layers. In other words, according to an embodiment, apparatus 20 derives, from the data stream 14, an indication indicating the count, or quantity, of layers of the set of layers, e.g., the syntax element nnpfc_num_aux_layers_minus1 described below. According to this embodiment, apparatus 20 derives, from data stream 14, for each of the set of layers, an identification of the respective layer within the plurality of layers of the data stream 14.

[0076] For example, each layer of the plurality of layers of a multilayer data stream may be associated with a respective layer identifier, which identifies the layer among the plurality of layers or by indicating a layer identifier, which is indicated in the data stream for each of the plurality of layers. Alternatively, the layer identifier may be mapped by the apparatus 20 to one of the plurality of layers, which may be uniquely identified by means of one or more properties signaled in the data stream for each of the plurality of layers, as it will be explained in more detail below.

[0077] For example, the identification of the respective layer may be signaled using the syntax element nnpfc_layer_target_id described below.

[0078] For example, in case of a layered video data stream, in which each layer is identified using a unique layer identifier, apparatus 20 may derive, from data stream 14, a layer identification syntax element, which signals the identification of the respective layer within the plurality of layers. According to this embodiment, each of the plurality of layers is associated with a respective layer identifier, and the layer identification syntax element indicates the layer identifier of the respective layer, e.g., by signaling a value corresponding to a value of the layer identifier of the respective layer.

[0079] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmIn the following, an exemplary embodiment is described. The following embodiment is presented in terms of an amendment of the above mentioned existing nnpf process. For example, this process uses two types of SEI messages, the nnpf CSEI message, which defines a neural network, and an nnpfa SEI message, which signals, for pictures of the video, an activation of the nnpf process and for the pictures, for which the nnpf process is to be used, respective neural networks defined by the nnpf CSEI message.

[0080] As a first embodiment information is provided into the bitstream that indicates which pictures or additional layers are used as input for an NNPF process. In the following, an instantiation thereof is shown by including such data into the NNPFC SEI message.

[0081] & >

[0082] & >

[0083] <

[0084]

[0085] nnpfc_num_aux_layers_minusl plus 1 indicates the number of auxiliary layers in addition to the current layer used as input for the NNPF.

[0086] nnpfc_layer_target_id[ i ] specifies the layer identifier of the i-th layer to be used as input for the NNPF. Sample arrays of auxiliary input pictures are used to form an input tensor to the NNPF as additional channels as specified in Equation 96.

[0087] Note that the above embodiment describes the usage of further pictures from other layers but could also apply to pictures that are non-primary coded pictures (somehow marked as

[0088] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmauxiliary pictures) or even pictures that are externally provided. It could be for instance that a particular id value or even a further nnpfc_auxiliary_inp_idc is used when the auxiliary pictures are not necessarily associated with a particular layer or syntax element but with a type of picture, e.g. where an AU has a primary picture and a secondary picture, with the secondary picture including such auxiliary information.

[0089] As shown above, this can be done by indicating a particular type of auxiliary data (e.g., nnpfc_auxiliary_inp_idc & 4 being larger than 0 or just the particular value 4 -nnpfc_auxiliary_inp_idc == 4 - could be used or some other particular value). If so, additional information could be provided such as how many layers are used (pictures form those layers) as well an identifier to indicate which are the used layers.

[0090] As already mentioned, the data stream is not necessarily a layered data stream in terms that the data stream comprises a plurality of layers having a specific layer identifier associated therewith, but in further embodiments, a layer may be defined by a combination of layer-specific properties. The layer-specific properties may include a layer id, a quality ID, which indicates a quality of the picture, a dependency ID, which indicates dependencies of the picture, e.g., with respect to other pictures, a view idx, which indicates a view to which the picture belongs, a depth flag, which indicates, whether the picture is a depth map, or a view id. For example, depending on the content of the data stream, a certain combination of syntax elements, which signal one or more of these properties for a layer, may provide for a unique identification of the respective layer.

[0091] In other words, according to an embodiment, apparatus 20 derives, from the data stream 14, for each of the plurality of layers, a set of layer-specific property indicators. Each of the layer-specific property indicators indicates a property of the respective layer by signaling a value out of the domain of the respective layer-specific property indicator, e.g., which value represents the property of the respective layer. According to this embodiment, apparatus 20 derives, from data stream 14, a layer identification syntax element, which signals the identification of the respective layer within the plurality of layers. According to this embodiment, the layer identification syntax element indicates a value, or comprises, or consists of a concatenation of values, for each layer-specific property indicator of the set of layer-specific property indicators. According to this embodiment, apparatus 20 identifies the respective layer based on the combination of values of the layer-specific property indicators.

[0092] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmAccording to an embodiment, in the data stream 14, each of the layer-specific property indicators is represented by a respective predefined count, or quantity, of bits. For example, the predefined count of bits covers the domain of the respective layer-specific property indicator.

[0093] According to an embodiment, the sum of the predefined counts of bits of the layer-specific property indicators of the set of layer-specific property indicators equals the count of bits with which the layer identification syntax element is signaled.

[0094] In other words, for example, the layer identification syntax element may be a concatenation of binary representations of the layer-specific property indicators, in which, for each of the layer-specific property indicators, the count of bits for representing the binary representation corresponds to the respective predefined count.

[0095] According to an embodiment, the count of bits with which the layer identification syntax element is signaled in the data stream 14 equals the sum of the counts of bits, which are required to represent the domains of the layer-specific property indicators of the set of layerspecific property indicators, e.g., when concatenating bitstreams for indicating the layerspecific property indicators. According to this embodiment, apparatus 20 infers that the counts of bits, which are required to represent the domains of the layer-specific property indicators are less than the counts of bits, which are used in the data stream to signal the layer-specific property indicators. For example, the latter may apply to one or more or all of the set of layer-specific property indicators. For example, apparatus 20 may infer that only a portion of a maximum domain provided by the bits used for signaling a layer-specific property indicator is used, e.g., that the layer-specific property indicator may only assume a subset out of values provided by a bit-length of the layer-specific property indicator.

[0096] For example, usage of the auxiliary pictures of the same time frame may be constrained to cases, in which the layer-specific property indicators used for identifying the auxiliary sample arrays assume only values, which are representable by a predefined count of bits, which is less than the count of bits required to represent the full possible domain of the layer-specific property indicators.

[0097] According to an embodiment, apparatus 20 is configured for deriving the set of layer-specific property indicators from an indication, e.g., a syntax element in the data stream, e.g., from the value of nnpfc_auxiliary_inp_idc.

[0098] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmIn other words, the above-mentioned plurality of modes may include multiple modes, in which auxiliary sample arrays of the same time frame are used as input filter data for filtering the reconstructed picture 12’, and for different ones of these modes, apparatus 20 may apply a different mode of deriving the set of layer-specific property indicators.

[0099] Additionally or alternatively, the same may apply to the derivation of the domains of the layer-specific property indicators, which may be determined by apparatus 20 based on an indication in the data stream, e.g., based on the value of nnpfc_auxiliary_inp_idc.

[0100] In the following, an example for the usage of layer-specific property indicators for identifying the auxiliary sample arrays 15 will be described by the example of AVC. However, the same approach may be applied for other codecs.

[0101] AVC does not have layerjd with 6 bits defined in the NAL unit header as VVC and HEVC. AVC defines the following dimensions:

[0102] quality_id(4), dependency_id(3); view_idx(8), depth_flag(1); or view_id(10)

[0103] In another embodiment there is a constraint that the auxiliary layers can only be used when a bitstream only contains combination of:

[0104] quality_id(4), dependency_id(3);

[0105] view_idx(8), depth_flag(1);

[0106] In addition, the syntax elements can only use a subset of their potential values so that they can be combined into a particular length (e.g. 6 bit element). For instance:

[0107] quality_id(4) can only take 3 bits, dependency_id(3) can take the 3 bits; or quality_id(4) can take 4 bits, but dependency_id(3) can take only 2 bits; view_idx(8) can take only 5 bits, and depth_flag(1) 1 bit;

[0108] In another embodiment further values of nnpfc_auxiliary_inp_idc are indicated so that the mapping can be done more flexible, indicating that the auxiliary picture identifier is a qualityjd, or dependencyjd or combinations thereofe using a subset of the potential values, or all, or different combinations with different syntax elements among quality_id(4), dependency_id(3); view_idx(8), depth_flag(1); or view_id(10).

[0109] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmIn the following, a further alternative on how to provide the identification of the one or more auxiliary sample arrays 15 in data stream 14 is described. According to an embodiment, apparatus 20 derives the indication indicating the count, or quantity, of layers of the set of layers from a syntax element contained in a first payload packet, e.g., an SEI NAL unit, of the data stream. According to this embodiment, for each of the set of layers, an identification of the respective layer among the plurality of layers of the data stream is derived by apparatus 20 from a second payload packet of the data stream. For example, the second payload packet is an SEI NAL unit.

[0110] For example, a supplemental enhancement information, SEI NAL unit referred to within this description may be a payload packet interspersed between, or signaled ahead, of payload packets carrying coded video data, e.g., VCL NAL units.

[0111] According to an embodiment, apparatus 20 derives, from the second payload packet, an indication, e.g., a syntax element, e.g., napm_mode_idc described below, indicating a set of layer-specific property indicators, e.g., as described above. Each of the layer-specific property indicators indicates a property of the respective layer by signaling a value out of a domain of the respective layer-specific property indicators. According to this embodiment, apparatus 20 derives, from the data stream 14, for each of the plurality of layers, the set of layer-specific property indicators, and, for each of the set of layers, apparatus 20 may derive the identification of the respective layer among the plurality of layers by deriving, from the data stream 14 a layer identification syntax element, e.g., syntax element napm_aux_pic_id described below, which signals the identification of the respective layer within the plurality of layers. According to this embodiment, the layer identification syntax element indicates a value for each layer-specific property indicator of the set of layer-specific property indicators and identifies the respective layer based on the combination of values of the layer-specific property indicators. For example, the layer identification syntax element comprises, or consists of, a concatenation of respective values for the layer-specific property indicators of the set of layer-specific property indicators.

[0112] According to an embodiment, apparatus 20 derives, for each of the set of layers, the identification of the respective layer among the plurality of layers by deriving, from the second payload packet, an indication, e.g., a syntax element, e.g., napm_mode_idc described below, indicating a set of layer-specific property indicators. For example, the indication identifies the set of layer-specific property indicators out of a plurality of layerspecific property indicators included in the data stream 14. For example, the set of layer-

[0113] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmspecific property indicators is a set of syntax elements, each of which indicates a property of the respective layer. For example, the set comprises one or more or all of, e.g., any combination out of, qualityjd, dependencyjd, viewjdx, depth_flag, or viewjd. Each of the layer-specific property indicators indicates a property of the respective layer by signaling a value out of a domain of the respective layer-specific property indicator. For example, the value represents the property of the respective layer. According to this embodiment, apparatus 20 derives, from data stream 14, the set of layer-specific property indicators, i.e. , the layer-specific property indicators of the set. Apparatus 20 further derives, from the data stream 14, a layer identification syntax element, which signals the identification of the respective layer within the plurality of layers. According to this embodiment, the layer identification syntax element indicates a value for each layer-specific property indicator of the set of layer-specific property indicators, and identifies the respective layer based on the combination of values of the layer-specific property indicators.

[0114] In other words, according to an embodiment, the layer identification syntax element, e.g., as already described above, may be signaled, for each of the set of layer-specific property indicators, which is to be used for identifying the set of layers carrying the auxiliary sample arrays 15, are carried in a separate payload packet, e.g., a separate SEI message, e.g., the nnpf_auxiliary_pictures_mapping SEI message described below. An example of such an embodiment is described in the following in terms of an amendment to the above-described nnpf process.

[0115] In other words, as an alternative on how to provide the mapping of auxiliary pictures to the actual pictures NNPFC could only provide the information that there are auxiliary pictures and at most how may. This is shown in the following:

[0116] & >

[0117]

[0118] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bm

[0119] & >

[0120]

[0121] In addition, a further SEI message could be provided that indicates the mapping to picture within the bitstream. For instance, a nnpf_auxiliary_pictures_mapping SEI message:

[0122] <

[0123]

[0124] napm_num_aux_pic_minusl plus 1 indicates the number of auxiliary pictures for which a mapping is provided.

[0125] napm_mode_idc specifies the type of identifier used to map auxiliary pictures. Examples thereof can be:

[0126] nuh layer id

[0127] view_id

[0128] depth id

[0129] or some syntax element for which the mapping is defined at each video coding standard Depending on the mode ide the length of the napm_aux_pic id ( u(v) ) could vary, having e.g., 6 bits for layer id, but 8 bit for view id or 10 bits, an so on.

[0130] napm_aux_pic_id[ i ] specifies identifier of the i-th auxiliary input pictures.

[0131] In addition, the modejdc could define an identifier as a mixture of two syntax elements. For instance, a first syntax element consisting of 4 bits followed by a two bits syntax element.

[0132] As a further option the modejdc could change for each entry:

[0133] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bm <

[0134]

[0135] Thus, each entry in the loop could be mapped to a different picture differentiated by a different syntax element or property.

[0136] In the following, further optional details regarding the indication indicating the set of layerspecific property indicators will be described, which may optionally apply to any of the embodiments described above, i.e. , independent of the question in which payload packet this indication is signaled.

[0137] According to an embodiment, the indication indicating the set of layer-specific property indicators is signaled by a syntax element, e.g., the layer-identification syntax element, which differentiates between a plurality of modes, the plurality of modes including one or more of:

[0138] • A first mode, according to which the set of layer-specific property indicators comprises, or consists of a layer identifier,

[0139] • A second mode, according to which the set of layer-specific property indicators comprises, or consists of, a view identifier and / or a depth identifier,

[0140] • And a third mode, according to which the set of layer-specific property indicators comprises, or consists of a layer identifier and one or more further layer-specific property indicators, e.g., a view identifier and / or a depth identifier.

[0141] According to an embodiment, apparatus 20 derives a further syntax element from the data stream, which indicates a bit-length of the layer identification syntax element in the data stream. For example, the bit-length is a count of bits using which the layer identification syntax element is encoded in the data stream.

[0142] For example, the layer identification syntax element may be encoded in the data stream with a fixed bit-length.

[0143] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmAccording to an embodiment, apparatus 20 derives the identification of the respective layer within the plurality of layers of the data stream 14 by deriving, from the data stream 14 a layer identification syntax element, which identifies the respective layer by signaling a layer identifier, which is unique to the respective layer, e.g., by signaling a value corresponding to a value of the layer identifier of the respective layer. According to this embodiment, each of the plurality of layers is associated with a respective layer identifier. Alternatively, the layer identification syntax element identifies the respective layer by indicating a value for each of its set of layer-specific property indicators, each of the layer-specific property indicators indicating a property of the respective layer by signaling a value out of a domain of the respective layer-specific property indicator. According to this embodiment, apparatus 20 derives a further syntax element from the data stream, the further syntax element indicating a bit-length of the layer identification syntax element in the data stream, e.g., as described above.

[0144] In other words, what has been described above, the layer identification syntax element, may either be a layer identifier, which may be directly mapped to a layer of the data stream in that, for example, a domain, in which the layer identifiers are defined may represent the count of layers in the data stream 14. Alternatively, the layer identification syntax element may indicate values for each of the set of layer-specific property indicators, e.g., the layer identification syntax element may be a concatenation of values for the set of layer-specific property indicators.

[0145] As already mentioned, the reconstructed picture 12’ may comprise one or more sample arrays and, furthermore, different types of data are possible for the one or more auxiliary sample arrays 15. Thus, the manner of deriving the filter input data 65 by means of filter input determinator 63 may depend on the properties of the reconstructed picture 12’, the count and the properties of the one or more auxiliary sample arrays 15, and furthermore, the manner of deriving the filter input data 65 may be performed in accordance with the neural network, e.g., the dimension of an input layer of the neural network applied by filtering module 61.

[0146] For example, filter input determinator 63 may determine the filter input data 65 using the luma sample arrays of the reconstructed picture 12’ and the one or more auxiliary sample arrays only, or the chroma sample arrays only, or both of them. Furthermore, the manner of deriving the filter input data may depend upon a chroma format of the reconstructed picture 12’ and the one or more auxiliary sample arrays.

[0147] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmFig. 2 illustrates an embodiment of the filter input determinator 63. According to this embodiment, filter input determinator 63 obtains the filter input data 65 in form of a tensor comprising, or consisting of, a plurality of channels, each of which provides an input channel for the neural network. According to this embodiment, filter input determinator 63 determines a first number of first channels 67 based on sample arrays of the reconstructed picture 12’, and, for each of the auxiliary pictures, a second number of second channels 69 of the plurality of channels. For example, the count of the second number may vary between the auxiliary pictures or may be equal for each of them.

[0148] For example, in Fig. 2, the reconstructed picture 12’ exemplarily comprises three sample arrays, e.g., a luma and two chroma sample arrays, and filter input determinator 63 determines an exemplary number of two first channels 67 based on the sample arrays of the reconstructed picture 12’. Furthermore, in the illustrated example of Fig. 2, the one or more auxiliary sample arrays 15 comprise two auxiliary pictures 15i, 152, each of which comprises three of the one or more auxiliary sample arrays, e.g., each one luma sample array and two chroma sample arrays. In the example of Fig. 2, filter input determinator determines, for each of the auxiliary pictures 15i, 152, two second channels 69, resulting in a number of four second channels 69. For example, filter input determinator 63 may determine, for each of the pictures 12’, 15i, 152, two channels based on the two chroma sample arrays of the respective picture. In other examples, the number of channels determined per input sample array may be one or three. Furthermore, the number of channels determined per sample array is not necessarily equal for each of the sample arrays which are input to filter input determinator 63. For example, the number of second channels determined per auxiliary picture may be selected individually for each of the auxiliary pictures, as will be described in more detail below.

[0149] According to an embodiment, filter input determinator 63 determines a count of the second number of second channels 69 based on the count of auxiliary pictures, i.e., a count of auxiliary pictures included in the set of auxiliary pictures, and based on an indication 64, which indicates a manner of deriving the second channels. For example, apparatus 20 derives the indication 64 from data stream 14, e.g., by decoding a syntax element from data stream 14. For example, indication 64 may indicate a number or a count of channels to be derived based on the auxiliary pictures and, optionally, an order in which the second channels are to be included in the input filter data 64, e.g., in the tensor period.

[0150] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmIn other words, according to an embodiment, apparatus 20 is configured for deriving, from the data stream 14, the indication 64, e.g., nnpfc_inp_order, compare, e.g., Table 1.

[0151] According to an embodiment, the count of second channels 69 is equal for all of the auxiliary pictures.

[0152] Alternatively, according to an embodiment, apparatus 20 derives, from data stream 14, for each of the auxiliary pictures 15i, 152, a respective indication of a manner of deriving the second channels. According to this embodiment, filter input determinator 63 derives the second number of second channels individually for each of the auxiliary pictures. For example, filter input determinator may determine the count of second channels to be derived based on the respective auxiliary picture individually for each of the auxiliary pictures.

[0153] An example for signaling the manner of deriving the second channels individually for each of the auxiliary pictures is given below by means of syntax element nnpfc_aux_inp_order_idc.

[0154] For example, one or more or all of the following variants of deriving the plurality of channels of the filter input data 65 may be possible, that is, filter input determinator 63 may be configured to use one of the variants described in the following, or the following variants may represent different modes of a plurality of modes which comprises one or more or all of the variants which will be described in the following, and filter input determinator 63 selects one of the modes out of the plurality of modes, e.g., based on an indication signaled in data stream 14. In other words, apparatus 20 may derive the filter input data 65 by selecting, e.g., in dependence on a syntax element derived from data stream 14, e.g., an nnpfc_inp_order_idc described below, one out of a plurality of modes, the plurality of modes comprising one or more or all of the following variants.

[0155] According to a first variant, filter input determinator derives the channels of the filter input data 65 to comprise a first channel comprising, or consisting of, a luma sample array derived from a luma sample array of the reconstructed picture, and for each of the auxiliary pictures, a second channel comprising, or consisting of, an auxiliary luma sample array, e.g., derived from a luma sample array of the auxiliary picture. For example, in deriving the first and the second channel, filter input determinator may select respective patches out of the luma sample arrays of the reconstructed picture 12’ and the luma sample array of the auxiliary picture.

[0156] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmAccording to a second variant, e.g., as illustrated in Fig. 2, filter input determinator 63 may derive the channels of the filter input data 65 to comprise a first first channel comprising, or consisting of, a first chroma sample array derived from a first chroma sample array of the reconstructed picture 12’, and a second first channel comprising, or consisting of, a second chroma sample array derived from a second chroma sample array of the reconstructed picture 12’. Additionally, filter input determinator 63 may determine, for each of the auxiliary pictures 15i , 152, a first second channel comprising, or consisting of, a first auxiliary sample array, e.g. , derived from a first chroma sample array of the auxiliary picture or default values, and a second second channel comprising, or consisting of, a second auxiliary sample array, e.g., derived from a second chroma sample array of the auxiliary picture.

[0157] According to a third variant, filter input determinator 63 may derive the filter input data 65 to comprise a first channel comprising, or consisting of, a luma sample array derived from a luma sample array of the reconstructed picture, and for each of the auxiliary pictures a second channel comprising, or consisting of, an auxiliary luma sample array, e.g., derived from a luma sample array of the auxiliary picture. Additionally, filter input determinator 63 may determine a first first channel comprising, or consisting of, a first chroma sample array derived from a first chroma sample array of the reconstructed picture, a second first channel comprising, or consisting of, a second chroma sample array derived from a second chroma sample array of the reconstructed picture, and, for each of the auxiliary pictures a first second channel comprising, or consisting of, a first auxiliary sample array, e.g., derived from a first chroma sample array of the auxiliary picture, and a second second channel comprising, or consisting of, a second auxiliary sample array, e.g., derived from a second chroma sample array of the auxiliary picture.

[0158] In other words, the filter input data 65 may comprise, for each of the reconstructed picture 12’ and the auxiliary pictures 15i, 152, a luma channel derived from the luma sample array of the respective picture and two chroma channels derived from the two chroma channels of the respective picture.

[0159] According to a fourth variant, filter input determinator 63 determines, for the filter input data 65, four first channels, each comprising or consisting of, a luma sample array derived from a luma sample array of the reconstructed picture 12’, and, for each of the auxiliary pictures, four second channels, each comprising, or consisting of, an auxiliary luma sample array, e.g., derived from a luma sample array of the auxiliary picture. Furthermore, according to

[0160] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmthis variant, filter input determinator 63 determines, for the filter input data 65, a first first channel comprising, or consisting of, a first chroma sample array derived from a first chroma sample array of the reconstructed picture, a second first channel comprising, or consisting of, a second chroma sample array derived from a second chroma sample array of the reconstructed picture, and, for each of the auxiliary pictures, a first second channel comprising, or consisting of, a first auxiliary sample array, e.g., derived from a first chroma sample array of the auxiliary picture, and a second second channel comprising, or consisting of, a second auxiliary sample array, e.g., derived from a second chroma sample array of the auxiliary picture.

[0161] In other words, according to the fourth variant, four luma channels are derived per input picture, the input pictures including the reconstructed picture 12’ and the auxiliary pictures, and additionally, for each of the input pictures, two chroma input channels are derived, one per chroma sample array of the respective input picture. For example, this variant may be applied in cases in which a chroma format of the reconstructed picture and the auxiliary pictures is YCrCb 4:2:0, i.e. , the subsampling rate of both chroma components compared to the luma component is 2:1. For example, according to the fourth variant, the luma samples may be distributed to four separate sample arrays. This manner of deriving the luma channels allows to derive the plurality of channels of the input filter data 65 with equal dimensions, i.e., each of the channels has the same size or the same number of samples.

[0162] Fig. 3 illustrates a derivation of the four luma channels 67 from a luma component 17 according to an embodiment. For example, the manner of deriving luma channels may be employed in case that the below-mentioned syntax element nnpfc_ inp_order_idc is equal to three. The manner of deriving the luma channels may be used for both the reconstructed picture and the auxiliary pictures in an equivalent manner.

[0163] According to an embodiment, the input channels of the input filter data 65 have equal dimensions.

[0164] According to an embodiment, the manner of deriving one or more of the second channels based on the auxiliary pictures may be determined individually for each of the auxiliary pictures.

[0165] In more general terms, according to an embodiment, apparatus 20 obtains one or more first channels of the plurality of channels based on the reconstructed picture 12’ and obtains one

[0166] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmor more second channels of the plurality of channels based on a set of auxiliary pictures, each of the auxiliary pictures being represented by one or more of the one or more auxiliary sample arrays. According to this embodiment, filter input determinator 63 derives, from the data stream, for each of the auxiliary pictures, an indication which indicates a manner of deriving one or more of the second channels based on the respective auxiliary picture.

[0167] In other words, the indication 64 may comprise an individual indication for each of the auxiliary pictures.

[0168] According to an embodiment, filter input determinator 63 determines, for each of the auxiliary pictures, one or more of the second channels based on a set of auxiliary sample arrays out of the auxiliary sample arrays of the respective auxiliary picture. According to this embodiment, the set of auxiliary sample arrays of the respective auxiliary picture is indicated to be used for the filter input data according to the indication, which indicates the manner of deriving the one or more second channels based on the respective auxiliary picture.

[0169] As already described, the one or more auxiliary pictures may be carried in respective layers of the video data stream, and may be identified by identifying a set of layers carrying the auxiliary pictures.

[0170] Accordingly, according to another embodiment, the indication 64 may indicate, for each of the layers of the set of layers, a manner of deriving one or more of the second channels based on the auxiliary picture carried in the respective layer.

[0171] In other words, according to embodiments, in which the one or more auxiliary pictures are identified by deriving, from the data stream, an indication of a set of layers out of a plurality of layers of the data stream 14, each of the set of layers carrying one of the auxiliary pictures, apparatus 20 may derive, from data stream 14, for each of the layers of the set of layers, an indication, which indicates a manner of deriving one or more of the second channels based on the auxiliary picture carried in the respective layer.

[0172] The further details of deriving the second channels described above may equivalently apply to the embodiments, in which the identification of the auxiliary pictures is performed by means of identifying a set of layers.

[0173] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmFor example, the embodiments described above may be implemented within the framework of the NN PF process in terms of an additional state of the syntax element nnpfc_auxiliary_inp_idc, which indicates the usage of the one or more auxiliary sample arrays of the same time frame. In this case, for example, the differentiation between the above-mentioned modes or variants of deriving the channels of the input filter data 65 may be signalled using the syntax element nnpfc_inp_order_idc, which for this case not only indicates the manner of deriving channels from the reconstructed picture 12’, i.e. , the first channels 67, but also the manner of deriving the channels from the auxiliary pictures, i.e., the second channels 69.

[0174] In other words, as a further aspect of the above-mentioned “first embodiment”, the pictures of different layers are added to the NN PF process as additional channels for which such pictures are in the following referred to as auxiliary pictures.

[0175] The neural-network post-filter characteristics (NNPFC) SEI message specifies a neural network that may be used as a post-processing filter. The use of specified neural-network post-processing filters (NNPFs) for specific pictures is indicated with neural -network post-filter activation (NNPFA) SEI messages.

[0176] Use of this SEI message requires the definition of the following variables:

[0177] - Input picture width and height in units of luma samples, denoted herein by CroppedWidth and CroppedHeight, respectively.

[0178] - Luma sample array CroppedYPic[ idx ] and chroma sample arrays CroppedCbPic[ idx ] and CroppedCrPic[ idx ], when present, of the input pictures with index idx in the range of 0 to numlnputPics - 1, inclusive, that are used as input for the NNPF.

[0179] - Bit depth BitDepthY for the luma sample array of the input pictures.

[0180] - Bit depth BitDepthC for the chroma sample arrays, if any, of the input pictures.

[0181] - A chroma format indicator, denoted herein by ChromaFormatldc, as described in clause 7.3. - When nnpfc auxiliary inp idc is equal to 1, a filtering strength control value array StrengthControl Vai [ idx ] that shall contain real numbers in the range of 0 to 1, inclusive, of the input pictures with index idx in the range of 0 to numlnputPics - 1, inclusive.

[0182] - When nnpfc auxiliary inp idc is equal to 4, luma auxiliary sample array CroppedYAuxPic[ idx ][ layerldx ] and chroma auxiliary sample arrays CroppedCbAuxPic[ idx ][ layerldx ] and CroppedCrAuxPic[ idx ][ layerldx ] when present, of the input pictures with index idx in the range of 0 to numlnputPics - 1 and layer index layerldx in the range of 0 to nnpfc_num_aux_layers_minusl, inclusive.

[0183] When nnpfc_inp_format_idc is equal to 0, the input values to the NNPF are real numbers and the functions InpY( ) and InpC( ) are specified as follows:

[0184] BitDepthy ) - 1 )

[0185]

[0186] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmInpC( x )= x ( ( 1 « BitDepthc ) - 1 )

[0187] (81)

[0188] When nnpfc_inp_format_idc is equal to 1, the input values to the NNPF are unsigned integer numbers and the functions InpY( ) and InpC( ) are specified as follows:

[0189] shiftY = BitDepthy - inpTensorBitDepthy

[0190] if( inpTensorBitDepthy >= BitDepthy)

[0191] InpY ( x ) = x « ( inpTensorBitDepthy - BitDepthy ) (82) else

[0192] InpY( x ) = Clip3(0, ( 1 « inpTensorBitDepthy ) - l, ( x + ( I « ( shiftY - 1 ) ) ) » shiftY )

[0193] shiftC = BitDepthc - inpTensorBitDepthc

[0194] if( inpTensorBitDepthc >= BitDepthc )

[0195] InpC( x ) = x « ( inpTensorBitDepthc - BitDepthc ) (83) else

[0196] InpC( x ) = Clip3(0, ( 1 « inpTensorBitDepthc ) - l, ( x + ( I « ( shiftC - 1 ) ) ) » shiftC )

[0197] nnpfc_auxiliary_inp_idc greater than 0 indicates that auxiliary input data is present in the input tensor of the NNPF. nnpfc auxiliary inp idc equal to 0 indicates that auxiliary input data is not present in the input tensor, nnpfc auxiliary inp idc equal to 1, 2, 3 or 4 specifies that auxiliary input data is derived as specified in Equation 96.

[0198] When nnpfc auxiliary inp idc is equal to 2 or 3, nnpfc_spatial_extrapolation_prompt_present_flag shall be equal to 1.

[0199] The value of nnpfc_auxiliary_inp_idc shall be in the range of 0 to 255, inclusive. Values of 5 to 255, inclusive, for nnpfc auxiliary inp idc are reserved for future use by ITU-T | ISO / IEC and shall not be present in bitstreams conforming to this version of this Specification. Decoders conforming to this version of this Specification shall ignore NNPFC SEI messages with nnpfc_auxiliary_inp_idc in the range of 5 to 255, inclusive.

[0200] When nnpfc auxiliary inp idc is equal to 1 the auxiliary input data consists of strengthControlScaledVal[ i ].

[0201] When nnpfc auxiliary inp idc is equal to 2 the auxiliary input data consists of nnpfc_prompt character values.

[0202] When nnpfc auxiliary inp idc is equal to 3, the auxiliary input data consists of strengthControlScaledVal[ i ] and nnpfc_prompt character values.

[0203] When nnpfc auxiliary inp idc is equal to 4, the auxiliary input data consists of sample arrays CroppedYAuxPic[ idx ] [ layerldx ] and CroppedCbAuxPic[ idx ] [ layerldx ] and CroppedCrAuxPic[ idx ][ layerldx ] when present, of auxiliary input pictures.

[0204] nnpfc_inp_order_idc indicates the method of ordering the sample arrays of an input picture to form an input tensor to the NNPF.

[0205] The value of nnpfc_inp_order_idc shall be in the range of 0 to 255, inclusive. Values of 4 to 255, inclusive, for nnpfc inp order idc are reserved for future use by ITU-T | ISO / IEC and shall not be present in bitstreams conforming to this version of this Specification. Decoders conforming to this version of this Specification shall ignore NNPFC SEI messages with nnpfc_inp_order_idc in the range of 4 to 255, inclusive.

[0206] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmWhen ChromaFormatldc is not equal to 1, nnpfc inp order idc shall not be equal to 3.

[0207] When ChromaFormatldc is equal to 0, nnpfc inp order idc shall be equal to 0.

[0208] When ChromaUpsamplingFlag is equal to 1, nnpfc inp order idc shall not be equal to 0.

[0209] Table 1 contains an informative description of nnpfc inp order idc values.

[0210] Table 1 - Description of nnpfc inp order idc values

[0211]

[0212] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmThe function InpSampleVal( y, x, picHeight, picWidth, croppedPic, cldx ) with inputs being a vertical sample location y, a horizontal sample location x, a picture height picHeight, a picture width picWidth, sample array croppedPic, and component index cldx (equal to 0 for luma, 1 for Cb, and 2 for Cr) returns the value of sampleVal derived as follows:

[0213] NOTE 7 - For the inputs to the function InpSampleVal( ), the vertical location is listed before the horizontal location for compatibility with input tensor conventions of some inference engines.

[0214] if( nnpfc_padding_type = = 0 )

[0215] if( y < 0 | | x < 0 | | y >= picHeight | | x >= picWidth )

[0216] sampleVal = 0

[0217] else

[0218] sampleVal = croppedPicf x ][ y ] (94) else if( nnpfc_padding_type = = 1 )

[0219] sampleVal = croppedPicf Clip3( 0, picWidth - 1, x ) ][ Clip3( 0, picHeight - 1, y ) ] else if( nnpfc_padding_type = = 2 )

[0220] sampleVal = croppedPicf Reflect( picWidth - 1, x ) ][ Reflect( picHeight - 1, y ) ] else if( nnpfc_padding_type = = 3 )

[0221] if( y >= 0 && y < picHeight )

[0222] sampleVal = croppedPicf Wrap( picWidth - 1, x ) ][ y ]

[0223] else if( nnpfc_padding_type = = 4 )

[0224] if( y < 0 | | x < 0 | | y >= picHeight | | x >= picWidth )

[0225] sampleVal = ( cldx = = 0 ? nnpfc_luma_padding_val :

[0226] ( cldx = = 1 ? nnpfc cb _padding_val : nnpfc_cr_padding_val ) ) else

[0227] sampleVal = croppedPicf x ][ y ]

[0228] When nnpfc_auxiliary_inp_idc is equal to 1, the variable strengthControlScaledVal is derived as follows:

[0229] f <

[0230]

[0231] Floor ( StrengthControl Vai [ i ] * ( ( 1 « inpTensorBitDepthy ) - 1 ) ) else if( nnpfc inp order idc = = 1 )

[0232] StrengthControl Scaled Vai | i ] =

[0233] Floor ( StrengthControl Vai [ i ] * ( ( 1 « inpTensorBitDepthc ) - 1 ) ) else

[0234] StrengthControl Seal cd Vai | i ] = StrengthControl Vai [ i ]

[0235] A patch is a rectangular array of samples from a component (e.g., a luma or chroma component) of a picture.

[0236] The process DeriveInputTensors( ), for deriving the input tensor inputTensor for a given vertical sample coordinate cTop and a horizontal sample coordinate cLeft specifying the top-left sample location for the patch of samples included in the input tensor, is specified as follows:

[0237] for( i = 0; i < numlnputPics; i++ ) {

[0238] if( nnpfc inp order idc = = 0 )

[0239] for( yP = -nnpfe overlap; yP < inpPatchHeight + nnpfe overlap; yP++)

[0240] for( xP = -nnpfe overlap; xP < inpPatchWidth + nnpfe o verlap; xP++ ) {

[0241] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bminpVal = InpY( InpSampleVal( cTop + yP, cLeft + xP, CroppedHeight, CroppedWidth, CroppedYPic[ i ], 0 ) )

[0242] yPovlp = yP + nnpfc overlap

[0243] xPovlp = xP + nnpfc overlap

[0244] if( !nnpfc_component_last_flag )

[0245] inputTensor

[0000] [ i ]

[0000] [ yPovlp ][ xPovlp ] = inpVal

[0246] else

[0247] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ]

[0000] = inpVal

[0248]

[0249] inputTensor

[0000] [ i ]

[0001] [ yPovlp ][ xPovlp ] = strengthControlScaledVal[ i ] else

[0250] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ]

[0001] = strengthControlScaledVal[ i ] if( nnpfc_auxiliary_inp_idc = = 2 | | nnpfc_auxiliary_inp_idc = = 3) { promptCharVal = utf8ToUInt( nnpfcPrompt )

[0251] if( !nnpfc_component_last_flag )

[0252] inputTensor

[0000] [ i ] [ nnpfc auxiliary inp idc - 1 ] [ yPovlp ] [ xPovlp ] = promptCharVal

[0253] else

[0254] inputTensor

[0000] [ i ] [ yPovlp ] [ xPovlp ] [ nnpfc auxiliary inp idc - 1 ] = promptCharVal

[0255] }

[0256] if( nnpfc auxiliary inp idc = = 4)

[0257] for(j=0;j <= nnpfc_num_aux_layers_minus 1 ; j++) {

[0258] inpAuxVal = InpY( InpSampleVal( cTop + yP, cLeft + xP, CroppedHeight, CroppedWidth, CroppedYAuxPicf i ][ j ], 0 ) )

[0259] if( !nnpfc_component_last_flag )

[0260] inputTensor

[0000] [ i ][ j + 1 ][ yPovlp ][ xPovlp ] = inpAuxVal

[0261] else

[0262] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ][ j + 1 ] = inpAuxVal

[0263] }

[0264] }

[0265] else if( nnpfc_inp_order_idc = = 1 ) (96) for( yP = -nnpfc overlap; yP < inpPatchHeight + nnpfc overlap; yP++)

[0266] for( xP = -nnpfc overlap; xP < inpPatchWidth + nnpfc o verlap; xP++ ) { inpCbVal = InpC( InpSampleVal( cTop + yP, cLeft + xP, CroppedHeight / SubHeightC,

[0267] CroppedWidth / SubWidthC, CroppedCbPicf i ], 1 ) )

[0268] inpCrVal = InpC( InpSampleVal( cTop + yP, cLeft + xP, CroppedHeight / SubHeightC,

[0269] CroppedWidth / SubWidthC, CroppedCrPicf i ], 2 ) )

[0270] yPovlp = yP + nnpfc overlap

[0271] xPovlp = xP + nnpfc overlap

[0272] if( !nnpfc_component_last_flag ) {

[0273] inputTensorf 0 ] [ i ]

[0000] [ yPovlp ] [ xPovlp ] = inpCbVal

[0274] inputTensorf 0 ][ i ]

[0001] [ yPovlp ][ xPovlp ] = inpCrVal

[0275] } else {

[0276] inputTensorf 0 ][ i ][ yPovlp ][ xPovlp ]

[0000] = inpCbVal

[0277] inputTensorf 0 ] [ i ] [ yPovlp ] [ xPovlp ]

[0001] = inpCrVal

[0278]

[0279] inputTensorf 0 ][ i ]

[0002] [ yPovlp ][ xPovlp ] = strengthControlScaledValf i ] else

[0280] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bminputTensor

[0000] [ i ][ yPovlp ][ xPovlp ]

[0002] = strengthControlScaledVal[ i ] if( nnpfc_auxiliary_inp_idc = = 2 | | nnpfc_auxiliary_inp_idc = = 3) { promptCharVal = utf8ToUInt( nnpfcPrompt )

[0281] if( !nnpfc_component_last_flag )

[0282] inputTensor

[0000] [ i ] [ nnpfc auxiliary inp idc ] [ yPovlp ] [ xPovlp ] = promptCharVal

[0283] else

[0284] inputTensor

[0000] [ i ] [ yPovlp ] [ xPovlp ] [ nnpfc auxiliary inp idc ] = promptCharVal

[0285] }

[0286] if ( nnpfc auxiliary inp idc = = 4)

[0287] for(j=0;j <= nnpfc_num_aux_layers_minus 1 ; j++) {

[0288] inpAuxCbVal = InpC( InpSampleVal( cTop + yP, cLeft + xP, CroppedHeight / SubHeightC, CroppedWidth / SubWidthC, CroppedCbAuxPic[ i ][ j ], 1 ) ) inpAuxCrVal = InpC( InpSampleVal( cTop + yP, cLeft + xP, CroppedHeight / SubHeightC, CroppedWidth / SubWidthC, CroppedCrAuxPic[ i ][ j ], 2 ) ) if( !nnpfc_component_last_flag ){

[0289] inputTensor

[0000] [ i ][2 *j +2 ][ yPovlp ][ xPovlp ] = inpAuxCbVal inputTensor

[0000] [ i ][2 *j + 3 ][ yPovlp ][ xPovlp ] = inpAuxCrVal } else {

[0290] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ][ 2 * j + 2 ] = inpAuxCbVal inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ][ 2 * j + 3 ] = inpAuxCrVal }

[0291] }

[0292] }

[0293] else if( nnpfc_inp_order_idc = = 2 )

[0294] for( yP = -nnpfc overlap; yP < inpPatchHeight + nnpfc overlap; yP++)

[0295] for( xP = -nnpfc overlap; xP < inpPatchWidth + nnpfc o verlap; xP++ ) {

[0296] yY = cTop + yP

[0297] xY = cLeft + xP

[0298] yC = yY / SubHeightC

[0299] xC = xY / SubWidthC

[0300] inpYVal = InpY( InpSampleVal( yY, xY, CroppedHeight,

[0301] CroppedWidth, CroppedYPic[ i ], 0 ) )

[0302] inpCbVal = InpC( InpSampleVal( yC, xC, CroppedHeight / SubHeightC, CroppedWidth / SubWidthC, CroppedCbPic[ i ], 1 ) )

[0303] inpCrVal = InpC( InpSampleVal( yC, xC, CroppedHeight / SubHeightC, CroppedWidth / SubWidthC, CroppedCrPic[ i ], 2 ) )

[0304] yPovlp = yP + nnpfc overlap

[0305] xPovlp = xP + nnpfc overlap

[0306] if( !nnpfc_component_last_flag ) {

[0307] inputTensor

[0000] [ i ]

[0000] [ yPovlp ][ xPovlp ] = inpYVal

[0308] inputTensor

[0000] [ i ]

[0001] [ yPovlp ] [ xPovlp ] = inpCbVal

[0309] inputTensor

[0000] [ i ]

[0002] [ yPovlp ][ xPovlp ] = inpCrVal

[0310] } else {

[0311] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ]

[0000] = inpYVal

[0312] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ]

[0001] = inpCbVal

[0313] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ]

[0002] = inpCrVal

[0314]

[0315] promptCharVal = utf8ToUInt( nnpfcPrompt )

[0316] if( !nnpfc_component_last_flag )

[0317] inputTensor

[0000] [ i ] [ nnpfc auxiliary inp idc + 1 ] [ yPovlp ] [ xPovlp ] = promptCharVal

[0318] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmelse

[0319] inputTensor

[0000] [ i ] [ yPovlp ] [ xPovlp ] [ nnpfc auxiliary inp idc + 1 ] = promptCharVal

[0320] }

[0321] if ( nnpfc auxiliary inp idc = = 4)

[0322] for(j=0;j <= nnpfc_num_aux_layers_minus 1 ; j++) {

[0323] inpAuxYVal = InpY( InpSampleVal( yY, xY, CroppedHeight, CroppedWidth, CroppedYAuxPic[ i ][ j ], 0 ) )

[0324] inpAuxCbVal = InpC( InpSampleVal( yC, xC, CroppedHeight / SubHeightC, CroppedWidth / SubWidthC, CroppedCbAuxPic[ i ][ j ], 1 ) ) inpAuxCrVal = InpC( InpSampleVal( yC, xC, CroppedHeight / SubHeightC, CroppedWidth / SubWidthC, CroppedCrAuxPic[ i ][ j ], 2 ) ) if( !nnpfc_component_last_flag ){

[0325] inputTensor

[0000] [i][3 *j + 3 ][ yPovlp ][ xPovlp ] = inpAuxYVal inputTensor

[0000] [i][3 *j +4 ][ yPovlp ][ xPovlp ] = inpAuxCbVal inputTensor

[0000] [i][3 *j + 5 ][ yPovlp ][ xPovlp ] = inpAuxCrVal } else {

[0326] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ][ 3 * j + 3 ] = inpAuxYVal inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ][ 3 * j + 4 ] = inpAuxCbVal inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ][ 3 * j + 5 ] = inpAuxCrVal }

[0327] }

[0328] }

[0329] else if( nnpfc_inp_order_idc = = 3 )

[0330] for( yP = -nnpfc overlap; yP < inpPatchHeight + nnpfc overlap; yP++) for( xP = -nnpfc overlap; xP < inpPatchWidth + nnpfc o verlap; xP++ ) { yTL = cTop

[0331] xTL = cLeft

[0332] yBR = yTL

[0333] xBR = xTL

[0334] yC = cTop /

[0335] xC = cLeft /

[0336]

[0337] inpTLVal = InpY( InpSampleVal( yTL, xTL, CroppedHeight, CroppedWidth, CroppedYPic[ i ], 0 ) )

[0338] inpTRVal = InpY( InpSampleVal( yTL, xBR, CroppedHeight, CroppedWidth, CroppedYPic[ i ], 0 ) )

[0339] inpBLVal = InpY( InpSampleVal( yBR, xTL, CroppedHeight, CroppedWidth, CroppedYPic[ i ], 0 ) )

[0340] inpBRVal = InpY( InpSampleVal( yBR, xBR, CroppedHeight, CroppedWidth, CroppedYPic[ i ], 0 ) )

[0341] inpCbVal = InpC( InpSampleVal( yC, xC, CroppedHeight / 2, CroppedWidth / 2, CroppedCbPic[ i ], 1 ) )

[0342] inpCrVal = InpC( InpSampleVal( yC, xC, CroppedHeight / 2, CroppedWidth / 2, CroppedCrPic[ i ], 2 ) )

[0343] yPovlp = yP + nnpfc overlap

[0344] xPovlp = xP + nnpfc overlap

[0345] if( !nnpfc_component_last_flag ) {

[0346] inputTensor

[0000] [ i ]

[0000] [ yPovlp ][ xPovlp ] = inpTLVal

[0347] inputTensor

[0000] [ i ]

[0001] [ yPovlp ][ xPovlp ] = inpTRVal

[0348] inputTensor

[0000] [ i ]

[0002] [ yPovlp ][ xPovlp ] = inpBLVal

[0349] inputTensor

[0000] [ i ]

[0003] [ yPovlp ][ xPovlp ] = inpBRVal

[0350] inputTensor

[0000] [ i ]

[0004] [ yPovlp ] [ xPovlp ] = inpCbVal

[0351] inputTensor

[0000] [ i ]

[0005] [ yPovlp ][ xPovlp ] = inpCrVal

[0352] } else {

[0353] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bminputTensor

[0000] [ i ][ yPovlp ][ xPovlp ]

[0000] = inpTLVal

[0354] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ]

[0001] = inpTRVal

[0355] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ]

[0002] = inpBLVal

[0356] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ]

[0003] = inpBRVal

[0357] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ]

[0004] = inpCbVal

[0358] inputTensor

[0000] [ i ] [ yPovlp ] [ xPovlp ]

[0005] = inpCrVal

[0359]

[0360] inputTensor

[0000] [ i ]

[0006] [ yPovlp ][ xPovlp ] = strengthControlScaledVal[ i ] else

[0361] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ]

[0006] = strengthControlScaledVal[ i ] if( nnpfc_auxiliary_inp_idc = = 2 | | nnpfc_auxiliary_inp_idc = = 3) { promptCharVal = utf8ToUInt( nnpfcPrompt )

[0362] if( !nnpfc_component_last_flag )

[0363] inputTensor

[0000] [ i ] [ nnpfc auxiliary inp idc + 4 ] [ yPovlp ] [ xPovlp ] = promptCharVal

[0364] else

[0365] inputTensor

[0000] [ i ] [ yPovlp ] [ xPovlp ] [ nnpfc auxiliary inp idc + 4 ] = promptCharVal

[0366] }

[0367] if( nnpfc auxiliary inp idc = = 4)

[0368] for(j=0;j <= nnpfc_num_aux_layers_minus 1 ; j++) {

[0369] inpAuxTLVal = InpY( InpSampleVal( yTL, xTL, CroppedHeight, CroppedWidth, CroppedYAuxPic[ i ][ j ], 0 ) )

[0370] inpAuxTRVal = InpY( InpSampleVal( yTL, xBR, CroppedHeight, CroppedWidth, CroppedYAuxPic[ i ][ j ], 0 ) )

[0371] inpAuxBLVal = InpY( InpSampleVal( yBR, xTL, CroppedHeight, CroppedWidth, CroppedYAuxPic[ i ][ j ], 0 ) )

[0372] inpAuxBRVal = InpY( InpSampleVal( yBR, xBR, CroppedHeight, CroppedWidth, CroppedYAuxPic[ i ][ j ], 0 ) )

[0373] inpAuxCbVal = InpC( InpSampleVal( yC, xC, CroppedHeight / 2, CroppedWidth / 2, CroppedCbAuxPic[ i ][ j ], 1 ) )

[0374] inpAuxCrVal = InpC( InpSampleVal( yC, xC, CroppedHeight / 2, CroppedWidth / 2, CroppedCrAuxPic[ i ][ j ], 2 ) )

[0375] if( !nnpfc_component_last_flag ){

[0376] inputTensor

[0000] [i][6 *j + 6 ][ yPovlp ][ xPovlp ] = inpAuxTLVal inputTensor

[0000] [ i ][ 6 * j + 7 ][ yPovlp ][ xPovlp ] = inpAuxTRVal inputTensor

[0000] [ i ][ 6 * j + 8 ][ yPovlp ][ xPovlp ] = inpAuxBLVal inputTensor

[0000] [ i ][ 6 * j + 9 ][ yPovlp ][ xPovlp ] = inpAuxBRVal inputTensor

[0000] [i][ 6 *j + 10 ][ yPovlp ][ xPovlp ] = inpAuxCbVal inputTensor

[0000] [i][ 6 *j + ll ][ yPovlp ][ xPovlp ] = inpAuxCrVal } else {

[0377] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp inpAuxTLVal inputTensor

[0000] [ i ][ yPovlp ][ xPovlp inpAuxTRVal inputTensor

[0000] [ i ][ yPovlp ][ xPovlp inpAuxBLVal inputTensor

[0000] [ i ][ yPovlp ][ xPovlp inpAuxBRVal inputTensor

[0000] [ i ][ yPovlp ][ xPovlp = inpAuxCbVal inputTensor

[0000] [ i ][ yPovlp ][ xPovlp

[0378]

[0379] = inpAuxCrVal }

[0380] }

[0381] }

[0382] }

[0383] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmutf8ToUInt( x ) {

[0384] result = 0

[0385] len = 0

[0386] / * Check end of text prompt string * /

[0387] if( x = = null )

[0388] return 0

[0389] / * Determine the number of bytes in the UTF-8 character * /

[0390] if( (x

[0000] & 0x80 ) = = 0 )

[0391] len = 1 / * 1-byte character * /

[0392] else if( (x

[0000] & OxEO ) = = OxCO )

[0393] len = 2 / * 2-byte character * /

[0394] else if( (x

[0000] & OxFO ) = = OxEO )

[0395] len = 3 / * 3 -byte character * /

[0396] else if( (x

[0000] & 0xF8 ) = = OxFO )

[0397] len = 4 / * 4-byte character * /

[0398] else

[0399] len = 0 / * Invalid UTF-8 character; this case shall not occur in bitstreams. * /

[0400] for( i = 0; i < len; i++ ) / * Construct an integer from the bytes * /

[0401] result = ( result « 8 ) | x[ i ]

[0402] x = x + len / * Modifies the input variable, which is a syntax element * /

[0403] return result

[0404] }

[0405] Note that the previous embodiment assumes that pictures of the different layers must have the same picture size, luma depth, chroma depth, chroma format.

[0406] In particular, in some interesting use-cases, such as having a texture layer and an enhancement layers with depth maps or object masks, it is quite common to have a different chroma format for the texture layer (with luma and chroma components) and depth map or object masks (typically monochrome with only a luma component).

[0407] In a further embodiment, further information is provided in the bitstream that indicates the number of components per additional auxiliary picture (e.g., of a different layer).

[0408] & >

[0409]

[0410] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bm

[0411] & >

[0412] <

[0413]

[0414] nnpfc_aux_inp_order_idc[ i ] indicates the method of ordering the sample arrays of an i-th auxiliary input picture to form an input tensor to the NNPF.

[0415] As already mentioned, the manner of deriving respective channels for the input filter data 65 based on the auxiliary pictures may differ between the one or more auxiliary pictures.

[0416] Thus, according to an embodiment, apparatus 20 derives, from data stream 14, an indication, e.g., a syntax element, nnpfc_aux_layers_same_order_idc_flag, which indicates whether a manner of deriving the second channels based on the set of auxiliary pictures is selected individually for each of the auxiliary pictures. If the indication indicates an individual selection for each of the auxiliary pictures, apparatus 20 derives from data steam 14, for each of the auxiliary pictures, the indication, which indicates the manner of deriving the filter input data 65 based on the respective auxiliary picture, or, alternatively, derives, from the data stream 14, for each of the auxiliary pictures an indication, which indicates the manner of deriving one or more of the second channels 69 based on the respective auxiliary picture.

[0417] According to an embodiment, if the indication, which indicates whether the manner of deriving the second channel based on the set of auxiliary pictures is selected individually for each of the auxiliary pictures indicates no individual selection, apparatus 20 derives the manner of deriving the second channels from an indication in the data stream which indicates the manner of deriving the first channels based on the reconstructed picture. In other words, in case that the manner of deriving the second channels is not signaled individually, a common indication for deriving the channels of the input filter data, which defines the manner of deriving the first input channels and the second input channels, may be used, e.g., the syntax element nnpfc_inp_order_idc, as described above.

[0418] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmAn exemplary implementation, which allows for a differentiation between a common mode for deriving the first channels and the second channels, and the signalization of individual modes for deriving the second channels, is described in the following.

[0419] It also could include indication of whether the number of components (or chroma format) is the same for each of the layers. An instantiation is shown in the following:

[0420] & >

[0421] & >

[0422] <

[0423]

[0424] nnpfc_aux_layers_same_order_idc_flag equal to 1 indicates that the method of ordering the sample array of any auxiliary input picture to form an input tensor to the NNPF is the same as for the input pictures and that the syntax element nnpfc_aux_inp_order_idc[ i ] is not present. nnpfc_aux_layers_same_order_idc_flag equal to 0 indicates the syntax element nnpfc_aux_inp_order_idc[ i ] is present.

[0425] nnpfc_aux_inp_order_idc[ i ] indicates the method of ordering the sample arrays of an i-th auxiliary input picture to form an input tensor to the NNPF. When not present, and nnpfc_aux_layers_same_order_idc_flag equal to 1 the value of nnpfc_aux_inp_order_idc[ i ] is inferred to be equal to nnpfc inp order idc for each value of i in the range of 0 to nnpfc num aux layers minus 1 , inclusive .

[0426] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmThe process of generating the input tensors would be needed to be change as follows: The process DeriveInputTensors( ), for deriving the input tensor inputTensor for a given vertical sample coordinate cTop and a horizontal sample coordinate cLeft specifying the top-left sample location for the patch of samples included in the input tensor, is specified as follows:

[0427] for( i = 0; i < numlnputPics; i++ ) {

[0428] inpOrderNumComp=0

[0429] if( nnpfc inp order idc = = 0 )

[0430] inpOrderNumComp= 1

[0431] for( yP = -nnpfc overlap; yP < inpPatchHeight + nnpfc overlap; yP++)

[0432] for( xP = -nnpfc overlap; xP < inpPatchWidth + nnpfc o verlap; xP++ ) { inpVal = InpY( InpSampleVal( cTop + yP, cLeft + xP, CroppedHeight, CroppedWidth, CroppedYPic[ i ], 0 ) )

[0433] yPovlp = yP + nnpfc overlap

[0434] xPovlp = xP + nnpfc overlap

[0435] if( !nnpfc_component_last_flag )

[0436] inputTensor

[0000] [ i ]

[0000] [ yPovlp ][ xPovlp ] = inpVal

[0437] else

[0438] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ]

[0000] = inpVal

[0439]

[0440] inputTensor

[0000] [ i ]

[0001] [ yPovlp ][ xPovlp ] = strengthControlScaledVal[ i ] else

[0441] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ]

[0001] = strengthControlScaledVal[ i ] if( nnpfc_auxiliary_inp_idc = = 2 | | nnpfc_auxiliary_inp_idc = = 3) { promptCharVal = utf8ToUInt( nnpfcPrompt )

[0442] if( !nnpfc_component_last_flag )

[0443] inputTensor

[0000] [ i ] [ nnpfc auxiliary inp idc - 1 ] [ yPovlp ] [ xPovlp ] = promptCharVal

[0444] else

[0445] inputTensor

[0000] [ i ] [ yPovlp ] [ xPovlp ] [ nnpfc auxiliary inp idc - 1 ] = promptCharVal

[0446] }

[0447] }

[0448] else if( nnpfc inp order idc = = 1 )

[0449] (96)

[0450] inpOrderNumComp=2

[0451] for( yP = -nnpfc overlap; yP < inpPatchHeight + nnpfc overlap; yP++)

[0452] for( xP = -nnpfc overlap; xP < inpPatchWidth + nnpfc o verlap; xP++ ) { inpCbVal = InpC( InpSampleVal( cTop + yP, cLeft + xP, CroppedHeight / SubHeightC,

[0453] CroppedWidth / SubWidthC, CroppedCbPic[ i ], 1 ) )

[0454] inpCrVal = InpC( InpSampleVal( cTop + yP, cLeft + xP, CroppedHeight / SubHeightC,

[0455] CroppedWidth / SubWidthC, CroppedCrPic[ i ], 2 ) )

[0456] yPovlp = yP + nnpfc overlap

[0457] xPovlp = xP + nnpfc overlap

[0458] if( !nnpfc_component_last_flag ) {

[0459] inputTensor

[0000] [ i ]

[0000] [ yPovlp ][ xPovlp ] = inpCbVal

[0460] inputTensor

[0000] [ i ]

[0001] [ yPovlp ][ xPovlp ] = inpCrVal

[0461] } else {

[0462] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ]

[0000] = inpCbVal

[0463] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ]

[0001] = inpCrVal

[0464] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bm

[0465]

[0466] inputTensor

[0000] [ i ]

[0002] [ yPovlp ][ xPovlp ] = strengthControlScaledVal[ i ] else

[0467] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ]

[0002] = strengthControlScaledVal[ i ] if( nnpfc_auxiliary_inp_idc = = 2 | | nnpfc_auxiliary_inp_idc = = 3) { promptCharVal = utf8ToUInt( nnpfcPrompt )

[0468] if( !nnpfc_component_last_flag )

[0469] inputTensor

[0000] [ i ] [ nnpfc auxiliary inp idc ] [ yPovlp ] [ xPovlp ] = promptCharVal

[0470] else

[0471] inputTensor

[0000] [ i ] [ yPovlp ] [ xPovlp ] [ nnpfc auxiliary inp idc ] = promptCharVal

[0472] }

[0473] }

[0474] else if( nnpfc_inp_order_idc = = 2 )

[0475] inpOrderNumComp=3

[0476] for( yP = -nnpfc overlap; yP < inpPatchHeight + nnpfc overlap; yP++) for( xP = -nnpfc overlap; xP < inpPatchWidth + nnpfc o verlap; xP++ ) { yY = cTop + yP

[0477] xY = cLeft + xP

[0478] yC = yY / SubHeightC

[0479] xC = xY / SubWidthC

[0480] inpYVal = InpY( InpSampleVal( yY, xY, CroppedHeight, CroppedWidth, CroppedYPic[ i ], 0 ) )

[0481] inpCbVal = InpC( InpSampleVal( yC, xC, CroppedHeight / SubHeightC, CroppedWidth / SubWidthC, CroppedCbPic[ i ], 1 ) )

[0482] inpCrVal = InpC( InpSampleVal( yC, xC, CroppedHeight / SubHeightC, CroppedWidth / SubWidthC, CroppedCrPic[ i ], 2 ) )

[0483] yPovlp = yP + nnpfc overlap

[0484] xPovlp = xP + nnpfc overlap

[0485] if( !nnpfc_component_last_flag ) {

[0486] inputTensor

[0000] [ i ]

[0000] [ yPovlp ][ xPovlp ] = inpYVal

[0487] inputTensor

[0000] [ i ]

[0001] [ yPovlp ][ xPovlp ] = inpCbVal

[0488] inputTensor

[0000] [ i ]

[0002] [ yPovlp ][ xPovlp ] = inpCrVal

[0489] } else {

[0490] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ]

[0000] = inpYVal

[0491] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ]

[0001] = inpCbVal

[0492] inputTensor

[0000] [ i ] [ yPovlp ] [ xPovlp ]

[0002] = inpCrVal

[0493]

[0494] promptCharVal = utf8ToUInt( nnpfcPrompt )

[0495] if( !nnpfc_component_last_flag )

[0496] inputTensor

[0000] [ i ] [ nnpfc auxiliary inp idc + 1 ] [ yPovlp ] [ xPovlp ] = promptCharVal

[0497] else

[0498] inputTensor

[0000] [ i ] [ yPovlp ] [ xPovlp ] [ nnpfc auxiliary inp idc + 1 ] = promptCharVal

[0499] }

[0500] }

[0501] else if( nnpfc_inp_order_idc = = 3 )

[0502] inpOrderNumComp=6

[0503] for( yP = -nnpfc overlap; yP < inpPatchHeight + nnpfc overlap; yP++)

[0504] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmfor( xP = -nnpfc overlap; xP < inpPatchWidth + nnpfc o verlap; xP++ ) { yTL = cTop

[0505] xTL = cLeft

[0506] yBR = yTL

[0507] xBR = xTL

[0508] yC = cTop /

[0509] xC = cLeft /

[0510]

[0511] inpTLVal = InpY( InpSampleVal( yTL, xTL, CroppedHeight, CroppedWidth, CroppedYPic[ i ], 0 ) )

[0512] inpTRVal = InpY( InpSampleVal( yTL, xBR, CroppedHeight, CroppedWidth, CroppedYPic[ i ], 0 ) )

[0513] inpBLVal = InpY( InpSampleVal( yBR, xTL, CroppedHeight, CroppedWidth, CroppedYPic[ i ], 0 ) )

[0514] inpBRVal = InpY( InpSampleVal( yBR, xBR, CroppedHeight, CroppedWidth, CroppedYPic[ i ], 0 ) )

[0515] inpCbVal = InpC( InpSampleVal( yC, xC, CroppedHeight / 2, CroppedWidth / 2, CroppedCbPic[ i ], 1 ) )

[0516] inpCrVal = InpC( InpSampleVal( yC, xC, CroppedHeight / 2, CroppedWidth / 2, CroppedCrPic[ i ], 2 ) )

[0517] yPovlp = yP + nnpfc overlap

[0518] xPovlp = xP + nnpfc overlap

[0519] if( !nnpfc_component_last_flag ) {

[0520] inputTensor

[0000] [ i ]

[0000] [ yPovlp ][ xPovlp ] = inpTLVal

[0521] inputTensor

[0000] [ i ]

[0001] [ yPovlp ][ xPovlp ] = inpTRVal

[0522] inputTensor

[0000] [ i ]

[0002] [ yPovlp ][ xPovlp ] = inpBLVal

[0523] inputTensor

[0000] [ i ]

[0003] [ yPovlp ][ xPovlp ] = inpBRVal

[0524] inputTensor

[0000] [ i ]

[0004] [ yPovlp ][ xPovlp ] = inpCbVal

[0525] inputTensor

[0000] [ i ]

[0005] [ yPovlp ][ xPovlp ] = inpCrVal

[0526] } else {

[0527] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ]

[0000] = inpTLVal

[0528] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ]

[0001] = inpTRVal

[0529] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ]

[0002] = inpBLVal

[0530] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ]

[0003] = inpBRVal

[0531] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ]

[0004] = inpCbVal

[0532] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ]

[0005] = inpCrVal

[0533]

[0534] inputTensor

[0000] [ i ]

[0006] [ yPovlp ][ xPovlp ] = strengthControlScaledVal[ i ] else

[0535] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ]

[0006] = strengthControlScaledVal[ i ] if( nnpfc_auxiliary_inp_idc = = 2 | | nnpfc_auxiliary_inp_idc = = 3) { promptCharVal = utf8ToUInt( nnpfcPrompt )

[0536] if( !nnpfc_component_last_flag )

[0537] inputTensor

[0000] [ i ] [ nnpfc auxiliary inp idc + 4 ] [ yPovlp ] [ xPovlp ] = promptCharVal

[0538] else

[0539] inputTensor

[0000] [ i ] [ yPovlp ] [ xPovlp ] [ nnpfc auxiliary inp idc + 4 ] = promptCharVal

[0540] }

[0541] }

[0542] if( nnpfc auxiliary inp idc = = 4) {

[0543] aux!npOrderNumComp=0

[0544] for(j=0;j <= nnpfc_num_aux_layers_minus 1 ; j++) {

[0545] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmAuxldx= inpOrderNumComp + auxInpOrderNumComp

[0546] if( nnpfc_aux_inp_order_idc[ i ] = = 0 )

[0547] for( yP = -nnpfc overlap; yP < inpPatchHeight + nnpfc overlap; yP++) for( xP = -nnpfc overlap; xP < inpPatchWidth + nnpfc o verlap; xP++ ) { inpAuxVal = InpY( InpSampleVal( cTop + yP, cLeft + xP, CroppedHeight, CroppedWidth, CroppedYAuxPic[ i ][ j ], 0 ) )

[0548] yPovlp = yP + nnpfc overlap

[0549] xPovlp = xP + nnpfc overlap

[0550] if( !nnpfc_component_last_flag )

[0551] inputTensor

[0000] [ i ] [ Auxldx] [ yPovlp ][ xPovlp ] = inpAuxVal

[0552] else

[0553] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ][ Auxldx ] = inpAuxVal

[0554] }

[0555] auxInpOrderNumComp + = 1

[0556] else if( nnpfc_aux_inp_order_idc[ i ] = = 1 )

[0557] (96)

[0558] for( yP = -nnpfc overlap; yP < inpPatchHeight + nnpfc o verlap; yP++) for( xP = -nnpfc overlap; xP < inpPatchWidth + nnpfc o verlap; xP++ ) { inpAuxCbVal = InpC( InpSampleVal( cTop + yP, cLeft + xP, CroppedHeight / SubHeightC, CroppedWidth / SubWidthC, CroppedCbAuxPic[ i ][ j ], 1 ) ) inpAuxCrVal = InpC( InpSampleVal( cTop + yP, cLeft + xP, CroppedHeight / SubHeightC, CroppedWidth / SubWidthC, CroppedCrAuxPic[ i ][ j ], 2 ) ) yPovlp = yP + nnpfc overlap

[0559] xPovlp = xP + nnpfc overlap

[0560] if( !nnpfc_component_last_flag ){

[0561] inputTensor

[0000] [ i ][ Auxldx ][ yPovlp ][ xPovlp ] = inpAuxCbVal inputTensor

[0000] [ i ][ Auxldx++ ][ yPovlp ][ xPovlp ] = inpAuxCrVal } else {

[0562] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ][ Auxldx ] = inpAuxCbVal inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ][ Auxldx++ ] = inpAuxCrVal }

[0563] }

[0564] auxInpOrderNumComp + = 2

[0565] else if( nnpfc_aux_inp_order_idc[ i ] = = 2 )

[0566] for( yP = -nnpfc overlap; yP < inpPatchHeight + nnpfc overlap; yP++) for( xP = -nnpfc overlap; xP < inpPatchWidth + nnpfc o verlap; xP++ ) { yY = cTop + yP

[0567] xY = cLeft + xP

[0568] yC = yY / SubHeightC

[0569] xC = xY / SubWidthC

[0570] inpAuxYVal = InpY( InpSampleVal( yY, xY, CroppedHeight, CroppedWidth, CroppedYAuxPic[ i ][ j ], 0 ) )

[0571] inpAuxCbVal = InpC( InpSampleVal( yC, xC, CroppedHeight / SubHeightC, CroppedWidth / SubWidthC, CroppedCbAuxPic[ i ][ j ], 1 ) ) inpAuxCrVal = InpC( InpSampleVal( yC, xC, CroppedHeight / SubHeightC, CroppedWidth / SubWidthC, CroppedCrAuxPic[ i ][ j ], 2 ) )

[0572] if( !nnpfc_component_last_flag ){

[0573] inputTensor

[0000] [ i ][ Auxldx ][ yPovlp ][ xPovlp ] = inpAuxYVal inputTensor

[0000] [ i ][ Auxldx++ ][ yPovlp ][ xPovlp ] = inpAuxCbVal inputTensor

[0000] [ i ][ Auxldx++ ][ yPovlp ][ xPovlp ] = inpAuxCrVal } else {

[0574] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ][ Auxldx ] = inpAuxYVal inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ][ Auxldx++ ] = inpAuxCbVal inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ][ Auxldx++ ] = inpAuxCrVal

[0575] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bm}

[0576] }

[0577] auxInpOrderNumComp + = 3

[0578] else if( nnpfc_aux_inp_order_idc[ i ] = = 3 )

[0579] for( yP = -nnpfc overlap; yP < inpPatchHeight + nnpfc overlap; yP++) for( xP = -nnpfc overlap; xP < inpPatchWidth + nnpfc o verlap; xP++ ) { yTL = cTop

[0580] xTL = cLeft

[0581] yBR = yTL

[0582] xBR = xTL

[0583] yC = cTop /

[0584] xC = cLeft /

[0585]

[0586] inpAuxTLVal = InpY( InpSampleVal( yTL, xTL, CroppedHeight, CroppedWidth, CroppedYAuxPic[ i ][ j ], 0 ) )

[0587] inpAuxTRVal = InpY( InpSampleVal( yTL, xBR, CroppedHeight, CroppedWidth, CroppedYAuxPic[ i ][ j ], 0 ) )

[0588] inpAuxBLVal = InpY( InpSampleVal( yBR, xTL, CroppedHeight, CroppedWidth, CroppedYAuxPic[ i ][ j ], 0 ) )

[0589] inpAuxBRVal = InpY( InpSampleVal( yBR, xBR, CroppedHeight, CroppedWidth, CroppedYAuxPic[ i ][ j ], 0 ) )

[0590] inpAuxCbVal = InpC( InpSampleVal( yC, xC, CroppedHeight / 2, CroppedWidth / 2, CroppedCbAuxPic[ i ][ j ], 1 ) ) inpAuxCrVal = InpC( InpSampleVal( yC, xC, CroppedHeight / 2, CroppedWidth / 2, CroppedCrAuxPic[ i ][ j ], 2 ) ) if( !nnpfc_component_last_flag ){

[0591] inputTensor

[0000] [ i ][ Auxldx ][ yPovlp ][ xPovlp ] = inpAuxTLVal inputTensor

[0000] [ i ][ Auxldx++ ][ yPovlp ][ xPovlp ] = inpAuxTRVal inputTensor

[0000] [ i ][ Auxldx++ ][ yPovlp ][ xPovlp ] = inpAuxBLVal inputTensor

[0000] [ i ][ Auxldx++ ][ yPovlp ][ xPovlp ] = inpAuxBRVal inputTensor

[0000] [ i ][ Auxldx++ ][ yPovlp ][ xPovlp ] = inpAuxCbVal inputTensor

[0000] [ i ][ Auxldx++ ][ yPovlp ][ xPovlp ] = inpAuxCrVal } else {

[0592] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ][ Auxldx ] = inpAuxTLVal inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ][ Auxldx++ ] = inpAuxTRVal inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ][ Auxldx++ ] = inpAuxBLVal inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ][ Auxldx++ ] = inpAuxBRVal inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ][ Auxldx++ ] = inpAuxCbVal inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ][ Auxldx++ ] = inpAuxCrVal }

[0593] }

[0594] auxInpOrderNumComp + = 6

[0595] }

[0596] }

[0597] utf8ToUInt( x ) {

[0598] result = 0

[0599] len = 0

[0600] / * Check end of text prompt string * /

[0601] if( x = = null )

[0602] return 0

[0603] / * Determine the number of bytes in the UTF-8 character * / if( (x

[0000] & 0x80 ) = = 0 )

[0604] len = 1 / * 1-byte character * /

[0605] else if( (x

[0000] & OxEO ) = = OxCO )

[0606] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmlen = 2 / * 2-byte character * /

[0607] else if( (x

[0000] & OxFO ) = = OxEO )

[0608] len = 3 / * 3 -byte character * /

[0609] else if( (x

[0000] & 0xF8 ) = = OxFO )

[0610] len = 4 / * 4-byte character * /

[0611] else

[0612] len = 0 / * Invalid UTF-8 character; this case shall not occur in bitstreams. * /

[0613] for( i = 0; i < len; i++ ) / * Construct an integer from the bytes * /

[0614] result = ( result « 8 ) | x[ i ]

[0615] x = x + len / * Modifies the input variable, which is a syntax element * / return result

[0616] The algorithm above identifies the channel index for each auxiliary pictures (Auxldx) based on the value of nnpfc_aux_inp_order_idc for the input pictures of the current layer and the value of each previous nnpfc_aux_inp_order_idc[ i ], if any. This allow the number of channels to be different per picture in different layers.

[0617] The input of the auxiliary pictures would be defined as follows for the case with the chroma format being the same for the pictures and auxiliary pictures:

[0618] Let currPic be the cropped decoded output picture for which the neural -network post-processing filter (NNPF) defined by the neural-network post-filter characteristics (NNPFC) SEI message is activated by a neural-network post-filter activation (NNPFA) SEI message and currLayerld be the nuh layer id value of currPic.

[0619] When nnpfc auxiliary inp idc is equal to 4, let currAuxPic[ Layerldx ] be the cropped decoded output picture of layer with nuh layer id equal to nnpfc layer target idf Layerldx ] with Layerldx in the range of 0 to nnpfc_num_aux_layers_minusl, inclusive, in the same AU as currentPic.

[0620] For each value of j in the range of 0 to numinferences - 1, inclusive, the following applies:

[0621] The variable numCandlnputPics, which indicates the number of candidate input pictures to the NNPF, is derived as follows:

[0622] numCandlnputPics = numlnputPics + nnpfa_num_input_pic_shift (xx) The arrays candlnputPicf i ] and candlnputPresentFlagf i ] for i in the range of 0 to numCandlnputPics - 1, inclusive, representing all the candidate input pictures and the presence of candidate input pictures, respectively, and, when nnpfc auxiliary inp idc is equal to 4 , the arrays candAuxInputPic[ i ][ m ] and candAuxInputPresentFlag[ i ][ m ] for i in the range of 0 to numCandlnputPics - 1 , inclusive, and m in the range of 0 to nnpfc num aux layers minus 1 , inclusive, representing all the auxiliary candidate input pictures and the presence of auxiliary candidate input pictures, respectively, are specified as follows:

[0623] When j is greater than 0, for each value of k in the range of 0 to j - 1, inclusive, candlnputPicf k ] is set to be currPic and candlnputPresentFlagf k ] is set equal to 0 and when nnpfc auxiliary inp idc is equal to 4 , for each value of m in the range of 0 to

[0624] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmnnpfc num aux layers minusl, candAuxInputPic[ k ][ m ] is set to be currAuxPic[ m ] and candAux!nputPresentFlag[ k ][ m ] is set equal to 0.

[0625] The j-th candidate input picture, candlnputPicf j ], is set to be currPic and candlnputPresentFlagf j ] is set equal to 1 and when nnpfc auxiliary inp idc is equal to 4 , for each value of m in the range of 0 to nnpfc num aux layers minusl, each auxiliary candidate input picture candAux!nputPic[ j ][ m ] is set to be currAuxPic[ m ] and candAux!nputPresentFlag[ k ][ m ] is set equal to 1.

[0626] When numCandlnputPics is greater than 1, the following applies for each value of i in the range of j + 1 to numCandlnputPics - 1, inclusive, in increasing order of i:

[0627] If both of the following conditions are true, candlnputPicf i ] is set to be prevPic and candlnputPresentFlagf i ] is set equal to 1 and when nnpfc auxiliary inp idc is equal to 4 , for each value of m in the range of 0 to nnpfc num aux layers minusl, each auxiliary candidate input picture candAuxInputPicf j ][ m ] is set to be prevAuxPicf m ] and candlnputAuxPresentFlagf k ][ m ] is set equal to 1, prevAuxPicf m ] being the cropped decoded output picture that has nuh layer id equal to nnpfc layer target idf m ], precede candAuxInputPicf i - 1 ][ m ] in output order:

[0628] Either of the following conditions is true:

[0629] pictureRateUpsamplingFlag is equal to 1 and currPic is associated with a frame packing arrangement SEI message with frame_packing_arrangement_type equal to 5 and a particular value of fp current frame is frameO flag, and there is a cropped decoded output picture prevPic that is the last picture in output order among all cropped decoded output pictures that have nuh layer id equal to currLayerld, precede candlnputPicf i - 1 ] in output order, and are associated with a frame packing arrangement SEI message with frame_packing_arrangement_type equal to 5 and the same value of fp_current_frame_is_frameO_flag. pictureRateUpsamplingFlag is equal to 0 or currPic is not associated with a frame packing arrangement SEI message with frame_packing_arrangement_type equal to 5, and there is a cropped decoded output picture prevPic that is the last picture in output order among all cropped decoded output pictures that have nuh layer id equal to currLayerld and precede candlnputPicf i - 1 ] in output order. nnpfa_no_prev_clvs_flag is equal to 0 or the coded picture corresponding to prevPic and the current picture are present in the same CLVS.

[0630] Otherwise, the following applies:

[0631] candlnputPicf i ] is set to be the same picture as candlnputPicf i - 1 ] and candlnputPresentFlagf i ] is set equal to 0 and when nnpfc auxiliary inp idc is equal to 4 , for each value of m in the range of 0 to nnpfc num aux layers minusl, candAuxInputPicf j ][ m ] is set to be the same as candAuxInputPicf j - 1 ][ m and candAuxInputPresentFlagf j ][ m ] is set equal to 0.

[0632] - It is a requirement of bitstream conformance that, when pictureRateUpsamplingFlag is equal to 1, nnpfc_interpolated_pics[ i - 1 ] shall be equal to 0.

[0633] The arrays inputPicf i ] and inputPresentFlagf i ] for i in the range of 0 to numlnputPics - 1, inclusive, representing all the input pictures and the presence of input pictures, respectively, are specified as follows:

[0634] for( i = 0, candldx = nnpfa_num_input_pic_shift; i <= nnpfc_num_input_pics_minus 1 ; i++, candldx++ ) {

[0635] inputPicf i ] = candlnputPicf candldx ] (xx) inputPresentFlagf i ] = candlnputPresentFlagf candldx ]

[0636] }

[0637] When nnpfc auxiliary inp idc is equal to 4 ,the arrays inputAuxPicf i ][ j ] and inputPresentFlagf i ][ j ] for i in the range of 0 to numlnputPics - 1, inclusive, and for j in the range of 0 to nnpfc num aux layers minusl, inclusive, representing all the input auxiliary pictures and the presence of input auxiliary pictures, respectively, are specified as follows:

[0638] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmfor( i = 0, candldx = nnpfa_num_input_pic_shift; i <= nnpfc_num_input_pics_minus 1 ; i++, candldx++ ) {

[0639] for(j = 0; j <= nnpfc_num_aux_layers_minusl; j++ ){

[0640] inputAuxPic[ i ][ j ] = candAuxInputPic[ candldx ][j ]

[0641] (xx)

[0642] inputAuxPresentFlag[ i ][ j ] = candAuxInputPresentFlag[ candldx ][ j ]

[0643] }

[0644] }

[0645] For each value of i in the range 0 to numlnputPics - 1, inclusive, it is a requirement of bitstream conformance that when inputPresentFlag[ i ] is equal to 0 and nnpfc_input_pic_output_flag[ i ] is equal to 1, the value of nnpfa_output_flag[ idx ] shall be equal to 0 for the value of idx such that Inpldx[ idx ] is equal to i.

[0646] For purposes of interpretation of the NNPFC SEI message, the following variables are specified:

[0647] CroppedWidth is set equal to the value of pps_pic_width_in_luma_samples - SubWidthC * ( pps_conf_win_left_offset + pps_conf_win_right_offset ) for currPic. CroppedHeight is set equal to the value of pps_pic_height_in_luma_samples - SubHeightC * ( pps conf win top offset + pps conf win bottom offset ) for currPic. The luma sample arrays CroppedYPic[ i ] and the chroma sample arrays CroppedCbPic[ i ] and CroppedCrPic[ i ], when present, are derived as follows for each value of i in the range of 0 to numlnputPics - 1, inclusive:

[0648] The variable source Pic is derived as follows:

[0649] If inputPresentFlag[ i ] is equal to 1 or nnpfc_absent_input_pic_zero_flag is equal to 0, source Pic is set to be inputPic[ i ] .

[0650] Otherwise(inputPresentFlag[ i ] is equal to 0 and nnpfc_absent_input_pic_zero_flag is equal to 1), sourcePic is set to be a picture with a luma sample array of CroppedWidth x CroppedHeight samples equal to 0 and Cb and Cr sample arrays of ( CroppedWidth / SubWidthC ) x ( CroppedHeight / SubHeightC ) samples equal to 0.

[0651] The luma sample array CroppedYPic[ i ] and the chroma sample arrays CroppedCbPic[ i ] and CroppedCrPic[ i ], when present, are set to be the 2-dimensional arrays of decoded sample values of the Y, Cb and Cr components, respectively, of sourcePic.

[0652] When nnpfc auxiliary inp idc is equal to 4, the luma sample arrays CroppedYAuxPic[ i ][ j ] and the chroma sample arrays CroppedCbPic[ i ][ j ] and CroppedCrAuxPic[ i ][ j ], when present, are derived as follows for each value of i in the range of 0 to numlnputPics - 1, inclusive, and for j in the range of 0 to nnpfc num aux layers minus 1 , inclusive :

[0653] The variable sourceAuxPic is derived as follows:

[0654] If inputAuxPresentFlag[ i ][ j ] is equal to 1 or nnpfc_absent_input_pic_zero_flag is equal to 0, sourceAuxPic is set to be inputAuxPic[ i ][ j ].

[0655] Otherwise(inputAuxPresentFlag[ i ][ j ] is equal to 0 and nnpfc_absent_input_pic_zero_flag is equal to 1), sourceAuxPic is set to be a picture with a luma sample array of CroppedWidth x CroppedHeight samples equal to 0 and Cb and Cr sample arrays of ( CroppedWidth / SubWidthC ) x ( CroppedHeight / SubHeightC ) samples equal to 0.

[0656] The luma sample array CroppedYAuxPic[ i ][ j ] and the chroma sample arrays CroppedCbAuxPic[ i ][ j ] and CroppedCrAuxPic[ i ][ j ], when present, are set to be the 2-dimensional arrays of decoded sample values of the Y, Cb and Cr components, respectively, of sourceAuxPic.

[0657] BitDepthy and BitDepthc are both set equal to BitDepth.

[0658] ChromaFormatldc is set equal to sps chroma format idc.

[0659] The array StrengthControlVal[ i ] for all values of i in the range of 0 to numlnputPics - 1, inclusive, specifying the filtering strength control value for the input pictures for the NNPF, is derived as follows:

[0660] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmStrengthControl Vai [ i ] is set equal to the value of ( firstSliceQpy + QpBdOffset ) ( 63 + QpBdOffset ), where firstSliceQpy is equal to SliceQpy of the first slice of inputPic[ i ].

[0661] Referring to any of the previous embodiments described with respect to Fig. 2, the indication 64 may indicate, for the respective auxiliary picture, for each of one or more sample array types, such as luma and two chroma types, for example, a number of channels to be derived for the plurality of channels of the input data. For example, apparatus 20 may derive, for each of the one or more sample array types, the respective number of channels based on a sample array of the respective sample array type of the respective auxiliary picture. For example, as described above, apparatus 20 may derive, for each of a luma and two chroma sample arrays of the respective auxiliary picture, one channel of the plurality of input channels of the filter input data 65 or, alternatively, e.g., as in the fourth variant as described above, derive four input channels for the luma sample array and each one channel for the two chroma sample arrays.

[0662] For example, if one of the sample array types does not exist for the respective auxiliary picture, filter input determinator 63 may use a substitute sample array for determining the respective channel, such as a sample array comprising default values.

[0663] According to an embodiment, apparatus 20 derives the order among the channels derived for the respective auxiliary picture in dependence on the indication which indicates the manner of deriving the second channels.

[0664] According to an embodiment, the indication, which indicates the manner of deriving the second channels, e.g., indication 64, different shades between a plurality of modes, which may include one or more or all of the following modes:

[0665] According to a first mode, the filter input determinator 63 derives, for the respective auxiliary picture, one, e.g., exactly one, of the channels comprising, for consisting of, an auxiliary luma sample array, e.g., derived from a luma sample array of the auxiliary picture.

[0666] According to a second mode, the filter input determinator 63 derives, for the respective auxiliary picture, one, e.g., exactly one, of the channels comprising, or consisting of, a first auxiliary chroma sample array, and a further one, e.g., exactly one, of the channels comprising, or consisting of, a second auxiliary chroma sample array.

[0667] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmAccording to a third mode, the filter input determinator 63 derives, for the respective auxiliary picture, one, e.g., exactly one, of the channels comprising, or consisting of, an auxiliary luma sample array, a further one, e.g., exactly one, of the channels comprising, or consisting of, a first auxiliary chroma sample array, and an even further one of the channels comprising, or consisting of, the second auxiliary chroma sample array.

[0668] According to a fourth mode, the filter input determinator 63 determines, for the respective auxiliary picture, four of the channels, each of which comprises, or consists of, an auxiliary luma sample array, a further one, e.g., exactly one, of the channels comprising, or consisting of, a first auxiliary chroma sample array, and an even further one, e.g., exactly one, of the channels comprising, or consisting of, a second auxiliary chroma sample array.

[0669] As already mentioned, the filter input data may be derived based on one or more further pictures belonging to a further time frame of the video. For example, the above described syntax examples may describe the case in which the number of input pictures is greater than 1 , in such case, in the beginning or at the end of a bitstream / CLVS (and using pictures of different CLVSs is not allowed), some pictures may be missing to run the NNPF. This would require to repeat the first and / or last picture of bitstream / CLVS so that the number of required input pictures are provided.

[0670] For each of the auxiliary layers, it is checked whether enough pictures exist in the beginning of the CLVS or bitstream and if they are not, the first or last pictures are repeated if the SEI message indicates so, or otherwise, a picture is generated with a predetermined value (e.g., 0) that is to be provided to the NNPF.

[0671] In more general terms, according to an embodiment, in which the picture belongs to a coded video sequence of a sequence of coded video sequences of the data stream 14, apparatus 20 derives, from data stream 14 an indication of a number, e.g., in terms of a count or quantity, of time frame for obtaining the filter input data for filtering the reconstructed picture 12’. In case that the indication of the number of time frames indicates to obtain the filter input data using, in addition to the time frame to which the picture 12’, i.e., the current picture to be filtered or coded, belongs, a further picture of a further time frame, i.e., a time frame different from the time frame of picture 12’, filter input determinator 63 may determine the filter input data 65 using further auxiliary sample arrays for the further time frame. If the further picture belongs to a preceding coded video sequence preceding the coded video sequence of the picture or if the further picture belongs to a succeeding coded video

[0672] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmsequence succeeding the coded video sequence of the picture, filter input determinator 63 may determine the further auxiliary sample arrays in dependence on an indication signaled in the data stream 14.

[0673] For example, in dependence on the indication signaled in the data stream, filter input determinator 63 may differentiate between using one or more sample arrays of the further time frame as the further auxiliary sample arrays, e.g., using the sample arrays of the further time frame belonging to the same layers as the one or more auxiliary pictures of the time frame of the picture 12’, and using, as the further auxiliary sample arrays, either corresponding ones, e.g., in terms of layer and data type (luma, chroma, depth, etc.) of the auxiliary sample arrays belonging to the first or the last time frame of the coded video sequence, or using a default sample array, e.g., zeros.

[0674] For example, the indication based on which the differentiation between using sample arrays of the further time frame on the one hand and using sample arrays of the time frame or a default sample array on the other hand is performed may be an indication, e.g., a syntax element, which indicates where the pictures of a coded video sequence preceding the coded video sequence to which the picture 12’ belongs, can be used for deriving the filter input data 65. If the indication indicates that pictures of the coded video sequence preceding the coded video sequence to which the picture 12’ belongs cannot be used for deriving the filter input data, the default sample array or the corresponding sample arrays of the time frame of picture 12’ may be used. In other words, in this case, the auxiliary sample arrays of the further time frame are substituted by the corresponding sample arrays of the first or the last time frame of the coded video sequence, or by a default sample array. Same may apply if the further time frame belongs to a succeeding coded video sequence, and in this case, the indication may be an indication which indicates whether pictures of the succeeding coded video sequence may be used for deriving the filter input data 65.

[0675] Further embodiments are concerned with the case in which the auxiliary picture does not include a sample array of each type of sample arrays required by the selected process for deriving the filter input data 65.

[0676] According to an embodiment, in which filter input determinator derives, for each of the one or more auxiliary pictures, for each of one or more sample array types, a number, e.g., in terms of a count, e.g., a predefined count or a count signaled individually for each of the auxiliary pictures, of channels of the plurality of channels, filter input determinator 63 derives

[0677] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmthe number of channels based on the auxiliary sample array if the respective auxiliary picture comprises an auxiliary sample array of the respective sample array type, and otherwise, i.e., if the respective auxiliary picture misses, or does not comprise, an auxiliary sample array of the respective sample array type, filter input determinator derives the number of channels using sample values of a predetermined value. For example, if an auxiliary picture has a chroma format which is different from the chroma format of the reconstructed picture, e.g., if the auxiliary picture has no chroma components, a sample array of sample values of the predetermined value may be used instead of the missing auxiliary sample array.

[0678] According to an embodiment, apparatus 20 derives the predetermined value from the data stream 14.

[0679] In other words, in a further embodiment, when the number of chroma components differs between auxiliary pictures or between auxiliary pictures and input pictures, but the nnpfc_inp_order_idc determines the method of ordering the sample array of input and auxiliary picture to the input of NNPF (there is no nnpfc_aux_inp_order_idc), for auxiliary picture not having some chroma components the following applies:

[0680] The Auxiliary picture has Cb and Cr sample arrays of ( CroppedWidth I SubWidthC ) * ( CroppedHeight I SubHeightC ) samples equal to 0 or to ( 1 « BitDepthYc ) - 1 or to a midgrey value ( 1 « ( BitDepthYc - 1) ) - 1 or to a predetermined value.

[0681] In addition, signaling could be added for the case in which the number of chroma components of the input pictures is different than auxiliary pictures of different layers that indicates whether a zero-sample value is used for non-existing chroma components or a different value. For instance, it could be a nnpfc_aux_default_chroma_sample_mode_idc that indicates which value is used to set the Cb Cr sample arrays.

[0682] Further embodiments are concerned with the case that a sample rate of the picture 12’ of the one or more auxiliary sample arrays are different from each other. For example, in this case, samples of the one of the one or more auxiliary sample arrays may be repeated in order to equalize the sample rates.

[0683] In more general terms, according to an embodiment, in which the reconstructed picture 12’ comprises a luma sample array and two chroma sample arrays, and in which each of the

[0684] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmauxiliary pictures is represented by an auxiliary luma sample array and two auxiliary chroma sample arrays, and in which the filter input determinator 63 determines one or more first channels of the plurality of channels based on the sample arrays of the reconstructed picture 12’ and one or more second channels based on the one or more auxiliary pictures, apparatus 20 derives one of the second channels based on the one of the auxiliary chroma sample arrays by repeating sample values of the auxiliary chroma sample array if the count of sample values of the auxiliary chroma sample array is less than the count of sample values of the chroma sample array of the reconstructed picture, for example if the chroma format of the respective auxiliary picture is different from 4:4:4. Alternatively, apparatus 20 may derive the one of the second channels based on the one of the auxiliary chroma sample arrays by determining a first ratio between the width of the one auxiliary chroma sample array and the width of the sample arrays of the reconstructed picture 12’, and determined a second ration between the height of the one auxiliary chroma sample array and the height of the sample arrays of the reconstructed picture 12’, and repeat sample values of the auxiliary chroma sample array according to the first ratio and the second radio, for example, so that all of the first and second channels have the same count of sample values, e.g., the same dimension.

[0685] According to a further embodiment, in which the reconstructed picture comprises at least a luma sample array, apparatus 20 obtains a second channel of the plurality of channels based on an auxiliary picture, which comprises an auxiliary luma sample array, and according to this embodiment, apparatus 20 derives the second channel based on the auxiliary luma sample array by repeating sample values of the auxiliary luma sample array if the count of sample values of the auxiliary luma sample array is less than the count of sample values of the luma sample array of the reconstructed picture. For example, the auxiliary picture may be a depth picture having a lower resolution. Alternatively, apparatus 20 may derive the second channel based on the auxiliary luma sample array by determining a first ratio between the width of the auxiliary luma sample array and the width of the luma sample array of the reconstructed picture, and determining a second ratio between the height of the auxiliary luma sample array and the height of the luma sample array of the reconstructed picture 12’, and repeating sample values of the auxiliary luma sample array according to the first ratio and the second ratio, for example, so that all of the first and second channels have the same count of sample values, e.g., the same dimension.

[0686] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmIn other words, so far, when chromaFormat is not the same between input pictures and auxiliary pictures, it was only considered the cases about the existence of Cb / Cr components.

[0687] In a further embodiment, when ChromaFormat has different values for each layer and the chroma format of each layer is used to derive the SubHeightAuxC[ i ] and SubWidhtAuxC[ i ] for each of the auxiliary pictures and as such when the chroma format is not 4:4:4 repeat some of the chroma samples in the input tensors for some of the auxiliary pictures.

[0688] else if( nnpfc_aux_inp_order_idc[ i ] = = 2 )

[0689] for( yP = -nnpfc overlap; yP < inpPatchHeight + nnpfc overlap; yP++) for( xP = -nnpfc overlap; xP < inpPatchWidth + nnpfc o verlap; xP++ ) { yY = cTop + y

[0690] xY = cLeft + x

[0691] yC = yY / Sub

[0692] xC = xY / Sub

[0693]

[0694] inpAuxYVal = InpY( InpSampleVal( yY, xY, CroppedHeight, CroppedWidth, CroppedYAuxPic[ i ][ j ], 0 ) )

[0695] inpAuxCbVal = InpC( InpSampleVal( yC, xC, CroppedHeight / SubHeightAuxC| i |,

[0696] CroppedWidth / SubWidhtAuxC| i |, CroppedCbAuxPic[ i ][ j ], 1 ) ) inpAuxCrVal = InpC( InpSampleVal( yC, xC, CroppedHeight / SubHeightAuxC| i |,

[0697] CroppedWidth / SubWidhtAuxC| i |, CroppedCrAuxPic[ i ][ j ], 2 ) ) if( !nnpfc_component_last_flag ){

[0698] inputTensor

[0000] [ i ][ Auxldx ] [ yPovlp ][ xPovlp ] = inpAuxYVal inputTensor

[0000] [ i ][ Auxldx++ ][ yPovlp ][ xPovlp ] = inpAuxCbVal inputTensor

[0000] [ i ][ Auxldx++ ][ yPovlp ][ xPovlp ] = inpAuxCrVal

[0699] } else {

[0700] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ][ Auxldx ] = inpAuxYVal inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ][ Auxldx++ ] = inpAuxCbVal inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ][ Auxldx++ ] = inpAuxCrVal

[0701] }

[0702] }

[0703] auxInpOrderNumComp + = 3

[0704] In the following, embodiments are described which address the case that the sample arrays of the picture 12’ and the auxiliary sample arrays 15 have different bit depth.

[0705] According to an embodiment, apparatus 20 is configured for deriving sample values of the first channels 67 and the second channels 67 at a common bit depth.

[0706] For example, the common bit depth may equal the smallest or the highest bit depth among the bit depths of the sample arrays of the reconstructed picture and the auxiliary sample

[0707] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmarrays. Alternatively, the common bit depth is smaller than the smallest bit depth among the bit depth of the sample arrays of the reconstructed picture and the auxiliary sample arrays. In a further alternative, the common bit depth is higher than the highest bit depth among the bit depths of the sample arrays of the reconstructed picture and the auxiliary sample arrays.

[0708] In other words, in a further embodiment, the bitdepth of the luma and or chroma components of the auxiliary pictures (e.g. of the different layers) has different the values for each layer and the bitdepth is used to compute the value of each of the input of the inputTensor: either by computing a floating point number or an integer number of a TensorBitdepth that is common to all pictures (auxiliary or not) which could have a greater bitdepth, equal or smaller than any bitdepth of the pictures (auxiliary or not).

[0709] According to a further embodiment, apparatus 20 may derive one of the second channels 69 from one of the auxiliary sample arrays by converting, or recalculating, sample values of the one auxiliary sample array to obtain a converted auxiliary sample array and deriving the second channel from the converted auxiliary sample array. For example, the conversation of the sample values may be performed using a function such a heavy side function or a multistep function. For example, converting the sample values may allow deriving the channels at equal bit depths, for example, in cases in which the one auxiliary sample array is a depth map or an alpha channel, in which case the domain of sample values may be substantially different from the domain of sample values in a luma or chroma component sample array.

[0710] According to an embodiment, apparatus 20 derives the manner of converting the sample values, e.g., a function or a mapping table, from the data stream 14, e.g., from an SEI message.

[0711] In a further embodiment, one of the auxiliary pictures is a particular type of picture, e.g., an alpha mask, that requires to modify its value previous to converting them to inputTensor at a particular bitdepth. For instance, this could include one or more thresholds for interpreting which “discrete-value” to interpret before inputting that value to the function lnpY( x ) which convert an integer value x to a floating point or to another integer with a bitdepth equal to TensorBitdepth. For instance, it could have a single threshold so that any value greater or equal to the threshold is set equal to (1 « BitdepthY ) -1 and any value smaller to the threshold equal to 0; or to some other specified values. Also, more than one threshold could be present to be able to assign the values to a larger set of discrete values. In other words,

[0712] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmthere might be some information within the bitstream, e.g., in an SEI message describing the auxiliary picture that indicates a transformation of sample values to another intermediate sample values that is used for input to the function lnpY( x ).

[0713] In the following embodiment, different sizes are prevented in the auxiliary pictures compared to the input pictures. An alternative would be to prevent different sizes if the ratio between the different Picturewidth sizes or PictureHeight sizes are not an integer number, i.e. the size (width or height) of the input picture is an integer multiple of the auxiliary pictures.

[0714] In such a case, two ratios are defined RatioWidth and RatioHeight that allow to do as many repetitions of samples as necessary to generate same channel sizes in the input tensors for all input pictures and auxiliary pictures. An example is shown for the case that only the luma and chroma samples are used for NNPF, but is applicable to other cases:

[0715] else if( nnpfc_aux_inp_order_idc[ i ] = = 2 )

[0716] for( yP = -nnpfc o verlap; yP < inpPatchHeight + nnpfc overlap; yP++) for( xP = -nnpfc overlap; xP < inpPatchWidth + nnpfc o verlap; xP++ ) { yY = cTop + yP

[0717] xY = cLeft + xP

[0718] RatioWidth = CroppedWidth / CroppedAuxWidth [ i ]

[0719] RatioHeight = CroppedHeight / CroppedAuxHeight [ i ]

[0720] yY = yY / RatioWidth

[0721] xY = xY / RatioHeight

[0722] yC = yY / SiibHcightAiixC’l i ]

[0723] xC = xY / SubWidhtAiixC’l i ]

[0724] inpAuxYVal = InpY( InpSampleVal( yY, xY, CroppedHeight / RatioHeight, CroppedWidth / RatioWidth, CroppedYAuxPicf i ][ j ], 0 ) ) inpAuxCbVal = InpC( InpSampleVal( yC, xC, CroppedHeight / RatioHeight / SiibHcightAiixC’l i ],

[0725] CroppedWidth / RatioWidth / SubWidhtAiixC’l i ], CroppedCbAuxPicf i ][ j ], 1 ) )

[0726] inpAuxCrVal = InpC( InpSampleVal( yC, xC, CroppedHeight / RatioHeight / SiibHcightAiixC’l i ],

[0727] CroppedWidth / RatioWidth / SubWidhtAiixC’l i ], CroppedCrAuxPicf i ][ j ], 2 ) )

[0728] if( !nnpfc_component_last_flag ){

[0729] inputTensor

[0000] [ i ][ Auxldx ] [ yPovlp ][ xPovlp ] = inpAuxYVal inputTensor

[0000] [ i ][ Auxldx++ ][ yPovlp ][ xPovlp ] = inpAuxCbVal inputTensor

[0000] [ i ][ Auxldx++ ][ yPovlp ][ xPovlp ] = inpAuxCrVal

[0730] } else {

[0731] inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ][ Auxldx ] = inpAuxYVal inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ][ Auxldx++ ] = inpAuxCbVal inputTensor

[0000] [ i ][ yPovlp ][ xPovlp ][ Auxldx++ ] = inpAuxCrVal

[0732] }

[0733] }

[0734] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmauxInpOrderNumComp + = 3

[0735] Such embodiments, facilitating the derivation of the filter input data 65 with equally sized channels, are described in the following in more general terms.

[0736] According to an embodiment, the one or more auxiliary sample arrays comprise one or more of a depth map, a mask, and a transparency map; and / or the one or more auxiliary sample arrays comprise one or more sample arrays of a further picture. According to this embodiment, the data stream 14 has encoded thereinto a multi-view video, wherein the picture belongs to a first view of the multi-view video, and wherein the further picture belongs to a second view of the multi-view video, or the further picture has a different resolution than the picture.

[0737] According to an embodiment, sample arrays of the reconstructed picture 12’ and the one or more auxiliary sample arrays 15 have equivalent sizes.

[0738] According to an embodiment, the reconstructed picture and any, e.g., all, auxiliary pictures, e.g., all color pictures, being represented by the one or more auxiliary sample arrays have the same chroma format.

[0739] According to an embodiment, the reconstructed picture and any, e.g., all, auxiliary pictures, e.g., all color pictures, are represented by the one or more auxiliary sample array have the same bit depth of luma and / or chroma.

[0740] In yet another embodiment, another value of nnpfc_auxiliary_inp_idc (e.g. value 5) can be assigned to additionally to the auxiliary pictures provide further information such as the quantization parameter used to encode each of the auxiliary pictures or some kind of quality indicator. Thereby, further channels could be provided in combination to the auxiliary pictures containing further information. For instance, as an alternative to modifying the sample values for alpha masks based on some thresholds, such thresholds could be passed as further channels to the NN PF. Besides, when it comes to depth maps information related to the depth values could be provided as a further channel, as for instance, the near and far (nearest depth value and farthest depth value) and whether the mapping is linear, inverse, some kind of step-wise linear approximation of a non-linear function and so on. Similarly, the same could be done for disparity maps where the max and min disparity values are provided in a liner, inverse, or non-linear mode.

[0741] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmIn another embodiment, at least one of the following attributes of a picture and an auxiliary picture (or layer they belong to) is equal:

[0742] Picture size

[0743] Chroma format

[0744] CLVS start and end

[0745] Bit depth of luma or chroma

[0746] In more general terms, according to an embodiment, apparatus 20 derives the filter input data 65 by including into the filter input data, e.g., as additional channel, e.g., a third channel, further to the first and second channels, one or more or all of

[0747] an information on quantization parameter used in reconstructing the picture, and / or

[0748] one or more threshold values for an alpha mask;

[0749] a smallest and / or a highest depth value for a depth map;

[0750] an indication of a mapping function for sample values of a depth map; an indication of a mapping function for sample values of a disparity map.

[0751] Fig. 4 illustrates an apparatus 10 for encoding a video into a data stream 14 according to an embodiment. Apparatus 10 may be referred to as encoder. Apparatus 10 comprises an encoder 81 configured for encoding a picture 12 of the video into the data stream 14. Encoding module 81 is further configured for inserting, into the data stream 14, an indication 82 of one or more auxiliary sample arrays 15, the one or more auxiliary sample array 15 deemed to be used for obtaining filter input data for filtering a reconstructed version of the picture. The filter input data is to be obtained based on the reconstructed picture and based on the one or more auxiliary sample arrays 15, and the filter input data is to be subjected to a neural network of the filtering. The picture 12 and the one or more auxiliary sample arrays 15 belong to the same time frame of the video.

[0752] For example, the indication 82 may be inserted into a syntax structure, e.g., an SEI NAL unit of the data stream 14, which is interspersed between coded video data payload packets, e.g., VCL NAL units of data stream 14.

[0753] Any of the features and details described with respect to decoder 20 may optionally also apply to encoder 10, which may be the encoder side equivalent of decoder 20. In other

[0754] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmwords, any of the indications derived by decoder 20 from data stream 14 may be inserted into a data stream by encoder 10.

[0755] It is noted that the block diagrams of Fig. 1 and Fig. 4 may alternatively be considered as flow diagrams of respective methods, in which each of the blocks represents a step of the respective method. Thus, what is further disclosed in the above description is:

[0756] A method for reconstructing a video from a data stream (14), the method comprising: reconstructing (21) a picture (12) of the video; and filtering (61) the reconstructed picture (12’) using a neural network by obtaining (63) filter input data (65) based on the reconstructed picture (12’) and based on one or more auxiliary sample arrays (15), and subjecting the filter input data (65) to the neural network; wherein the picture and the one or more auxiliary sample arrays belong to the same time frame (11*) of the video.

[0757] A method for encoding a video (9) into a data stream (14), the method comprising: encoding (81) a picture (12) of the video into the data stream; and inserting (81), into the data stream, an indication (82) of one or more auxiliary sample arrays (15), the one or more auxiliary sample arrays being to be used for obtaining filter input data for filtering a reconstructed version of the picture, which filter input data is to be obtained based on the reconstructed picture and based on the one or more auxiliary sample arrays, and which filter input data is to be subjected to the neural network; wherein the picture (12) and the one or more auxiliary sample arrays 15) belong to the same time frame of the video.

[0758] Video coding schemes

[0759] The following description presents a description of an encoder and a decoder of a blockbased predictive codec for coding pictures of a video in order to form an example for a coding framework into which embodiments of the present invention may be built in. The respective encoder and decoder are described with respect to Fig. 5, Fig. 6, and Fig. 7. Thereinafter the description of embodiments of the concept of the present invention is presented along with a description as to how such concepts could be built into the encoder and decoder of Fig. 5, and Fig. 6, respectively, although the embodiments described abvoe, may also be used to form encoders and decoders not operating according to the coding framework underlying the encoder and decoder of Fig. 5, and Fig. 6, and Fig. 7.

[0760] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmFig. 5 shows an apparatus for predictively coding a picture 12 into a data stream 14 exemplarily using transform-based residual coding. The apparatus, or encoder, is indicated using reference sign 10. Fig. 6 shows a corresponding decoder 20, i.e. an apparatus 20 configured to predictively decode the picture 12’ from the data stream 14 also using transform-based residual decoding, wherein the apostrophe has been used to indicate that the picture 12’ as reconstructed by the decoder 20 deviates from picture 12 originally encoded by apparatus 10 in terms of coding loss introduced by a quantization of the prediction residual signal. Fig. 5 and Fig. 6 exemplarily use transform based prediction residual coding, although embodiments of the present application are not restricted to this kind of prediction residual coding. This is true for other details described with respect to Fig.

[0761] 5, and Fig. 6, too, as will be outlined hereinafter.

[0762] The encoder 10 is configured to subject the prediction residual signal to spatial-to-spectral transformation and to encode the prediction residual signal, thus obtained, into the data stream 14. Likewise, the decoder 20 is configured to decode the prediction residual signal from the data stream 14 and subject the prediction residual signal thus obtained to spectral-to-spatial transformation.

[0763] Internally, the encoder 10 may comprise a prediction residual signal former 22 which generates a prediction residual 24 so as to measure a deviation of a prediction signal 26 from the original signal, i.e. from the picture 12. The prediction residual signal former 22 may, for instance, be a subtractor which subtracts the prediction signal from the original signal, i.e. from the picture 12. The encoder 10 then further comprises a transformer 28 which subjects the prediction residual signal 24 to a spatial-to-spectral transformation to obtain a spectral-domain prediction residual signal 24’ which is then subject to quantization by a quantizer 32, also comprised by the encoder 10. The thus quantized prediction residual signal 24” is coded into bitstream 14. To this end, encoder 10 may optionally comprise an entropy coder 34 which entropy codes the prediction residual signal as transformed and quantized into data stream 14. The prediction signal 26 is generated by a prediction stage 36 of encoder 10 on the basis of the prediction residual signal 24” encoded into, and decodable from, data stream 14. To this end, the prediction stage 36 may internally, as is shown in Fig. 5, comprise a dequantizer 38 which dequantizes prediction residual signal 24” so as to gain spectral-domain prediction residual signal 24’”, which corresponds to signal 24’ except for quantization loss, followed by an inverse transformer 40 which subjects the latter prediction residual signal 24’” to an inverse transformation, i.e. a spectral-to-spatial transformation, to obtain prediction residual signal 24””, which corresponds to the

[0764] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmoriginal prediction residual signal 24 except for quantization loss. A combiner 42 of the prediction stage 36 then recombines, such as by addition, the prediction signal 26 and the prediction residual signal 24”” so as to obtain a reconstructed signal 46, i.e. a reconstruction of the original signal 12. Reconstructed signal 46 may correspond to signal 12’. A prediction module 44 of prediction stage 36 then generates the prediction signal 26 on the basis of signal 46 by using, for instance, spatial prediction, i.e. intra-picture prediction, and / or temporal prediction, i.e. inter-picture prediction.

[0765] Likewise, decoder 20, as shown in Fig. 6, may be internally composed of components corresponding to, and interconnected in a manner corresponding to, prediction stage 36. In particular, entropy decoder 50 of decoder 20 may entropy decode the quantized spectral-domain prediction residual signal 24” from the data stream, whereupon dequantizer 52, inverse transformer 54, combiner 56 and prediction module 58, interconnected and cooperating in the manner described above with respect to the modules of prediction stage 36, recover the reconstructed signal on the basis of prediction residual signal 24” so that, as shown in Fig. 6, the output of combiner 56 results in the reconstructed signal, namely picture 12’.

[0766] Although not specifically described above, it is readily clear that the encoder 10 may set some coding parameters including, for instance, prediction modes, motion parameters and the like, according to some optimization scheme such as, for instance, in a manner optimizing some rate and distortion related criterion, i.e. coding cost. For example, encoder 10 and decoder 20 and the corresponding modules 44, 58, respectively, may support different prediction modes such as intra-coding modes and inter-coding modes. The granularity at which encoder and decoder switch between these prediction mode types may correspond to a subdivision of picture 12 and 12’, respectively, into coding segments or coding blocks. In units of these coding segments, for instance, the picture may be subdivided into blocks being intra-coded and blocks being inter-coded. Intra-coded blocks are predicted on the basis of a spatial, already coded / decoded neighborhood of the respective block as is outlined in more detail below. Several intra-coding modes may exist and be selected for a respective intra-coded segment including directional or angular intra-coding modes according to which the respective segment is filled by extrapolating the sample values of the neighborhood along a certain direction which is specific for the respective directional intra-coding mode, into the respective intra-coded segment. The intra-coding modes may, for instance, also comprise one or more further modes such as a DC coding mode, according to which the prediction for the respective intra-coded block assigns

[0767] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bma DC value to all samples within the respective intra-coded segment, and / or a planar intracoding mode according to which the prediction of the respective block is approximated or determined to be a spatial distribution of sample values described by a two-dimensional linear function over the sample positions of the respective intra-coded block with driving tilt and offset of the plane defined by the two-dimensional linear function on the basis of the neighboring samples. Compared thereto, inter-coded blocks may be predicted, for instance, temporally. For inter-coded blocks, motion vectors may be signaled within the data stream, the motion vectors indicating the spatial displacement of the portion of a previously coded picture of the video to which picture 12 belongs, at which the previously coded / decoded picture is sampled in order to obtain the prediction signal for the respective inter-coded block. This means, in addition to the residual signal coding comprised by data stream 14, such as the entropy-coded transform coefficient levels representing the quantized spectral-domain prediction residual signal 24”, data stream 14 may have encoded thereinto coding mode parameters for assigning the coding modes to the various blocks, prediction parameters for some of the blocks, such as motion parameters for inter-coded segments, and optional further parameters such as parameters for controlling and signaling the subdivision of picture 12 and 12’, respectively, into the segments. The decoder 20 uses these parameters to subdivide the picture in the same manner as the encoder did, to assign the same prediction modes to the segments, and to perform the same prediction to result in the same prediction signal.

[0768] Fig. 7 illustrates the relationship between the reconstructed signal, i.e. the reconstructed picture 12’, on the one hand, and the combination of the prediction residual signal 24”” as signaled in the data stream 14, and the prediction signal 26, on the other hand. As already denoted above, the combination may be an addition. The prediction signal 26 is illustrated in Fig. 7 as a subdivision of the picture area into intra-coded blocks which are illustratively indicated using hatching, and inter-coded blocks which are illustratively indicated nothatched. The subdivision may be any subdivision, such as a regular subdivision of the picture area into rows and columns of square blocks or non-square blocks, or a multi-tree subdivision of picture 12 from a tree root block into a plurality of leaf blocks of varying size, such as a quadtree subdivision or the like, wherein a mixture thereof is illustrated in Fig. 7 in which the picture area is first subdivided into rows and columns of tree root blocks which are then further subdivided in accordance with a recursive multi-tree subdivisioning into one or more leaf blocks.

[0769] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmAgain, data stream 14 may have an intra-coding mode coded thereinto for intra-coded blocks 80, which assigns one of several supported intra-coding modes to the respective intra-coded block 80. For inter-coded blocks 82, the data stream 14 may have one or more motion parameters coded thereinto. Generally speaking, inter-coded blocks 82 are not restricted to being temporally coded. Alternatively, inter-coded blocks 82 may be any block predicted from previously coded portions beyond the current picture 12 itself, such as previously coded pictures of a video to which picture 12 belongs, or picture of another view or an hierarchically lower layer in the case of encoder and decoder being scalable encoders and decoders, respectively.

[0770] The prediction residual signal 24”” in Fig. 7 is also illustrated as a subdivision of the picture area into blocks 84. These blocks might be called transform blocks in order to distinguish same from the coding blocks 80 and 82. In effect, Fig. 7 illustrates that encoder 10 and decoder 20 may use two different subdivisions of picture 12 and picture 12’, respectively, into blocks, namely one subdivisioning into coding blocks 80 and 82, respectively, and another subdivision into transform blocks 84. Both subdivisions might be the same, i.e. each coding block 80 and 82, may concurrently form a transform block 84, but Fig. 7 illustrates the case where, for instance, a subdivision into transform blocks 84 forms an extension of the subdivision into coding blocks 80, 82 so that any border between two blocks of blocks 80 and 82 overlays a border between two blocks 84, or alternatively speaking each block 80, 82 either coincides with one of the transform blocks 84 or coincides with a cluster of transform blocks 84. However, the subdivisions may also be determined or selected independent from each other so that transform blocks 84 could alternatively cross block borders between blocks 80, 82. As far as the subdivision into transform blocks 84 is concerned, similar statements are thus true as those brought forward with respect to the subdivision into blocks 80, 82, i.e. the blocks 84 may be the result of a regular subdivision of picture area into blocks (with or without arrangement into rows and columns), the result of a recursive multi-tree subdivisioning of the picture area, or a combination thereof or any other sort of blockation. Just as an aside, it is noted that blocks 80, 82 and 84 are not restricted to being of quadratic, rectangular or any other shape.

[0771] Fig. 7 further illustrates that the combination of the prediction signal 26 and the prediction residual signal 24”” directly results in the reconstructed signal 12’. However, it should be noted that more than one prediction signal 26 may be combined with the prediction residual signal 24”” to result into picture 12’ in accordance with alternative embodiments.

[0772] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmIn Fig. 7, the transform blocks 84 shall have the following significance. Transformer 28 and inverse transformer 54 perform their transformations in units of these transform blocks 84. For instance, many codecs use some sort of DST or DCT for all transform blocks 84. Some codecs allow for skipping the transformation so that, for some of the transform blocks 84, the prediction residual signal is coded in the spatial domain directly. However, in accordance with embodiments described below, encoder 10 and decoder 20 are configured in such a manner that they support several transforms. For example, the transforms supported by encoder 10 and decoder 20 could comprise:

[0773] o DCT-II (or DCT-III), where DCT stands for Discrete Cosine Transform

[0774] o DST-IV, where DST stands for Discrete Sine Transform

[0775] o DCT-IV

[0776] o DST- VI I

[0777] o Identity Transformation (IT)

[0778] Naturally, while transformer 28 would support all of the forward transform versions of these transforms, the decoder 20 or inverse transformer 54 would support the corresponding backward or inverse versions thereof:

[0779] o Inverse DCT-II (or inverse DCT-III)

[0780] o Inverse DST-IV

[0781] o Inverse DCT-IV

[0782] o Inverse DST-VII

[0783] o Identity Transformation (IT)

[0784] The subsequent description provides more details on which transforms could be supported by encoder 10 and decoder 20. In any case, it should be noted that the set of supported transforms may comprise merely one transform such as one spectral-to-spatial or spatial-to-spectral transform.

[0785] As already outlined above, Fig. 5, Fig. 6 and Fig. 7 have been presented as an example where the inventive concept described further below may be implemented in order to form specific examples for encoders and decoders according to the present application. Insofar, the encoder and decoder of Fig. 5, and Fig. 6, respectively, may represent possible implementations of the encoders and decoders described herein below. Fig. 5, and Fig. 6 are, however, only examples. An encoder according to embodiments of the present application may, however, perform block-based encoding of a picture 12 using the concept outlined in more detail below and being different from the encoder of Fig. 5 such as, for

[0786] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bminstance, in that same is no video encoder, but a still picture encoder, in that same does not support inter-prediction, or in that the sub-division into blocks 80 is performed in a manner different than exemplified in Fig. 7. Likewise, decoders according to embodiments of the present application may perform block-based decoding of picture 12’ from data stream 14 using the coding concept further outlined below, but may differ, for instance, from the decoder 20 of Fig. 6 in that same is no video decoder, but a still picture decoder, in that same does not support intra-prediction, or in that same sub-divides picture 12’ into blocks in a manner different than described with respect to Fig. 7 and / or in that same does not derive the prediction residual from the data stream 14 in transform domain, but in spatial domain, for instance.

[0787] For example, embodiments described with respect to Fig. 1 to Fig. 4 may relate to a postfiltering, which may be employed by decoder 20 of Fig. 6 to a picture of the reconstructed video 12’. For example, the post-filtering may be employed for improving the quality of a picture, e.g., to reduce artifacts, or for upsampling a resolution of the picture or for temporal upsampling the reconstructed video 12’, i.e., for increasing the frame rate of the reconstructed video 12’, or for changing a viewpoint of the picture (e.g., in case of a multiview video), or for adding or adapting 3D effects (e.g., in case the video includes depth information).

[0788] Embodiments of the invention may optionally embody one of three variants of encoding and decoding video data streams and the related video data streams described in the following. The first and the second variants may be combined with each other.

[0789] According to the first one of the two variants of the invention the video is decoded from the video data stream by block-based predictive and transform based residual decoding by decoding prediction residual data of a residual block into / from the video data stream.

[0790] According to the first variant, the decoding of the prediction residual data of the residual block is performed by use of context-adaptive variable length decoding by using

[0791] a first syntax element indicating a total number of non-zero transform coefficients in a transform block representing the residual block, and a trailing-one number, indicating a number of non-zero transform coefficients having an absolute value of one when traversing the coefficients along a scan order,

[0792] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmone or more second syntax elements indicating a sign of the non-zero transform coefficients having an absolute value of one when traversing the coefficients along the scan order,

[0793] one or more third syntax elements indicating a value of the non-zero transform coefficients except for the number of non-zero transform coefficients having an absolute value of one when traversing the coefficients along the scan order, a fourth syntax element indicating a total number of zero-valued transform coefficient levels in the transform block from a firstly-encountered non-zero transform coefficient in the scan order onwards, and

[0794] one or more fifth syntax elements indicting positions of the non-zero transform coefficients along the scan order by indicating a number of consecutive zero-valued transform coefficients in the scan order between in the scan order consecutively encountered non-zero transform coefficients, or,

[0795] in alternative to the use of context-adaptive variable length decoding, the decoding of the prediction residual data of the residual block is performed by use of context-adaptive binary arithmetic decoding by

[0796] decoding a significance map which indicates positions of non-zero transform coefficients in a transform block representing the residual block by, in a forward scan traversing transform coefficients of the transform block, decoding a significance flag which indicates whether a non-zero transform coefficient is positioned at a current position, and, if so, and if the current position is not the last in the forward scan, decoding a last-significance flag which indicates whether the non-zero transform coefficient positioned at the current position is the last non-zero transform coefficient in the forward scan order, and

[0797] decoding the non-zero transform coefficients’ values sequentially in a reverse scan order, reversing the forward scan order.

[0798] According to the first one of the two variants of the invention the video is encoded into the video data stream by block-based predictive and transform based residual encoding by encoding prediction residual data of a residual block into the video data stream.

[0799] According to the first variant, the encoding of the prediction residual data of the residual block is performed by use of context-adaptive variable length encoding by using

[0800] a first syntax element indicating a total number of non-zero transform coefficients in a transform block representing the residual block, and a trailing-one number, indicating a

[0801] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmnumber of non-zero transform coefficients having an absolute value of one when traversing the coefficients along a scan order,

[0802] one or more second syntax elements indicating a sign of the non-zero transform coefficients having an absolute value of one when traversing the coefficients along the scan order,

[0803] one or more third syntax elements indicating a value of the non-zero transform coefficients except for the number of non-zero transform coefficients having an absolute value of one when traversing the coefficients along the scan order, a fourth syntax element indicating a total number of zero-valued transform coefficient levels in the transform block from a firstly-encountered non-zero transform coefficient in the scan order onwards, and

[0804] one or more fifth syntax elements indicting positions of the non-zero transform coefficients along the scan order by indicating a number of consecutive zero-valued transform coefficients in the scan order between in the scan order consecutively encountered non-zero transform coefficients, or

[0805] in alternative to the use of context-adaptive variable length encoding, the encoding of the prediction residual data of the residual block is performed by use of context-adaptive binary arithmetic encoding by

[0806] encoding a significance map which indicates positions of non-zero transform coefficients in a transform block representing the residual block by, in a forward scan traversing transform coefficients of the transform block, encoding a significance flag which indicates whether a non-zero transform coefficient is positioned at a current position, and, if so, and if the current position is not the last in the forward scan, encoding a last-significance flag which indicates whether the non-zero transform coefficient positioned at the current position is the last non-zero transform coefficient in the forward scan order, and

[0807] encoding the non-zero transform coefficients’ values sequentially in a reverse scan order, reversing the forward scan order.

[0808] According to the second variant, a video decoder (or apparatus for decoding a video from a video data stream) comprises a coded picture buffer (CPB) and a decoded picture buffer (DPB). The video decoder is configured to

[0809] receive a data stream having pictures of a video encoded thereinto along a coding order as a sequence of access units (AU) (e.g., the term “access unit” refers to a portion of the video data stream, which comprises the coded video data, or information, relating to one time frame of the video),

[0810] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmdecode a current AU removed from the CPB using inter-picture prediction from a referenced reference picture stored in the DPB to acquire a decoded picture, and insert the decoded picture into the DPB,

[0811] assign to each reference picture stored in the DPB a classification as one of a shortterm reference picture, a long-term reference picture and an unused-for-reference picture, read DPB mode information from the current AU,

[0812] if the DPB mode information indicates a first mode, remove one or more reference pictures classified as a short-term picture, according to a first-in-first-out (FIFO) strategy, from the DPB,

[0813] if the DPB mode information indicates a second mode, read memory management control information comprising at least one command in the current AU and execute the at least one command so as to change the classification assigned to at least one of the reference pictures stored in the DPB, and use the classification of the reference pictures in the DPB, for managing reference picture removal from the DPB.

[0814] According to the second variant, a video data stream is ought to be decoded by being fed to a decoder comprising a coded picture buffer (CPB). According to the second variant, a video encoder (or apparatus for encoding a video into a video data stream) is configured to encode, into a data stream, pictures of a video encoded in a coding order as a sequence of access units (AU),

[0815] wherein the apparatus is configured to, in encoding the AUs,

[0816] encode a current picture using inter-picture prediction from a referenced reference picture stored in a decoded picture buffer (DPB) into a current AU, and

[0817] insert a decoded version of the current picture in the DPB into the DPB, assign to each reference picture stored in the DPB a classification as one of a shortterm reference picture, a long-term reference picture and an unused-for-reference picture, write DPB mode information into the current AU,

[0818] if the DPB mode information indicates a first mode, remove one or more reference pictures classified as a short-term picture, according to a FIFO strategy, from the DPB, if the DPB mode information indicates a second mode, write memory management control information comprising at least one command into the current AU, the command being instructive to change the classification assigned to at least one of the reference pictures stored in the DPB, wherein the classification of the reference pictures in the DPB, is used for managing reference picture removal from the DPB.

[0819] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmAccording to the third variant, the video is decoded from the video data stream by blockbased predictive decoding and transform-based residual decoding by decoding prediction residual data of a residual block from the video data stream by use of context-adaptive binary arithmetic decoding of quantization indices of transform coefficients of a transform block representing the residual block and sequential dequantization of the quantization indices according to which a value of a current transform coefficient depends on a parity of quantization indices of previous quantization indices.

[0820] According to the third variant, the video is encoded into the video data stream by block based predictive encoding and transform based residual encoding by encoding the prediction residual data of a residual block into the video data stream by use of context-adaptive binary arithmetic coding of quantization indices of transform coefficients of a transform block representing the residual block and sequential quantization of the transform coefficients to obtain the quantization indices, according to which a quantizer for quantizing a current transform coefficient depends on a parity of quantization indices of previous quantization indices.

[0821] All three variants relate to a video encoder, a video decoder, a method for decoding a video, a method for encoding a video, and a video data stream as obtained by the respective encoding method.

[0822] In the following, further optional details and features of the first and second variants are described.

[0823] Embodiments of the first variant and the second variant may be compliant to H.264 / AVC. Embodiments of the third variant may be compliant to H.266 / VVC.

[0824] One aspect of the first and second variants relates to the handling of decoded pictures and their buffering in a decoded picture buffer, DPB.

[0825] According to an embodiment, two types of reference pictures may be distinguished: shortterm and long-term. The encoder does the same in emulating the DPB fill state of the decoder at each point in time during decoding. A reference picture may be marked as “unused for reference” when it becomes no longer needed for prediction reference. The conversion among these three statuses (short-term, long-term, and unused for reference) is controlled by a decoded reference picture marking process. There are two alternative

[0826] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmdecoded reference picture marking mechanisms, the implicit sliding window process and the explicit memory management control operation (MMCO) process. For each currently decoded picture or each currently decoded AU, it is signalled in the data stream as to which process shall be used for DPB management. The sliding window process marks a shortterm reference picture as “unused for reference” when the number of reference frames is equal to a given maximum number (max-num-ref-frames in SPS). The short-term reference pictures are stored in a first-in, first-out manner so that the most recently decoded short-term pictures are kept in the DPB. The explicit MMCO process is controlled via multiple MMCO commands. If this mode is selected for a current AU or currently decoded picture, the bitstream contains for this, or in this, AU one or more of these commands. An MMCO command may any of 1) mark one or more short-term or long-term reference picture as “unused for reference,” 2) mark all the pictures as “unused for reference,” or 3) mark the current reference picture or an existing short-term reference picture as long-term, and assign a long-term picture index to that long-term picture. The reference picture marking operations as well as any output - for sake of presentation - and removal of pictures from the DPB may be performed after a picture has been decoded.

[0827] Some possible but optional details of the reference picture marking mechanism are discussed in the following. 1) A first aspect relates to gaps in frame number and non-Existing pictures. Although not explained above, it might be that each reference picture in the DPB is associated with a frame number. Normally this number increases by one for each reference picture, but gaps in frame number may be allowed by setting a corresponding high level (such as sequence level) flag, which might be called parameter-gaps-in-frame-num-allowed-flag, to one for example in order to allow that an encoder or a MANE (media aware network element) can deliver a bitstream in which the frame numbers increase by more than one for a reference picture relative to the preceding reference picture in decoding order. This might be favourable in order to support temporal scalability. A sequence of AUs with gaps in the frame numbers may be received, and non-existing pictures to fill the gap may be created. The non-existing pictures are assigned with frame number values in the gap and are considered as reference pictures during decoded reference picture marking, but will not be used for output (hence not displayed). The nonexisting pictures ensure that the status of the DPB, with respect to the frame numbers of the pictures residing in it, is the same for a decoder that received the pictures as for a decoder that did not receive the pictures.

[0828] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmAnother possible aspect of the first and second variants relates to the loss of a reference picture when using sliding Window. When a reference picture is lost, it may be possible to conceal the picture and to possibly report the loss to the encoder if a feedback channel is available given that the loss is detected. If gaps in frame number are disallowed, a discontinuity in the frame number values indicates an unintentional loss of a reference picture. If gaps in frame number are allowed, a discontinuity in frame number values may be caused by either intentional removal of temporal layers or subsequences or an accidental picture loss, and decoders should infer a picture loss only if a non-existing picture is referred in the inter prediction process. The picture order count of a concealed picture may not be known which can cause the decoder to use incorrect reference pictures without detecting any errors when decoding B-pictures.

[0829] An even further possible aspect of the first and second variants relates to the loss of a reference picture with MMCO. When losing a reference picture that contains an MMCO command marking a short-term reference picture as “unused for reference,” then the status of reference pictures in the DPB becomes incorrect and consequently, reference picture lists for a few pictures following the lost picture may become incorrect. If a picture containing MMCO commands related to long- term reference pictures is lost there is a risk that the number of long-term reference pictures in the DPB is different from what it would have been if the picture was received, resulting in an “incorrect” sliding window process for all the following pictures. That is, the encoder and decoder may contain a different number of shortterm reference pictures resulting in out- of-sync behaviour of the sliding window process. What makes the situation even worse is that a decoder will not necessarily know that the sliding window process is out-of-sync.

[0830] In the following, MMCO commands are shown. One or more or all of the commands may apply to yield in different embodiments:

[0831] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bm

[0832] " "

[0833] " "

[0834] " "

[0835] " "

[0836] " "

[0837] " "

[0838] " "

[0839]

[0840] A further option for the implementation of decoder and encoder according to the first and second variants is described now, may optionally be combined with the one concerning the DPB management described before, and relates to entropy decoding of some syntax element such as the residual data in form of transform coefficients into the bitstream. Lossless entropy coding of lossy quantized transform coefficients is a crucial part of an efficient video codec. One such method is referred to as context-adaptive-variable-length-coding (CAVLC) in which the encoder switches between different variable length code (VLC) tables for various syntax elements, depending on the values of the previously transmitted syntax elements in the same slice in a context-adaptive fashion. Encoder and decoder may use the CAVLC. Due to the fact that each syntax element is coded into the bitstream by writing a corresponding codeword into the bitstream which has been selected for that syntax element from the context-adaptively selected code table, each CAVLC encoded bit in the bitstream can be associated to a single syntax element. The relevant information about the transform coefficient levels in scan order to be present in bitstream is, thus, available in a direct accessible form as syntax elements when CAVLC is used. Encoder and decoder may use CAVLC to signal the transform coefficients in the bitstream. The following syntax elements may be used, e.g. syntax elements having the following semantics:

[0841] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmOne syntax element indicating the total number of non-zero transform coefficient levels in a transform block (as indicated by CoeffToken)

[0842] One or more syntax elements indicating the number of trailing one transform coefficient levels (as indicated by CoeffToken), e.g. a run of syntax elements occurring at the end of scanning the syntax elements in a scan order up to the last non-zero syntax element which are all one, and their sign (as indicated by trailing_ones_sign_flag)

[0843] One or more syntax element per non-zero transform coefficient except the trailing one transform coefficients, which indicates the transform coefficient level value One syntax element indicating the total number of zero-valued transform coefficient levels

[0844] Syntax elements indicting the number of consecutive transform coefficient levels in scan order with zero value from a current scan position onwards before a non-zero valued transform coefficient level is encountered.

[0845] It might alternatively or additionally be that the encoder might select between the usage of CABAC, thus context-adaptive binary arithmetic coding, and CAVLC and signal the selection in the bitstream and that the decoder reads this signal and uses the indicated way of decoding the residual data.

[0846] A further option for the implementation of decoder and encoder according to the first and second variants is described now, may optionally be combined with any of the one concerning the DPB management and the one concerning CAVLC described before, and relates to a quarter pel interpolation filter. In order to allow inter-prediction at a finer granularity than the regular full-pel sample grid, a sample interpolation process is used to derive sample values at sub-pel sample positions which can range from half-pel positions to quarter-pel position. One method to carry out quarter-pel interpolation may be used by encoder and decoder and is as follows. First, a 6-tap FIR filter is used to generate sample values at half-pel positions followed by an averaging of the generated half-pel position sample values through interpolation to generate sample values at quarter-pel position for luma components.

[0847] Further embodiments according to the first and second variants are described in the following:

[0848] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmAccording to an embodiment, the apparatus may further comprise a decoded picture buffer (DPB) and be configured to decode a current AU using inter-picture prediction from a referenced reference picture stored in the DPB to obtain a decoded picture, and insert the decoded picture into the DPB, assign to each reference picture stored in the DPB a classification as one of a short-term reference picture, a long-term reference picture and an unused-for-reference picture, read DPB mode information from the current AU, if the DPB mode information indicates a first mode, remove one or more reference pictures classified as a short-term picture, according to a FIFO strategy, from the DPB, if the DPB mode information indicates a second mode, read memory management control information comprising at least one command in the current AU and execute the at least one command so as to change the classification assigned to at least one of the reference pictures stored in the DPB, and use the classification of the reference pictures in the DPB, for managing reference picture removal 424 from the DPB.

[0849] In an embodiment, the apparatus may be configured to read from the current AU an indication whether the decoded picture is not used for inter-picture prediction; perform the insertion of the decoded picture into the DPB, if the decoded picture is not indicated to be not used for inter-picture prediction or not directly to be output, and directly output the decoded picture without buffering same in the DPB, if the decoded picture is indicated to be not used for inter-picture prediction and directly to be output.

[0850] According to an embodiment, the apparatus may be configured to assign a frame index to each reference picture in the DPB, classified to be a long-term picture, and use a predetermined reference picture in the DPB, classified to be a long-term picture, as the referenced reference picture in the DPB if the frame index assigned to the predetermined reference picture is referred to in the current AU.

[0851] In an embodiment, the apparatus may be configured to one or more of: if the at least one command in the current AU is a first command, re-classify a reference picture in the DPB, classified to be a short-term reference picture, as an unused-for-reference picture, if the at least one command in the current AU is a second command, re-classify a reference picture in the DPB, classified to be a long-term reference picture, as an unused-for-reference picture, if the at least one command in the current AU is a third command, re-classify a reference picture in the DPB, classified to be a short-term picture, as a long-term reference picture, and assign a frame index to the re-classified reference picture, if the at least one command in the current AU is a fourth command, set an upper frame index limit according

[0852] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmto the fourth command, and re-classify all reference picture in the DPB, classified to be a long-term picture, and having assigned thereto a frame index exceeding the upper frame index limit, as an unused-for-reference picture, if the at least one command in the current AU is a sixth command, classify the current picture as a long-term picture, as an unused-for-reference picture, and assign a frame index to the re-classified reference picture.

[0853] According to an embodiment, the apparatus may be configured to remove any reference picture from the DPB, which is classified as an unused-for-reference picture, and which is no longer to be output.

[0854] In an embodiment the apparatus may be configured to read an entropy coding mode indicator from the data stream, and decode prediction residual data from the current AU using a context adaptive variable length coding mode if the entropy coding mode indicator indicates the context adaptive variable length coding mode, and using a context adaptive binary arithmetic coding mode if the entropy coding mode indicator indicates the context adaptive binary arithmetic coding mode.

[0855] According to an embodiment, the apparatus may be configured to derive quarter pel values in the referenced reference picture based on a motion vector in the current AU and using 6-tap FIR filter so as to derive half-pel values and averaging neighboring half-pel values.

[0856] In an embodiment, the apparatus may be configured to:

[0857] decode a current AU using inter-picture prediction from a referenced reference picture stored in the DPB to acquire a decoded picture, and to insert the decoded picture into the DPB,

[0858] assign to each reference picture stored in the DPB a classification as one of a shortterm reference picture, a long-term reference picture and an unused-for-reference picture, read DPB mode information from the current AU,

[0859] if the DPB mode information indicates a first mode, remove one or more reference pictures classified as a short-term picture, according to a first-in-first-out (FIFO) strategy, from the DPB,

[0860] if the DPB mode information indicates a second mode, read memory management control information comprising at least one command in the current AU and execute the at least one command so as to change the classification assigned to at least one of the

[0861] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmreference pictures stored in the DPB, and use the classification of the reference pictures in the DPB, for managing reference picture removal from the DPB.

[0862] According to an embodiment, the apparatus may be configured to:

[0863] read from the current AU an indication whether the decoded picture is not used for inter-picture prediction;

[0864] perform the insertion of the decoded picture into the DPB, if the decoded picture is not indicated to be not used for inter-picture prediction or not directly to be output, and directly output the decoded picture without buffering same in the DPB, if the decoded picture is indicated to be not used for inter-picture prediction and directly to be output.

[0865] In an embodiment, the apparatus may be configured to:

[0866] assign a frame index to each reference picture in the DPB, classified to be a longterm picture, and

[0867] use a predetermined reference picture in the DPB, classified to be a long-term picture, as the referenced reference picture in the DPB if the frame index assigned to the predetermined reference picture is referred to in the current AU.

[0868] According to an embodiment, the apparatus may be configured to one or more of:

[0869] if the at least one command in the current AU is a first command,

[0870] re-classify a reference picture in the DPB, classified to be a short-term reference picture, as an unused-for-reference picture,

[0871] if the at least one command in the current AU is a second command,

[0872] re-classify a reference picture in the DPB, classified to be a long-term reference picture, as an unused-for-reference picture,

[0873] if the at least one command in the current AU is a third command,

[0874] re-classify a reference picture in the DPB, classified to be a short-term picture, as a long-term reference picture, and assign a frame index to the re-classified reference picture,

[0875] if the at least one command in the current AU is a fourth command,

[0876] set an upper frame index limit according to the fourth command, and reclassify all reference picture in the DPB, classified to be a long-term picture, and having assigned thereto a frame index exceeding the upper frame index limit, as an unused-for-reference picture,

[0877] if the at least one command in the current AU is a sixth command,

[0878] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmclassify the current picture as a long-term picture, as an unused-for-reference picture, and assign a frame index to the re-classified reference picture.

[0879] In an embodiment, the apparatus may be configured to:

[0880] remove any reference picture from the DPB, which is classified as an unused-for-reference picture, and which is no longer to be output.

[0881] In the following, further optional details and features of the third variant are described.

[0882] Multiple Reference Lines (MRL): For intra prediction, not only the adjacent line of neighboring samples can be used but also one of the two non-adjacent reference lines as the reference line for intra-picture prediction of luma samples, corresponding to the two or three lines away from the current block.

[0883] Adaptive MV Resolution (AMVR): A selection of the MV resolution at the CU level is performed. For inter predicted CUs, the selected MV resolution is indicated and can be one quarter, one half, whole integer, or four, in units of luma samples. If half luma sample resolution is selected, an alternative luma interpolation filter is used for the half-sample position in this block, i.e. different interpolation filter are used when the resolution is quarter-per or half-pel for the MV.

[0884] History-Based MV Prediction (HMVP): In addition to spatial and temporal neighbor MV predictions, a new candidate type is added for MV prediction in the merge mode and AMVP candidate list. The HMVP candidates are established using a five-entry table that is maintained and updated using a first-in-first-out (FIFO) rule. The motion vector candidates list are generated by using the spatial and temporal neighbors and HMVP candidates.

[0885] Affine Motion: An affine motion model with CU-level signaling is used for luma. The CU-level affine motion can be either a 4-parameter model or a 6-parameter model. The 4-parameter model uses two MVs, which correspond to two control points located at the topleft and top-right corners of the CU and the 6-parameter model uses three MVs, which corresponds to three control points located at the top-left, bottom-left and top-right corners. When a CU is coded in affine motion mode, the luma block of the CU is spilt into 4 x 4 subblocks and the MV at the central sample position of each subblock is calculated according to the affine motion model and set as the subblock MV based on the control points. The subblock MV is rounded to 1 / 16 luma sample precision during the calculation

[0886] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmand a set of 6-tap interpolation filters is applied to generate the prediction of each subblock. As for the case of non-affine, a merge mode and AM VP mode are used for prediction and coding of affine motion parameters.

[0887] Coefficient coding: When encoding the coefficient level, first a flag (SigFlag) is indicated, that specifies that whether coefficient level is not 0. When the flag is equal to 1 (the coeficient level is not 0) a further flag is indicated (Gt1-Flag), which indicates whether the absolute level is greater than 1, in which case (absolute level is greater than 1) two further flags are present, a parity level flag (Parity-Flag) that specifies the parity of the transform coefficient level and a further flag (Gt3-Flag) that specifies whether the absolute value of the transform coefficient level is greater than a 3.

[0888] All embodiments of the invention described herein may optionally embody one of the first, the second, and the third variant. All embodiments of the invention described above may be embodied in either one of the first, the second, and the third variant. In other words, all embodiments of the invention described above may be embodied in the first variant, all embodiments of the invention described in the following may be embodied in the second variant, all embodiments of the invention described in the following may be embodied in the third variant.

[0889] In other words, embodiments may provide an H.264 / AVC decoder, an H.264 / AVC video data stream, a video encoder for providing an H.264 / AVC video data streama video encoder for providing an H.265 / HEVC video data stream, or an H.266 / VVC decoder, an H.266 / VVC video data stream, a video encoder for providing an H.266 / VVC video data stream.

[0890] Further embodiments

[0891] It is noted, that details described above may be individually combined with the subjectmatter of the following embodiments. Text in brackets represents optional features, explanations, examples, which may optionally be combined with the embodiments.

[0892] 1. Apparatus 20 for reconstructing a video from a data stream 14, configured for

[0893] reconstructing 21 a picture 12’ of the video [e.g., the picture comprises or consists of one or more sample arrays, e.g., the picture comprises or consists of a luma

[0894] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmsample array and two chroma sample arrays] [e.g., the picture belongs to one of a sequence of time frames of the video; E.g., the data stream has encoded thereinto, for each of a sequence of time frames, one or more pictures, and optionally, in addition to the one or more pictures, one or more sample arrays carrying spatially sampled information]; and

[0895] filtering 61 the reconstructed picture 12’ using a neural network by obtaining 63 [e.g., deriving, forming] filter input data 65 [e.g., an input tensor] based on the reconstructed picture 12’ and based on one or more auxiliary sample arrays 15 [e.g., two-dimensional sample arrays, e.g., representing spatially sampled information], and subjecting [or inputting] the filter input data 65 to the neural network [e.g., the filter input data comprises the reconstructed picture or a portion thereof and further comprises the one or more auxiliary sample arrays or respective portions of the one or more auxiliary sample arrays] [E.g., throughout the embodiments, deriving or obtaining the filter input data may optionally be implemented by the apparatus including, into the filter input data, the reconstructed picture, or a portion thereof (e.g. a patch thereof), and the one or more auxiliary sample arrays, or respective portions thereof (e.g., respective patches thereof)],

[0896] wherein the picture and the one or more auxiliary sample arrays belong to the same time frame 11* [e.g., access unit] of the video.

[0897] 2. Apparatus according to embodiment 1, wherein the filter input data is represented by a tensor having the same dimension as an input [e.g., an input layer] of the neural network.

[0898] 3. Apparatus according to embodiment 1 or 2, wherein the data stream is a multi-layer video data stream, and wherein the picture belongs to a first layer of the data stream and wherein the one or more auxiliary sample arrays belong to one of one or more further layers of the data stream [e.g., the one or more auxiliary sample array being distributed onto the one or more further layers, e.g., each belongs to a different one or all to the same one of the further layers] [E.g., the apparatus is configured for decoding the picture from the first layer, and for decoding the auxiliary sample arrays from the one or more further layers].

[0899] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bm4. Apparatus according to embodiment 3, wherein coded video sequences [e.g., CLVSs] of the first layer and the one or more further layers start at the same time frame and / or end at the same time frame.

[0900] 5. Apparatus according to embodiment 3, configured for

[0901] depending on an indication in the data stream [e.g., deriving, from the data stream, an indication, e.g., a syntax element, which indicates whether pictures of a coded video sequence preceding a coded video sequence to which the picture belongs can be used for deriving the filter input data, and if the indication indicates that pictures of a coded video sequence preceding a coded video sequence to which the picture belongs cannot be used for deriving the filter input data], inferring that the coded video sequence of the first layer and coded video sequences of the one or more further layers start at the same time frame, and / or

[0902] depending on an indication in the data stream [e.g., deriving, from the data stream, an indication, e.g., a syntax element, which indicates whether pictures of a coded video sequence suceeding a coded video sequence to which the picture belongs can be used for deriving the filter input data, and if the indication indicates that pictures of a coded video sequence succeeding a coded video sequence to which the picture belongs cannot be used for deriving the filter input data], inferring that the coded video sequence of the first layer and coded video sequences of the one or more further layers end at the same time frame.

[0903] 6. Apparatus according to any of embodiments 1 to 2, wherein the data stream has encoded thereinto a sequence of primary pictures including the picture, the sequence of primary pictures representing the video [e.g., providing a representation of the video, e.g., from which the video is can be reconstructed, e.g., without further pictures or sample arrays included in the data stream], and wherein the data stream further has encoded thereinto the one or more auxiliary sample arrays [e.g., as nonprimary sample arrays, e.g. in payload packets interspersed between payload packets in which the primary pictures are encoded],

[0904] 7. Apparatus according to any of embodiments 1 to 2 or 6, wherein the data stream has encoded thereinto a sequence of primary pictures including the picture, the sequence of primary pictures being encoded in payload packets of a first type [e.g.,

[0905] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmVCL NAL units], and wherein the data stream further has encoded thereinto the one or more auxiliary sample arrays in payload packets [e.g., NAL units] of a second type [e.g, the payload packets of the second type being interspersed between the payload packets of the first type],

[0906] 8. Apparatus according to any of the embodiments 1 to 7, wherein the picture belongs to a first time frame of the video, and wherein the apparatus is configured for reconstructing a further picture of the video, the further picture belonging to a second time frame of the video, and for obtaining the filter input data further based on the reconstructed further picture and based on one or more further auxiliary sample arrays, the one or more further auxiliary sample arrays belonging to the second time frame [e.g., the further picture is encoded into the first layer of the data stream, and the one or more further auxiliary sample arrays belong to the one of one or more further layers of the data stream],

[0907] 9. Apparatus according to any of the embodiments 1 to 8, configured for deriving, from the data stream, a syntax element [e.g., nnpfc_auxiliary_inp_idc] which differentiates between a plurality of modes of obtaining the filter input data, wherein the plurality of modes includes a mode according to which the filter input data for the picture is to be obtained based on one or more auxiliary sample arrays belonging to the same time frame as the picture [and optionally, the plurality of modes includes one or more or all of a mode according to which no auxiliary information is used for obtaining the filter input data, e.g. the filter input data is obtained based on the reconstructed picture only, or based on a plurality of pictures belonging to different time frames of the video only; a mode according to which a quantization parameter for reconstructing the picture is to be used for obtaining the filter input data, and a mode according to which a text string is to be used for obtaining the filter input data],

[0908] 10. Apparatus according to any of the embodiments 1 to 9, wherein the apparatus is configured for obtaining the filter input data in form of a tensor comprising [or consisting of] a plurality of channels, each of which provides an input channel for the neural network [e.g., the channels representing one dimension of the input tensor],

[0909] 11. Apparatus according to any of the embodiments 1 to 10, configured for

[0910] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmderiving, from the data stream [e.g., from one or more syntax elements], an indication of a set of auxiliary pictures, each of the auxiliary pictures being represented by one or more of the one or more auxiliary sample arrays [e.g., the indication indicates a count or quantity of the auxiliary pictures],

[0911] 12. Apparatus according to embodiment 11, wherein the set of auxiliary pictures is indicated by indicating a set of layers out of a plurality of layers of the data stream, each of the layers of the set of layers carrying one of the auxiliary pictures.

[0912] 13. Apparatus according to embodiment 11 or 12, configured for deriving, from the data stream, an indication of the count of auxiliary pictures of the set of auxiliary pictures.

[0913] 14. Apparatus according to any of the embodiments 1 to 13, configured for

[0914] deriving, from the data stream [e.g., from one or more syntax elements], an indication of a set of layers [e.g., comprising one or more layers, e.g., the one or more further layers] out of a plurality of layers of the data stream, each of the layers of the set of layers carrying an auxiliary picture [the auxiliary pictures carried by the set of layers may be referred to as set of auxiliary pictures], the auxiliary picture being represented by one or more of the one or more auxiliary sample arrays.

[0915] 15. Apparatus according to embodiment 14, configured for deriving the set or layers by

[0916] deriving, from the data stream, an indication [e.g., a syntax element, e.g., nnpfc_num_aux_layers_minus1] indicating the count [or quantity] of layers of the set of layers, and

[0917] deriving, from the data stream, for each of the set of layers, an identification of the respective layer within the plurality of layers of the data stream [e.g., a syntax element, e.g., nnpfc_layer_target_id] [E.g., the syntax element identifies the layer by indicating a layer identifier, identifying the layer among the plurality of layers or by indicating a layer identifier which is indicated in the data stream for each of the plurality of layers. Alternatively, the layer identifier may be mapped by the apparatus to one of the plurality of layers, which may be uniquely identified by means of one or more properties signaled in the data stream for each of the plurality of layers].

[0918] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bm16. Apparatus according to embodiment 15, configured for deriving, from the data stream, a layer identification syntax element which signals the identification of the respective layer within the plurality of layers, wherein each of the plurality of layers is associated with a respective layer identifier [e.g., which is signaled in the data stream; E.g., each of all layers of the data stream may be associated with a layer identifier which is unique, that is, e.g., the, the domain in which the layer identifiers are defined may represent the count of layers of the data stream], and wherein the layer identification syntax element indicates the layer identifier of the respective layer [e.g., by signaling a value corresponding to a value of the layer identifier of the respective layer],

[0919] 17. Apparatus according to embodiment 15, configured for deriving, from the data stream, for each of the plurality of layers, a set of layer-specific property indicators [e.g., a set of syntax elements, each of which indicates a property of the respective layer; E.g., the set comprises one or more or all of (e.g., any combination out of) layerjd, quality_id(4), dependency_id(3); view_idx(8), depth_flag(1); or view_id(10)], each of the layer-specific property indicators indicating a property of the respective layer by signaling a value [e.g., the indicator having a value] out of a domain of the respective layer-specific property indicator, [e.g., which value represents the property of the respective layer] and

[0920] wherein the apparatus is configured for deriving, from the data stream, a layer identification syntax element, which signals the identification of the respective layer within the plurality of layers, wherein the layer identification syntax element indicates a value [or comprises, or consists of, a concatenation of values] for each layerspecific property indicator of the set of layer-specific property indicators, and identifying the respective layer based on the combination of values of the layerspecific property indicators.

[0921] 18. Apparatus according to embodiment 17, wherein, in the data stream, each of the layer-specific property indicators is represented by a [respective] predefined count [or quantity] of bits [e.g., to cover the (respective) domain of the respective layerspecific property indicator], and

[0922] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmwherein the sum of the predefined counts of bits of the layer-specific property indicators of the set of layer-specific property indicators equals the count of bits with which the layer identification syntax element is signaled.

[0923] 19. Apparatus according to embodiment 17, wherein the count of bits with which the layer identification syntax element is signaled in the data stream equals the sum of the counts of bits, which are required to represent the domains of the layer-specific property indicators of the set of layer-specific property indicators [e.g., when concatenating bitstrings for indicating the layer-specific property indicators], wherein the apparatus is configured to infer that [e.g., for one or more or all of the set of layer-specific property indicators] the counts of bits, which are required to represent the domains of the layer-specific property indicators are less than the counts of bits, which are used in the data stream to signal the layer-specific property indicators [e.g., the apparatus infers that only a portion of a maximum domain provided by the bits used for signaling a layer-specific property indicator is used, e.g. that the layerspecific property indicator may only assume a subset out of values provided by a bitlength of the layer-specific property indicator],

[0924] 20. Apparatus according to any of embodiments 17 to 19, configured for deriving the set of layer-specific property indicators from an indication [e.g., a syntax element] in the data stream [e.g., from the value of nnpfc_auxiliary_inp_idc],

[0925] 21. Apparatus according to any of embodiments 17 to 20, configured for deriving the domains of the layer-specific property indicators [the domains of values that can actually be assumed] based on an indication [e.g., a syntax element] in the data stream [e.g., based on the value of nnpfc_auxiliary_inp_idc, e.g., for one or more or all of the layer-specific property indicators],

[0926] 22. Apparatus according to embodiment 16, configured for deriving the indication indicating the count [or quantity] of layers of the set of layers from a syntax element contained in a first payload packet [e.g., a SEI NAL unit] of the data stream [e.g., a payload packed interspersed between or signaled ahead of coded video payload packets, e.g., VCL NALunits, of the data stream, which have encoded thereinto the picture], and deriving, for each of the set of layers, an identification of the respective layer among the plurality of layers of the data stream from a second payload packet of the data stream [e.g., a SEI NAL unit] [e.g., a payload packed interspersed

[0927] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmbetween or signaled ahead of coded video payload packets, e.g., VCL NALunits, of the data stream, which have encoded thereinto the picture],

[0928] 23. Apparatus according to embodiment 22, configured for

[0929] deriving, from the second payload packet, an indication [e.g., a syntax element, e.g., napm_mode_idc] indicating a set of layer-specific property indicators [e.g., identifying the set out of a plurality of layer-specific property indicators included in the data stream] [e.g., a set of syntax elements, each of which indicates a property of the respective layer; E.g., the set comprises one or more or all of (e.g., any combination out of) quality_id4, dependency_id3; view_idx8, depth_flag1; or view_id10], each of the layer-specific property indicators indicating a property of the respective layer by signaling a value [e.g., the indicator having a value] out of a domain of the respective layer-specific property indicator, [e.g., which value represents the property of the respective layer],

[0930] deriving, from the data stream, for each of the plurality of layers, the set of [one or more] layer-specific property indicators, and

[0931] for each of the set of layers, deriving the identification of the respective layer among the plurality of layers by

[0932] deriving, from the data stream, a layer identification syntax element, which signals the identification of the respective layer within the plurality of layers, wherein the layer identification syntax element indicates a value [or comprises, or consists of, a concatenation of values] for each layer-specific property indicator of the set of layer-specific property indicators, and identifying the respective layer based on the combination of values of the layer-specific property indicators.

[0933] 24. Apparatus according to embodiment 22, configured for

[0934] for each of the set of layers, deriving the identification of the respective layer among the plurality of layers by

[0935] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmderiving, from the second payload packet, an indication [e.g., a syntax element, e.g., napm_mode_idc] indicating a set of layer-specific property indicators [e.g., identifying the set out of a plurality of layer-specific property indicators included in the data stream] [e.g., a set of syntax elements, each of which indicates a property of the respective layer; E.g., the set comprises one or more or all of (e.g., any combination out of) quality_id(4), dependency_id(3); view_idx(8), depth_flag(1); or view_id(10)], each of the layer-specific property indicators indicating a property of the respective layer by signaling a value [e.g., the indicator having a value] out of a domain of the respective layer-specific property indicator, [e.g., which value represents the property of the respective layer], and

[0936] deriving, from the data stream, the set of [one or more] layer-specific property indicators, and

[0937] deriving, from the data stream, a layer identification syntax element, which signals the identification of the respective layer within the plurality of layers, wherein the layer identification syntax element indicates a value [or comprises, or consists of, a concatenation of values] for each layer-specific property indicator of the set of layer-specific property indicators, and identifying the respective layer based on the combination of values of the layer-specific property indicators.

[0938] 25. Apparatus according to any of embodiments 17 to 24, wherein the indication indicating the set of layer-specific property indicators is signaled by a syntax element [e.g., the layer identification syntax element], which differentiates between a plurality of modes including one or more of

[0939] a first mode according to which the set of layer-specific property indicators comprises, or consists of, a layer identifier,

[0940] a second mode according to which the set of layer-specific property indicators comprises, or consists of, a view identifier and / or a depth identifier, a third mode according to which the set of layer-specific property indicators comprises, or consists of, a layer identifier and one or more further layer-specific property indicators [e.g., view identifier and / or depth identifier].

[0941] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bm26. Apparatus according to embodiment 25, the apparatus is configured for deriving a further syntax element from the data steam, which indicates a bit-length of the syntax element in the data stream [e.g., a count of bits using which the syntax element is encoded in the data stream],

[0942] 27. Apparatus according to any of embodiments 15 to 26, configured for deriving the identification of the respective layer within the plurality of layers of the data stream by deriving, from the data stream, a layer identification syntax element, which identifies the respective layer by

[0943] signaling a layer identifier which is unique to the respective layer [e.g., by signaling a value corresponding to a value of the layer identifier of the respective layer], wherein each of the plurality of layers is associated with a respective layer identifier [e.g., which is signaled in the data stream; E.g., each of all layers of the data stream may be associated with a layer identifier which is unique, that is, e.g., the, the domain in which the layer identifiers are defined may represent the count of layers of the data stream], and / or

[0944] indicating [or signaling] a value [or comprises, or consists of, a concatenation of values] for each of a set of layer-specific property indicators [e.g., a set of syntax elements, each of which indicates a property of the respective layer; E.g., the set comprises one or more or all of (e.g., any combination out of) quality_id(4), dependency_id(3); view_idx(8), depth_flag(1); or view_id(10)], each of the layer-specific property indicators indicating a property of the respective layer by signaling a value [e.g., the indicator having a value] out of a domain of the respective layer-specific property indicator, [e.g., which value represents the property of the respective layer],

[0945] wherein the apparatus is configured for deriving a further syntax element from the data steam, which indicates a bit-length of the layer-identification syntax element in the data stream [e.g., a count of bits using which the layer-identification syntax element is encoded in the data stream],

[0946] 28. Apparatus according to any of the embodiments 11 to 27, wherein the apparatus is configured for obtaining the filter input data in form of a tensor comprising [or consisting of] a plurality of channels, each of which provides an input channel for the

[0947] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmneural network [e.g., the channels representing one dimension of the input tensor] by

[0948] determining a first number of first channels of the plurality of channels based on sample arrays of the reconstructed picture, and

[0949] determining, for each of the auxiliary pictures, a second number of second channels of the plurality of channels [e.g., the count of the second number may vary between the auxiliary pictures or may be equal for each of them],

[0950] 29. Apparatus according to embodiment 28, configured for determining a count of the second number of second channels based on the count of auxiliary pictures [of the set of auxiliary pictures] and based on an indication 64, which indicates a manner of deriving the second channels [e.g., an indication, which indicates a number of channels to be derived based on the auxiliary pictures and optionally, an order thereof],

[0951] 30. Apparatus according to embodiment 29, configured for deriving, from the data stream, the indication [e.g., nnpfc_inp_order, see Table 21] of the manner of deriving the second channels.

[0952] 31. Apparatus according to embodiment 29 or 30,

[0953] wherein the second number of second channels is equal for all of the auxiliary pictures, or

[0954] wherein the apparatus is configured for deriving, from the data stream, for each of the auxiliary pictures a respective indication [e.g., nnpfc_aux_inp_order_idc[ i ]] of a manner of deriving the second channels, and for deriving the second number of second channels individually for each of the auxiliary pictures.

[0955] 32. Apparatus according to any of embodiments 11. to 29, wherein the apparatus is configured for obtaining the filter input data in form of a tensor comprising [or consisting of] a plurality of channels, each of which provides an input channel for the neural network [e.g., the channels representing one dimension of the input tensor],

[0956] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmwherein the apparatus is configured for deriving the plurality of channels to comprise [or consist of] [e.g., in deriving the input filter data, the apparatus selects, e.g., in dependence on a syntax element derived from the data stream (e.g., nnpfc_inp_order_idc), one out of a plurality of modes, the plurality of modes comprising one or more or all of the following]

[0957] [e.g., in case nnpfc_inp_order_idc = 0] a first channel comprising [or consisting of] a luma sample array derived from a luma sample array of the reconstructed picture and, for each of the auxiliary pictures, a second channel comprising [or consisting of] an auxiliary luma sample array [e.g., derived from a luma sample array of the auxiliary picture],

[0958] or

[0959] [e.g., in case nnpfc_inp_order_idc = 1] a first first channel comprising [or consisting of] a first chroma sample array derived from a first chroma sample array of the reconstructed picture, a second first channel comprising [or consisting of] a second chroma sample array derived from a second chroma sample array of the reconstructed picture, and, for each of the auxiliary pictures,

[0960] a first second channel comprising [or consisting of] a first auxiliary sample array [e.g., a first auxiliary sample array] [e.g., derived from a first chroma sample array of the auxiliary picture or default values], and a second second channel comprising [or consisting of] a second auxiliary sample array [e.g., a second auxiliary sample array] [e.g., derived from a second chroma sample array of the auxiliary picture],

[0961] or

[0962] [e.g., in case nnpfc_inp_order_idc = 2] a first channel comprising [or consisting of] a luma sample array derived from a luma sample array of the reconstructed picture and, for each of the auxiliary pictures, a second channel comprising [or consisting of] an auxiliary luma sample array [e.g., derived from a luma sample array of the auxiliary picture], and

[0963] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bma first first channel comprising [or consisting of] a first chroma sample array derived from a first chroma sample array of the reconstructed picture, a second first channel comprising [or consisting of] a second chroma sample array derived from a second chroma sample array of the reconstructed picture, and, for each of the auxiliary pictures,

[0964] a first second channel comprising [or consisting of] a first auxiliary sample array [e.g., a first auxiliary sample array] [e.g., derived from a first chroma sample array of the auxiliary picture], and a second second channel comprising [or consisting of] a second auxiliary sample array [e.g., a second auxiliary sample array] [e.g., derived from a second chroma sample array of the auxiliary picture],

[0965] or

[0966] [e.g., in case nnpfc_inp_order_idc = 3] four first channels, each comprising [or consisting of] a luma sample array derived from a luma sample array of the reconstructed picture and, for each of the auxiliary picture [e.g., for each of the set of layers], four second channels, each comprising [or consisting of] an auxiliary luma sample array [e.g., derived from a luma sample array of the auxiliary picture], and

[0967] a first first channel comprising [or consisting of] a first chroma sample array derived from a first chroma sample array of the reconstructed picture, a second first channel comprising [or consisting of] a second chroma sample array derived from a second chroma sample array of the reconstructed picture, and, for each of the auxiliary pictures,

[0968] a first second channel comprising [or consisting of] a first auxiliary sample array [e.g., a first auxiliary sample array] [e.g., derived from a first chroma sample array of the auxiliary picture], and a second second channel comprising [or consisting of] a second auxiliary sample array [e.g., a second auxiliary sample array] [e.g., derived from a second chroma sample array of the auxiliary picture],

[0969] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bm[e.g., wherein a chroma format of the reconstructed picture and the pictures carried in the set of layers is YCrCb 4:2:0],

[0970] 33. Apparatus according to any of embodiments 10 to 32, wherein the input channels have equal dimensions.

[0971] 34. Apparatus according to any of the embodiments 1 to 33, configured for

[0972] obtaining the filter input data in form of a tensor comprising [or consisting of] a plurality of channels, each of which provides an input channel for the neural network [e.g., the channels representing one dimension of the input tensor], wherein the apparatus is configured for, in obtaining the filter input data,

[0973] obtaining one or more first channels of the plurality of channels based on the reconstructed picture and obtaining one or more second channels of the plurality of channels based on a set of auxiliary pictures, each of the auxiliary pictures being represented by one or more of the one or more auxiliary sample arrays [e.g., a luma and tow chroma sample arrays] [E.g., the apparatus is configured for reconstructing the auxiliary pictures from the data stream, e.g., from respective layers of the data stream],

[0974] deriving, from the data stream, for each of the auxiliary pictures, an indication [e.g., a syntax element, e.g., nnpfc_aux_inp_order_idc] which indicates a manner of deriving one or more of the second channels based on the respective auxiliary picture [e.g., by indicating one or more of the one or more auxiliary sample arrays of the respective auxiliary picture based on which the filter input data is to be derived],

[0975] 35. Apparatus according to embodiment 34, configured for, for each of the auxiliary pictures, deriving one or more of the second channels based on a set of auxiliary sample arrays [e.g., a set may comprise one or more ...] out of the auxiliary sample arrays of the respective auxiliary picture, which set of auxiliary sample arrays of the respective auxiliary picture is indicated to be used for the filter input data according to the indication which indicates the manner of deriving the one or more second channels based on the respective auxiliary picture.

[0976] 36. Apparatus according to any of the embodiments 1 to 35, configured for

[0977] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmobtaining one or more first channels of the plurality of channels based on the reconstructed picture and obtaining one or more second channels of the plurality of channels based on one or more auxiliary pictures, each of the auxiliary pictures being represented by one or more of the one or more auxiliary sample arrays [e.g., a luma and tow chroma sample arrays] [E.g., the apparatus is configured for reconstructing the auxiliary pictures from the data stream, e.g., from respective layers of the data stream],

[0978] deriving, from the data stream [e.g., from one or more syntax elements], an indication of a set of layers [e.g., comprising one or more layers, e.g., the one or more further layers] out of a plurality of layers of the data stream, each of the set of layers carrying one of the auxiliary pictures, and

[0979] deriving, from the data stream, for each of the layers of the set of layers, an indication [e.g., a syntax element, e.g., nnpfc_aux_inp_order_idc] which indicates a manner of deriving one or more of the second channels based on the auxiliary picture carried in the respective layer [e.g., by indicating a selection out of the one or more auxiliary sample arrays of the respective auxiliary picture based on which the filter input data is to be derived and / or by indicating an order in which channels of the input tensor are to be arranged],

[0980] 37. Apparatus according to embodiment 36, configured for, for each of the layers, deriving one or more of the second channels based on a set of auxiliary sample arrays [e.g., a set may comprise one or more ...] out of the auxiliary sample arrays of the respective auxiliary picture, which set of auxiliary sample arrays of the respective auxiliary picture is indicated to be used for the filter input data according to the indication which indicates the manner of deriving the one or more second channels based on the auxiliary picture of the respective layer.

[0981] 38. Apparatus according to any of the embodiments 1 to 37, configured for

[0982] obtaining one or more first channels of the plurality of channels based on the reconstructed picture and obtaining one or more second channels of the plurality of channels based on one or more auxiliary pictures, each of the auxiliary pictures being represented by one or more of the one or more auxiliary sample arrays [e.g.,

[0983] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bma luma and tow chroma sample arrays] [E.g., the apparatus is configured for reconstructing the auxiliary pictures from the data stream, e.g., from respective layers of the data stream],

[0984] deriving, from the data stream [e.g., from one or more syntax elements], an indication of a set of layers [e.g., comprising one or more layers, e.g., the one or more further layers] out of a plurality of layers of the data stream, each of the set of layers carrying one of the auxiliary pictures, and

[0985] deriving, from the data stream, an indication [e.g., a syntax element, e.g., nnpfc_aux_layers_same_order_idc_flag] which indicates whether a manner of deriving the second channels based on the set of auxiliary pictures is selected individually for each of the auxiliary pictures [e.g., in terms of number of sample arrays per layer and / or an order among the sample arrays in the input tensor] [e.g., or if the manner is set individually for each of the layers], and

[0986] if the indication which indicates that the manner of deriving the second channels is selected individual for each of the auxiliary pictures,

[0987] deriving, from the data stream, for each of the auxiliary pictures [e.g., for each of the set of layers], an indication [e.g., a syntax element, e.g., nnpfc_aux_inp_order_idc] which indicates a manner of deriving the filter input data based on the respective auxiliary picture [e.g., by indicating a selection out of the one or more auxiliary sample arrays of the respective auxiliary picture based on which the filter input data is to be derived and / or by indicating an order in which channels of the input tensor are to be arranged], or

[0988] deriving, from the data stream, for each of the auxiliary pictures, an indication [e.g., a syntax element, e.g., nnpfc_aux_inp_order_idc] which indicates a manner of deriving one or more of the second channels based on the respective auxiliary picture [e.g., by indicating a selection out of the one or more auxiliary sample arrays of the respective auxiliary picture based on which the filter input data is to be derived and / or by indicating an order in which channels of the input tensor are to be arranged].

[0989] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bm39. Apparatus according to embodiment 38, configured for,

[0990] if the indication indicates that the manner of deriving the second channels based on the auxiliary pictures is not selected individually for each of the auxiliary pictures, deriving the manner of deriving the second channels from an indication in the data stream [e.g., a syntax element, e.g., nnpfc_inp_order_idc], which indicates a manner of deriving the first channels based on the reconstructed picture.

[0991] 40. Apparatus according to any of embodiments 36 to 39, wherein the indication, which indicates the manner of deriving the second channels, indicates, for the respective auxiliary picture, for each of one or more sample array types [e.g., luma and two chroma types], a number of channels to be derived for the plurality of channels of the input data, [e.g., the apparatus derives, for each of the one or more sample array types, the respective number of channels based on a sample array of the respective sample array type of the respective auxiliary picture. E.g., if not existing, based on a substitute sample array.].

[0992] [E.g, a count (or quantity) of the number of channels depends on the indication which indicates the manner of deriving the second channels] [e.g., the count of channels derived from a auxiliary sample array of a first one of the layers may differ from a count of channels derived from a auxiliary sample array of a second one of the layers, the auxiliary sample array of the second layer having the same type as the auxiliary sample array of the first layer (e.g., luma or chroma)].

[0993] 41. Apparatus according to embodiment 40, configured for deriving an order among the channels derived for the respective auxiliary picture in dependence on the indication which indicates the manner of deriving the second channels.

[0994] 42. Apparatus according to embodiment 40 or 41 , wherein the indication which indicates the manner of deriving the second channels differentiates between a plurality of modes including one or more or all of the following modes:

[0995] a first mode [e.g., nnpfc_auxiliary_inp_order_idc = 0] according to which the apparatus derives, for the respective auxiliary picture, one [e.g., exactly one] of the channels comprising [or consisting of] an auxiliary luma sample array [e.g., derived from a luma sample array of the auxiliary picture],

[0996] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bma second mode [e.g., nnpfc_auxiliary_inp_order_idc = 1] according to which the apparatus derives, for the respective auxiliary picture, one [e.g., exactly one] of the channels comprising [or consisting of] a first auxiliary chroma sample array [e.g., derived from a first chroma sample array of the auxiliary picture], and a further one [e.g., exactly one] of the channels comprising [or consisting of] a second auxiliary chroma sample array [e.g., derived from a second chroma sample array of the auxiliary picture],

[0997] a third mode [e.g., nnpfc_auxiliary_inp_order_idc = 2] according to which the apparatus derives, for the respective auxiliary picture, one [e.g., exactly one] of the channels comprising [or consisting of] an auxiliary luma sample array [e.g., derived from a luma sample array of the auxiliary picture], a further one [e.g., exactly one] of the channels comprising [or consisting of] a first auxiliary chroma sample array [e.g., derived from a first chroma sample array of the auxiliary picture], and an even further one [e.g., exactly one] of the channels comprising [or consisting of] a second auxiliary chroma sample array [e.g., derived from a second chroma sample array of the auxiliary picture],

[0998] a fourth mode [e.g., nnpfc_auxiliary_inp_order_idc = 3] according to which the apparatus derives, for the respective auxiliary picture, four of the channels, each of which comprises [or consists of] an auxiliary luma sample array [e.g., derived from a luma sample array of the of the auxiliary picture], and a further one [e.g., exactly one] of the channels comprising [or consisting of] a first auxiliary chroma sample array [e.g., derived from a first chroma sample array of the auxiliary picture], and an even further one [e.g., exactly one] of the channels comprising [or consisting of] a second auxiliary chroma sample array [e.g., derived from a second chroma sample array of the auxiliary picture],

[0999] 43. Apparatus according to any of the embodiments 1 to 42, wherein the picture belongs to a coded video sequence [e.g., CLVS] of a sequence of coded video sequences of the data stream, and wherein the apparatus is configured for

[1000] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmderiving, from the data stream, an indication of a number [or count or quantity] of time frames for obtaining the filter input data for filtering the reconstructed picture, and

[1001] if the indication of the number of time frames indicates to obtain the filter input data using, in addition to the time frame to which the picture [the current picture to be coded] belongs, a further picture of a further time frame, and

[1002] if the further picture belongs to a preceding coded video sequence preceding the coded video sequence of the picture [e.g., if the further picture precedes the first picture of the coded video sequence], in dependence on an indication signaled in the data stream [e.g., deriving, from the data stream, an indication, e.g., a syntax element, which indicates whether pictures of a coded video sequence preceding a coded video sequence to which the picture belongs can be used for deriving the filter input data, and if the indication indicates that pictures of a coded video sequence preceding a coded video sequence to which the picture belongs cannot be used for deriving the filter input data], substitute the one of the one or more auxiliary sample arrays belonging to the respective time frame by

[1003] a corresponding one [e.g., in terms of layer and data type (luma, chroma, depth, ...)] of the auxiliary sample arrays belonging to the first or the last time frame of the coded video sequence, or

[1004] a default sample array [e.g., zeros],

[1005] and / or

[1006] if the further picture belongs to a succeeding coded video sequence succeeding the coded video sequence of the picture [e.g., if the further picture precedes the first picture of the coded video sequence], in dependence on an indication signaled in the data stream [e.g., deriving, from the data stream, an indication, e.g., a syntax element, which indicates whether pictures of a coded video sequence suceeding a coded video sequence to which the picture belongs can be used for deriving the filter input data, and if the indication indicates that pictures of a coded video sequence

[1007] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmsucceeding a coded video sequence to which the picture belongs cannot be used for deriving the filter input data], substitute the one of the one or more auxiliary sample arrays belonging the respective time frame by

[1008] a corresponding one [e.g., in terms of layer and data type (luma, chroma, depth, ...)] of the auxiliary sample arrays belonging to the first or the last time frame of the coded video sequence, or

[1009] a default sample array [e.g., zeros],

[1010] 44. Apparatus according to any of the embodiments 1 to 43, configured for

[1011] obtaining the filter input data in form of a tensor comprising [or consisting of] a plurality of channels, each of which provides an input channel for the neural network [e.g., the channels representing one dimension of the input tensor] by

[1012] obtaining one or more first channels of the plurality of channels based on the sample arrays of the reconstructed picture [e.g., obtaining each of one or more first channels of the plurality of channels based on one of the sample arrays of the reconstructed picture], and

[1013] obtaining one or more second channels of the plurality of channels based on one or more auxiliary pictures, each of the auxiliary pictures being represented by one or more of the one or more auxiliary sample arrays [e.g., a luma and two chroma sample arrays] [E.g., the apparatus is configured for reconstructing the auxiliary pictures from the data stream, e.g., from respective layers of the data stream], and

[1014] wherein the apparatus is configured for deriving, for each of the one or more auxiliary pictures, for each of one or more sample array types [e.g., luma and two chroma types], a number [e.g., a count, e.g., a predefined count or a count signaled individually for each of the auxiliary pictures] of channels of the plurality of channels [e.g., the predefined count equals the count of sample arrays of the reconstructed picture, e.g., three] by

[1015] if the respective auxiliary picture comprises an auxiliary sample array of the respective sample array type, deriving the number of channels based on [or

[1016] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmfrom] the auxiliary sample array, and if the respective auxiliary picture misses [or does not comprise] an auxiliary sample array of the respective sample array type [e.g., if an auxiliary picture has a chroma format which is different from the chroma format of the reconstructed picture, e.g., if the auxiliary picture has no chroma components], deriving the number of channels using sample values of a predetermined value [e.g. deriving the channel using an auxiliary sample array consisting of sample values of the predefined value],

[1017] 45. Apparatus according to embodiment 44, configured for deriving the predetermined value from the data stream [e.g., deriving a syntax element from the data stream, which indicates the predetermined value, e.g., nnpfc_aux_default_chroma_sample_mode_idc].

[1018] 46. Apparatus according to any of the preceding embodiment 1 to 45, configured for

[1019] obtaining the filter input data in form of a tensor comprising [or consisting of] a plurality of channels, each of which provides an input channel for the neural network [e.g., the channels representing one dimension of the input tensor] by

[1020] obtaining one or more first channels of the plurality of channels based on the sample arrays of the reconstructed picture, the sample arrays of the reconstructed picture comprising a luma sample array and two chroma sample arrays, and

[1021] obtaining one or more second channels of the plurality of channels based on one or more auxiliary pictures, each of the auxiliary pictures being represented by an auxiliary luma sample array and two auxiliary chroma sample array [e.g., which are part of the one or more auxiliary sample arrays], wherein the apparatus is configured for deriving one of the second channels based on one of the auxiliary chroma sample arrays by

[1022] if the count [or quantity] of sample values of the auxiliary chroma sample array is less than the count [or quantity] of sample values of the chroma sample array of the reconstructed picture [e.g., if the chroma format of the respective auxiliary picture is different from 4:4:4], repeating [or duplicating] sample values of the auxiliary chroma sample array [e.g. so that all of the

[1023] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmfirst and second channels have the same count of sample values, e.g., the same dimension], or

[1024] determining a first ratio between the width of the one auxiliary chroma sample array and the width of the sample arrays of the reconstructed picture [e.g., the latter have equal widths], and determining a second ratio between the height of the one auxiliary chroma sample array and the height of the sample arrays of the reconstructed picture [e.g., the latter have equal heights], and repeating [or duplicating] sample values of the auxiliary chroma sample array according to the first ratio and the second ratio [e.g. so that all of the first and second channels have the same count of sample values, e.g., the same dimension],

[1025] 47. Apparatus according to any of the preceding embodiment 1 to 46, configured for

[1026] obtaining the filter input data in form of a tensor comprising [or consisting of] a plurality of channels, each of which provides an input channel for the neural network [e.g., the channels representing one dimension of the input tensor] by

[1027] obtaining one or more first channels of the plurality of channels based on the reconstructed picture, the reconstructed picture comprising a luma sample array, and

[1028] obtaining a second channel of the plurality of channels based on an auxiliary picture, the auxiliary picture comprising an auxiliary luma sample array [e.g., which are part of the one or more auxiliary sample arrays], wherein the apparatus is configured for deriving the second channel based on the auxiliary luma sample array by

[1029] if the count [or quantity] of sample values of the auxiliary luma sample array is less than the count [or quantity] of sample values of the luma sample array of the reconstructed picture [e.g., the auxiliary picture may be a depth picture having a lower resolution], repeating [or duplicating] sample values of the auxiliary luma sample array [e.g. so that all of the first and second channels have the same count of sample values, e.g., the same dimension], or

[1030] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmdetermining a first ratio between the width of the auxiliary luma sample array and the width of the luma sample array of the reconstructed picture [e.g., the latter have equal widths], and determining a second ratio between the height of the auxiliary luma sample array and the height of the luma sample array of the reconstructed picture [e.g., the latter have equal heights], and repeating [or duplicating] sample values of the auxiliary luma sample array according to the first ratio and the second ratio [e.g. so that all of the first and second channels have the same count of sample values, e.g., the same dimension],

[1031] 48. Apparatus according to any of the embodiments 1 to 47, configured for

[1032] obtaining the filter input data in form of a tensor comprising [or consisting of] a plurality of channels, each of which provides an input channel for the neural network [e.g., the channels representing one dimension of the input tensor] by

[1033] obtaining one or more first channels of the plurality of channels based on the sample arrays of the reconstructed picture, and

[1034] obtaining one or more second channels of the plurality of channels based on one or more auxiliary pictures, each of the auxiliary pictures being represented by one or more of the one or more auxiliary sample arrays [e.g., a luma and two chroma sample arrays] [E.g., the apparatus is configured for reconstructing the auxiliary pictures from the data stream, e.g., from respective layers of the data stream],

[1035] wherein the apparatus is configured for deriving sample values of the first channels and the second channels at a common bit depth [e.g., the common bit depth equals the smallest or the highest bit depth among the bit depths of the sample arrays of the reconstructed picture and the auxiliary sample arrays, or the common bit depth is smaller than the smallest bit depth among the bit depths of the sample arrays of the reconstructed picture and the auxiliary sample arrays, or the common bit depth is higher than the highest bit depth among the bit depths of the sample arrays of the reconstructed picture and the auxiliary sample arrays].

[1036] 49. Apparatus according to any of the embodiments 1 to 48, configured for

[1037] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmobtaining the filter input data in form of a tensor comprising [or consisting of] a plurality of channels, each of which provides an input channel for the neural network [e.g., the channels representing one dimension of the input tensor] by

[1038] obtaining one or more first channels of the plurality of channels based on the sample arrays of the reconstructed picture, and

[1039] obtaining one or more second channels of the plurality of channels based on the one or more auxiliary sample arrays,

[1040] wherein the apparatus is configured for deriving one of the second channels from one of the auxiliary sample arrays by converting [or recalculating] sample values of the one auxiliary sample array to obtain a converted auxiliary sample array and deriving the second channel from the converted auxiliary sample array [e.g., by including the auxiliary sample array or a portion thereof into the one second channel] [E.g., the conversation of the sample values is performed using a function, e.g., a heavyside function, or a mulit-step function],

[1041] 50. Apparatus according to embodiment 49, configured for deriving a manner [e.g., a function or a mapping table] of converting the sample values of the one auxiliary array from the data stream [e.g., from an SEI message],

[1042] 51. Apparatus according to any of the embodiments 1 to 50, configured for deriving the filter input data by including into the filter input data [e.g., as additional channel, e.g., a third channel, further to the first and second channels] one or more or all of

[1043] an information on quantization parameter used in reconstructing the picture, and / or

[1044] one or more threshold values for an alpha mask [e.g., transparency mask] [e.g., which is part of the one or more auxiliary sample arrays]

[1045] a smallest and / or a highest depth value for a depth map [e.g., which is part of the one or more auxiliary sample arrays]

[1046] an indication of a mapping function for sample values of a depth map [e.g., which is part of the one or more auxiliary sample arrays], e.g., whether the mapping function is linear or non-linear or a step-wise linear approximation of the mapping function

[1047] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bman indication of a mapping function for sample values of a disparity map [e.g., which is part of the one or more auxiliary sample arrays], e.g., whether the mapping function is linear or non-linear or a step-wise linear approximation of the mapping function.

[1048] 52. Apparatus according to any of the embodiments 1 to 51.,

[1049] wherein the one or more auxiliary sample arrays comprise one or more of a depth map, a mask, and a transparency map; and / or

[1050] wherein the one or more auxiliary sample arrays comprise one or more sample arrays of a further picture [e.g., referred to as auxiliary picture], wherein

[1051] the data stream has encoded thereinto a multi-view video, wherein the picture belongs to a first view of the multi-view video, and wherein the further picture belongs to a second view of the multi-view video, or

[1052] the further picture has a different resolution than the picture.

[1053] 53. Apparatus according to any of the embodiments 1 to 52, wherein sample arrays of the reconstructed picture and the one or more auxiliary sample arrays have equivalent sizes.

[1054] 54. Apparatus according to any of the embodiments 1 to 53, wherein the reconstructed picture and any [e.g., all] auxiliary picture [e.g., color picture] being represented by the one or more auxiliary sample arrays have the same chroma format.

[1055] 55. Apparatus according to any of the embodiments 1 to 54, wherein the reconstructed picture and any [e.g., all] auxiliary picture [e.g., color picture] being represented by the one or more auxiliary sample array have the same bit depth of luma and / or chroma.

[1056] 56. Apparatus according to any of the embodiments 1 to 55, wherein the apparatus is configured for reconstructing the video from the data stream by block based predictive and transform based residual decoding by

[1057] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmdecoding prediction residual data of a residual block from the data stream

[1058] by use of context-adaptive variable length decoding by using

[1059] a first syntax element indicating a total number of non-zero transform coefficients in a transform block representing the residual block, and a trailing-one number, indicating a number of non-zero transform coefficients having an absolute value of one when traversing the coefficients along a scan order,

[1060] one or more second syntax elements indicating a sign of the non-zero transform coefficients having an absolute value of one when traversing the coefficients along the scan order,

[1061] one or more third syntax elements indicating a value of the non-zero transform coefficients except for the number of non-zero transform coefficients having an absolute value of one when traversing the coefficients along the scan order,

[1062] a fourth syntax element indicating a total number of zero-valued transform coefficient levels in the transform block from a firstly- encountered non-zero transform coefficient in the scan order onwards, and

[1063] one or more fifth syntax elements indicting positions of the non-zero transform coefficients along the scan order by indicating a number of consecutive zero-valued transform coefficients in the scan order between in the scan order consecutively encountered non-zero transform coefficients, or

[1064] by use of context-adaptive binary arithmetic decoding by

[1065] decoding a significance map which indicates positions of non-zero transform coefficients in a transform block representing the residual block by, in a forward scan traversing transform coefficients of the transform block, decoding a significance flag which indicates whether a non-zero transform coefficient is positioned at a current position, and, if so, and if the current position is not the last in the forward scan, decoding a last-significance flag which indicates whether the nonzero transform coefficient positioned at the current position is the last non-zero transform coefficient in the forward scan order, and

[1066] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmdecoding the non-zero transform coefficients’ values sequentially in a reverse scan order, reversing the forward scan order, or

[1067] decoding prediction residual data of a residual block from the data stream by use of context-adaptive binary arithmetic decoding of quantization indices of transform coefficients of a transform block representing the residual block and sequential dequantization of the quantization indices according to which a value of a current transform coefficient depends on a parity of quantization indices of previous quantization indices.

[1068] 57. Apparatus 10 for encoding a video 9 into a data stream, configured for

[1069] encoding 81 a picture 12 of the video into the data stream [e.g., the picture comprises or consists of one or more sample arrays, e.g., the picture comprises or consists of a luma sample array and two chroma sample arrays] [e.g., the picture belongs to one of a sequence of time frames of the video; E.g., encoding into the data stream, for each of a sequence of time frames, one or more pictures, and optionally, in addition to the one or more pictures, one or more sample arrays carrying spatially sampled information]; and

[1070] inserting 81, into the data stream, an indication 82 [e.g., a syntax structure] of one or more auxiliary sample arrays 15, the one or more auxiliary sample arrays being to be used for obtaining filter input data for filtering a reconstructed version of the picture, which filter input data is to be obtained based on the reconstructed picture and based on the one or more auxiliary sample arrays, and which filter input data is to be subjected to a neural network,

[1071] wherein the picture 12 and the one or more auxiliary sample arrays 15 belong to the same time frame [e.g., access unit] of the video.

[1072] 58. Apparatus according to embodiment 57, wherein the filter input data is represented by a tensor having the same dimension as an input [e.g., an input layer] of the neural network.

[1073] 59. Apparatus according to embodiment 57 or 58, wherein the data stream is a multilayer video data stream, and wherein picture belongs to a first layer of the data

[1074] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmstream and wherein the one or more auxiliary sample arrays belong to one of one or more further layers of the data stream [e.g., the one or more auxiliary sample array being distributed onto the one or more further layers, e.g., each belongs to a different one or all to the same one of the further layers] [E.g., the apparatus is configured for decoding the picture from the first layer, and for decoding the auxiliary sample arrays from the one or more further layers],

[1075] 60. Apparatus according to embodiment 60, wherein coded video sequences [e.g., CLVSs] of the first layer and the one or more further layers start at the same time frame and end at the same time frame.

[1076] 61. Apparatus according to any of embodiments 57 to 58, configured for encoding into the data stream

[1077] a sequence of primary pictures including the picture, the sequence of primary pictures representing the video [e.g., providing a representation of the video, e.g., from which the video is can be reconstructed, e.g., without further pictures or sample arrays included in the data stream], and

[1078] the one or more auxiliary sample arrays [e.g., as non-primary sample arrays, e.g. in payload packets interspersed between payload packets in which the primary pictures are encoded],

[1079] 62. Apparatus according to any of embodiments 57 to 58 or 61 , configured for encoding, into the data stream, a sequence of primary pictures including the picture by encoding the sequence of primary pictures in payload packets of a first type [e.g., VCL NAL units], and encoding the one or more auxiliary sample arrays into payload packets [e.g., NAL units] of a second type [e.g, the payload packets of the second type being interspersed between the payload packets of the first type],

[1080] 63. Apparatus according to any of the embodiments 57 to 62, wherein the picture belongs to a first time frame of the video, and wherein the filter input data is further to be obtained based on a further picture, the further picture belonging to a second time frame of the video, and based on one or more further auxiliary sample arrays, the one or more further auxiliary sample arrays belonging to the second time frame [e.g., the further picture is encoded into the first layer of the data stream, and the

[1081] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmone or more further auxiliary sample arrays belong to the one of one or more further layers of the data stream].

[1082] 64. Apparatus according to any of the embodiments 57 to 63, configured for inserting, into the data stream, a syntax element [e.g., nnpfc_auxiliary_inp_idc] which differentiates between a plurality of modes of obtaining the filter input data, wherein the plurality of modes includes a mode according to which the filter input data for the picture is to be obtained based on one or more auxiliary sample arrays belonging to the same time frame as the picture [and optionally, the plurality of modes includes one or more or all of a mode according to which no auxiliary information is used for obtaining the filter input data, e.g. the filter input data is obtained based on the reconstructed picture only, or based on a plurality of pictures belonging to different time frames of the video only; a mode according to which a quantization parameter for reconstructing the picture is to be used for obtaining the filter input data, and a mode according to which a text string is to be used for obtaining the filter input data],

[1083] 65. Apparatus according to any of the embodiments 57 to 64, wherein the filter input data is to be obtained in form of a tensor comprising [or consisting of] a plurality of channels, each of which provides an input channel for the neural network [e.g., the channels representing one dimension of the input tensor],

[1084] 66. Apparatus according to any of the embodiments 57 to 65, configured for

[1085] inserting, into the data stream [e.g., from one or more syntax elements], an indication of a set of auxiliary pictures, each of the auxiliary pictures being represented by one or more of the one or more auxiliary sample arrays [e.g., the indication indicates a count or quantity of the auxiliary pictures],

[1086] 67. Apparatus according to embodiment 66, wherein the set of auxiliary pictures is indicated by indicating a set of layers out of a plurality of layers of the data stream, each of the layers of the set of layers carrying one of the auxiliary pictures.

[1087] 68. Apparatus according to embodiment 66 or 67, configured for inserting, into the data stream, and indication of the count of auxiliary pictures of the set of auxiliary pictures.

[1088] 69. Apparatus according to any of the embodiments 57 to 68, configured for

[1089] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bminserting, into the data stream [e.g., from one or more syntax elements], an indication of a set of layers [e.g., comprising one or more layers, e.g., the one or more further layers] out of a plurality of layers of the data stream, each of the layers of the set of layers carrying an auxiliary picture [the auxiliary pictures carried by the set of layers may be referred to as set of auxiliary pictures], the auxiliary picture being represented by one or more of t the one or more auxiliary sample arrays.

[1090] 70. Apparatus according to embodiment 69, configured for

[1091] inserting, into the data stream, an indication [e.g., a syntax element, e.g., nnpfc_num_aux_layers_minus1] indicating the count [or quantity] of layers of the set of layers, and

[1092] inserting, into the data stream, for each of the set of layers, an identification of the respective layer within the plurality of layers of the data stream, [e.g., a syntax element, e.g., nnpfc_layer_target_id] [E.g., the syntax element identifies the layer by indicating a layer identifier, identifying the layer among the plurality of layers or by indicating a layer identifier which is indicated in the data stream for each of the plurality of layers. Alternatively, the layer identifier may be mapped by the apparatus to one of the plurality of layers, which may be uniquely identified by means of one or more properties signaled in the data stream for each of the plurality of layers],

[1093] 71. Apparatus according to embodiment 70, configured for inserting, into the data stream, a layer identification syntax element which signals the identification of the respective layer within the plurality of layers, wherein each of the plurality of layers is associated with a respective layer identifier [e.g., which is signaled in the data stream], and wherein the syntax element indicates the layer identifier of the respective layer [e.g., by signaling a value corresponding to a value of the layer identifier of the respective layer],

[1094] 72. Apparatus according to embodiment 70, configured for inserting, into the data stream, for each of the plurality of layers, a set of layer-specific property indicators [e.g., a set of syntax elements, each of which indicates a property of the respective layer; E.g., the set comprises one or more or all of (e.g., any combination out of) quality_id(4), dependency_id(3); view_idx(8), depth_flag(1); or view_id(10)], each of

[1095] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmthe layer-specific property indicators indicating a property of the respective layer by signaling a value [e.g., the indicator having a value] out of a domain of the respective layer-specific property indicator, [e.g., which value represents the property of the respective layer] and

[1096] wherein the apparatus is configured for inserting, into the data stream, a layer identification syntax element, which signals the identification of the respective layer within the plurality of layers, wherein the syntax element indicates a value [or comprises, or consists of, a concatenation of values] for each layer-specific property indicator of the set of layer-specific property indicators, and identifying the respective layer based on the combination of values of the layer-specific property indicators.

[1097] 73. Apparatus according to embodiment 72, wherein, in the data stream, each of the layer-specific property indicators is represented by a [respective] predefined count [or quantity] of bits [e.g., to cover the [respective] domain of the respective layerspecific property indicator], and

[1098] wherein the sum of the predefined counts of bits of the layer-specific property indicators of the set of layer-specific property indicators equals the count of bits with which the layer identification syntax element.

[1099] 74. Apparatus according to embodiment 72, wherein the count of bits with which the layer identification syntax element is signaled in the data stream equals the sum of the counts of bits, which are required to represent the domains of the layer-specific property indicators of the set of layer-specific property indicators [e.g., when concatenating bitstrings for indicating the layer-specific property indicators], wherein the [e.g., for one or more or all of the set of layer-specific property indicators] the counts of bits, which are required to represent the domains of the layer-specific property indicators are less than the counts of bits, which are used in the data stream to signal the layer-specific property indicators.

[1100] 75. Apparatus according to any of embodiments 72 to 74, configured for inserting the set of layer-specific property indicators into an indication [e.g., a syntax element] in the data stream [e.g., from the value of nnpfc_auxiliary_inp_idc].

[1101] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bm76. Apparatus according to any of embodiments 72 to 75, configured for inserting an indication of the domains of the layer-specific property indicators [the domains of values that can actually be assumed] in the data stream [e.g., based on the value of nnpfc_auxiliary_inp_idc, e.g., for one or more or all of the layer-specific property indicators],

[1102] 77. Apparatus according to embodiment 71, configured for inserting a syntax element indicating the count [or quantity] of layers of the set of layers from a syntax element contained into a first payload packet [e.g., a SEI NAL unit] of the data stream [e.g., a payload packed interspersed between or signaled ahead of coded video payload packets, e.g., VCL NALunits, of the data stream, which have encoded thereinto the picture], and inserting, for each of the set of layers, an identification of the respective layer among the plurality of layers of the data stream into a second payload packet of the data stream [e.g., a SEI NAL unit] [e.g., a payload packed interspersed between or signaled ahead of coded video payload packets, e.g., VCL NALunits, of the data stream, which have encoded thereinto the picture],

[1103] 78. Apparatus according to embodiment 77, configured for

[1104] inserting, into the second payload packet, an indication [e.g., a syntax element, e.g., napm_mode_idc] indicating a set of layer-specific property indicators [e.g., identifying the set out of a plurality of layer-specific property indicators included in the data stream] [e.g., a set of syntax elements, each of which indicates a property of the respective layer; E.g., the set comprises one or more or all of (e.g., any combination out of) quality_id(4), dependency_id(3); view_idx(8), depth_flag(1); or view_id(10)], each of the layer-specific property indicators indicating a property of the respective layer by signaling a value [e.g., the indicator having a value] out of a domain of the respective layer-specific property indicator, [e.g., which value represents the property of the respective layer],

[1105] inserting, into the data stream, for each of the plurality of layers, the set of [one or more] layer-specific property indicators, and

[1106] for each of the set of layers, inserting, into the data stream, a layer identification syntax element, which signals the identification of the respective layer within the plurality of layers, wherein the syntax element indicates a value [or comprises, or

[1107] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmconsists of, a concatenation of values] for each layer-specific property indicator of the set of layer-specific property indicators, and identifying the respective layer based on the combination of values of the layer-specific property indicators.

[1108] 79. Apparatus according to embodiment 77, configured for

[1109] for each of the set of layers,

[1110] inserting, into the second payload packet, an indication [e.g., a syntax element, e.g., napm_mode_idc] indicating a set of layer-specific property indicators [e.g., identifying the set out of a plurality of layer-specific property indicators included in the data stream] [e.g., a set of syntax elements, each of which indicates a property of the respective layer; E.g., the set comprises one or more or all of (e.g., any combination out of) quality_id(4), dependency_id(3); view_idx(8), depth_flag(1); or view_id(10)], each of the layer-specific property indicators indicating a property of the respective layer by signaling a value [e.g., the indicator having a value] out of a domain of the respective layer-specific property indicator, [e.g., which value represents the property of the respective layer], and

[1111] inserting, into the data stream, the set of [one or more] layer-specific property indicators, and

[1112] inserting, into the data stream, a layer identification syntax element, which signals the identification of the respective layer within the plurality of layers, wherein the syntax element indicates a value [or comprises, or consists of, a concatenation of values] for each layer-specific property indicator of the set of layer-specific property indicators, and identifying the respective layer based on the combination of values of the layer-specific property indicators.

[1113] 80. Apparatus according to any of embodiments 72 to 79, wherein the indication indicating the set of layer-specific property indicators is signaled by a syntax element [e.g., the layer identification syntax element], which differentiates between a plurality of modes including one or more of

[1114] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bma first mode according to which the set of layer-specific property indicators comprises, or consists of, a layer identifier,

[1115] a second mode according to which the set of layer-specific property indicators comprises, or consists of, a view identifier and / or a depth identifier, a third mode according to which the set of layer-specific property indicators comprises, or consists of, a layer identifier and one or more further layer-specific property indicators [e.g., view identifier and / or depth identifier],

[1116] 81. Apparatus according to embodiment 80, wherein the apparatus is configured for inserting a further syntax element into the data steam, which indicates a bit-length of the syntax element in the data stream [e.g., a count of bits using which the syntax element is encoded in the data stream],

[1117] 82. Apparatus according to any of embodiments 70 to 81, configured for inserting, into the data stream, a layer identification syntax element, which identifies the respective layer by

[1118] signaling a layer identifier which is unique to the respective layer [e.g., by signaling a value corresponding to a value of the layer identifier of the respective layer], wherein each of the plurality of layers is associated with a respective layer identifier [e.g., which is signaled in the data stream; E.g., each of all layers of the data stream may be associated with a layer identifier which is unique, that is, e.g., the, the domain in which the layer identifiers are defined may represent the count of layers of the data stream], and / or

[1119] indicating [or signaling] a value [or comprises, or consists of, a concatenation of values] for each of a set of layer-specific property indicators [e.g., a set of syntax elements, each of which indicates a property of the respective layer; E.g., the set comprises one or more or all of (e.g., any combination out of) quality_id(4), dependency_id(3); view_idx(8), depth_flag(1); or view_id(10)], each of the layer-specific property indicators indicating a property of the respective layer by signaling a value [e.g., the indicator having a value] out of a domain of the respective layer-specific property indicator, [e.g., which value represents the property of the respective layer],

[1120] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmwherein the apparatus is configured for inserting a further syntax element into the data steam, which indicates a bit-length of the layer-identification syntax element in the data stream [e.g., a count of bits using which the layer-identificaiton syntax element is encoded in the data stream],

[1121] 83. Apparatus according to any of the embodiments 68 or 70, wherein the filter input data is to be obtained in form of a tensor comprising [or consisting of] a plurality of channels, each of which provides an input channel for the neural network [e.g., the channels representing one dimension of the input tensor] by

[1122] determining a first number of first channels of the plurality of channels based on sample arrays of the reconstructed picture, and

[1123] determining, for each of the auxiliary pictures, a second number of second channels of the plurality of channels [e.g., the count of the second number may vary between the auxiliary pictures or may be equal for each of them],

[1124] 84. Apparatus according to embodiment 84, configured for inserting, into the data stream, the indication of the manner of deriving the second channels, wherein the indication of the manner of deriving the second channels.

[1125] 85. Apparatus according to embodiment 84,

[1126] wherein the second number of second channels is equal for all of the auxiliary pictures, or

[1127] wherein the apparatus is configured for inserting, into the data stream, for each of the auxiliary pictures a respective indication of a manner of deriving the second channels, and for deriving the second number of second channels individually for each of the auxiliary pictures.

[1128] 86. Apparatus according to any of embodiments 66. to 84, wherein the filter input data is to be obtained in form of a tensor comprising [or consisting of] a plurality of channels, each of which provides an input channel for the neural network [e.g., the channels representing one dimension of the input tensor],

[1129] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmwherein the apparatus is configured for inserting, into the data stream a syntax element which indicates a mode for obtaining the filter input data out of a plurality of modes, the plurality of modes comprising one or more of all of the following modes, according to which the plurality of channels to comprise [or consist of] [e.g., in deriving the input filter data, the apparatus selects, e.g., in dependence on a syntax element derived from the data stream (e.g., nnpfc_inp_order_idc), one out of a plurality of modes, the plurality of modes comprising one or more or all of the following]

[1130] [e.g., in case nnpfc_inp_order_idc = 0] a first channel comprising [or consisting of] a luma sample array derived from a luma sample array of the reconstructed picture and, for each of the auxiliary pictures, a second channel comprising [or consisting of] an auxiliary luma sample array [e.g., derived from a luma sample array of the auxiliary picture],

[1131] or

[1132] [e.g., in case nnpfc_inp_order_idc = 1] a first first channel comprising [or consisting of] a first chroma sample array derived from a first chroma sample array of the reconstructed picture, a second first channel comprising [or consisting of] a second chroma sample array derived from a second chroma sample array of the reconstructed picture, and, for each of the auxiliary pictures,

[1133] a first second channel comprising [or consisting of] a first auxiliary chroma sample array [e.g., derived from a first chroma sample array of the auxiliary picture], and a second second channel comprising [or consisting of] a second auxiliary chroma sample array [e.g., derived from a second chroma sample array of the auxiliary picture],

[1134] or

[1135] [e.g., in case nnpfc_inp_order_idc = 2] a first channel comprising [or consisting of] a luma sample array derived from a luma sample array of the reconstructed picture and, for each of the auxiliary pictures, a second

[1136] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmchannel comprising [or consisting of] an auxiliary luma sample array [e.g., derived from a luma sample array of the auxiliary picture], and

[1137] a first first channel comprising [or consisting of] a first chroma sample array derived from a first chroma sample array of the reconstructed picture, a second first channel comprising [or consisting of] a second chroma sample array derived from a second chroma sample array of the reconstructed picture, and, for each of the auxiliary picture,

[1138] a first second channel comprising [or consisting of] a first auxiliary chroma sample array [e.g., derived from a first chroma sample array of the auxiliary picture], and a second second channel comprising [or consisting of] a second auxiliary chroma sample array derived from a second chroma sample array of the auxiliary picture],

[1139] or

[1140] [e.g., in case nnpfc_inp_order_idc = 3] four first channels, each comprising [or consisting of] a luma sample array derived from a luma sample array of the reconstructed picture and, for each of the set of layers, four second channels, each comprising [or consisting of] an auxiliary luma sample array [e.g., derived from a luma sample array of the auxiliary picture], and

[1141] a first first channel comprising [or consisting of] a first chroma sample array derived from a first chroma sample array of the reconstructed picture, a second first channel comprising [or consisting of] a second chroma sample array derived from a second chroma sample array of the reconstructed picture, and, for each of the auxiliary pictures,

[1142] a first second channel comprising [or consisting of] a first auxiliary chroma sample array [e.g., derived from a first chroma sample array of the auxiliary picture ], and a second second channel comprising [or consisting of] a second auxiliary chroma sample array [e.g., derived from a second chroma sample array of the auxiliary picture ],

[1143] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bm[e.g., wherein a chroma format of the reconstructed picture and the pictures carried in the set of layers is YCrCb 4:2:0],

[1144] 87. Apparatus according to any of embodiments 65 to 86, wherein the input channels have equal dimensions.

[1145] 88. Apparatus according to any of the embodiments 57 to 87,

[1146] wherein the filter input data is to be obtained in form of a tensor comprising [or consisting of] a plurality of channels, each of which provides an input channel for the neural network [e.g., the channels representing one dimension of the input tensor] by

[1147] obtaining one or more first channels of the plurality of channels based on the reconstructed picture and obtaining one or more second channels of the plurality of channels based on a set of auxiliary pictures, each of the auxiliary pictures being represented by one or more of the one or more auxiliary sample arrays [e.g., a luma and tow chroma sample arrays] [E.g., the apparatus is configured for reconstructing the auxiliary pictures from the data stream, e.g., from respective layers of the data stream],

[1148] wherein the apparatus is configured for inserting, into the data stream, for each of the auxiliary pictures, an indication [e.g., a syntax element, e.g., nnpfc_aux_inp_order_idc] which indicates a manner of deriving one or more of the second channels based on the respective auxiliary picture [e.g., by indicating one or more of the one or more auxiliary sample arrays of the respective auxiliary picture based on which the filter input data is to be derived],

[1149] 89. Apparatus according to embodiment 88, wherein the indication which indicates the manner of deriving the one or more second channels based on the respective auxiliary picture indicates, for each of the auxiliary pictures, a set of auxiliary sample arrays [e.g., a set may comprise one or more ...] out of the auxiliary sample arrays of the respective auxiliary picture, which set of auxiliary sample arrays of the respective auxiliary picture is to be used for the filter input data.

[1150] 90. Apparatus according to any of the embodiments 57 to 89,

[1151] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmwherein the filter input data is to be obtained by obtaining one or more first channels of the plurality of channels based on the reconstructed picture and obtaining one or more second channels of the plurality of channels based on one or more auxiliary pictures, each of the auxiliary pictures being represented by one or more of the one or more auxiliary sample arrays [e.g., a luma and tow chroma sample arrays] [E.g., the apparatus is configured for reconstructing the auxiliary pictures from the data stream, e.g., from respective layers of the data stream],

[1152] wherein the apparatus is configured for

[1153] inserting, into the data stream [e.g., from one or more syntax elements], an indication of a set of layers [e.g., comprising one or more layers, e.g., the one or more further layers] out of a plurality of layers of the data stream, each of the set of layers carrying one of the auxiliary pictures, and

[1154] inserting, into the data stream, for each of the layers of the set of layers, an indication [e.g., a syntax element, e.g., nnpfc_aux_inp_order_idc] which indicates a manner of deriving one or more of the second channels based on the auxiliary picture carried in the respective layer [e.g., by indicating a selection out of the one or more auxiliary sample arrays of the respective auxiliary picture based on which the filter input data is to be derived and / or by indicating an order in which channels of the input tensor are to be arranged],

[1155] 91. Apparatus according to embodiment 90, wherein the indication which indicates the manner of deriving the one or more second channels based on the auxiliary picture of the respective layer indicates, for each of the layers, a set of auxiliary sample arrays [e.g., a set may comprise one or more ...] out of the auxiliary sample arrays of the respective auxiliary picture, which set of auxiliary sample arrays of the respective auxiliary picture is to be used for the filter input data.

[1156] 92. Apparatus according to any of the embodiments 57 to 91 , configured for

[1157] wherein the filter input data is to be obtained by obtaining one or more first channels of the plurality of channels based on the reconstructed picture and obtaining one or more second channels of the plurality of channels based on one or more auxiliary

[1158] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmpictures, each of the auxiliary pictures being represented by one or more of the one or more auxiliary sample arrays [e.g., a luma and tow chroma sample arrays] [E.g., the apparatus is configured for reconstructing the auxiliary pictures from the data stream, e.g., from respective layers of the data stream],

[1159] wherein the apparatus is configured for

[1160] inserting, into the data stream [e.g., from one or more syntax elements], an indication of a set of layers [e.g., comprising one or more layers, e.g., the one or more further layers] out of a plurality of layers of the data stream, each of the set of layers carrying one of the auxiliary pictures, and

[1161] inserting, into the data stream, an indication [e.g., a syntax element, e.g., nnpfc_aux_layers_same_order_idc_flag] which indicates whether a manner of deriving the second channels based on the set of auxiliary pictures is selected individual for each of the auxiliary pictures [e.g., in terms of number of sample arrays per layer and / or an order among the sample arrays in the input tensor] [e.g., or if the manner is set individually for each of the layers], and

[1162] if the indication which indicates that the manner of deriving the second channels is selected individual for each of the auxiliary pictures,

[1163] inserting, into the data stream, for each of the auxiliary pictures [e.g., for each of the set of layers], an indication [e.g., a syntax element, e.g., nnpfc_aux_inp_order_idc] which indicates a manner of deriving the filter input data based on the respective auxiliary picture [e.g., by indicating a selection out of the one or more auxiliary sample arrays of the respective auxiliary picture based on which the filter input data is to be derived and / or by indicating an order in which channels of the input tensor are to be arranged], or

[1164] inserting, into the data stream, for each of the auxiliary pictures, an indication [e.g., a syntax element, e.g., nnpfc_aux_inp_order_idc] which indicates a manner of deriving one or more of the second channels based on the respective auxiliary picture [e.g., by indicating a selection out of the one or more auxiliary sample arrays of the respective auxiliary picture based on

[1165] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmwhich the filter input data is to be derived and / or by indicating an order in which channels of the input tensor are to be arranged],

[1166] 93. Apparatus according to embodiment 92, configured for,

[1167] if the indication indicates that the manner of deriving the second channels based on the set of layers is not individual for each of the layers, inserting an indication in the data stream [e.g., a syntax element, e.g., nnpfc_inp_order_idc], which indicates a manner of deriving the first channels based on the reconstructed picture.

[1168] 94. Apparatus according to any of embodiments 90 to 93, wherein the indication, which indicates the manner of deriving the second channels, indicates, for the respective auxiliary picture, for each of one or more sample array types [e.g., luma and two chroma types], a number of channels to be derived for the plurality of channels of the input data, [e.g., the apparatus derives, for each of the one or more sample array types, the respective number of channels based on a sample array of the respective sample array type of the respective auxiliary picture. E.g., if not existing, based on a substitute sample array.].

[1169] [E.g, a count (or quantity) of the number of channels depends on the indication which indicates the manner of deriving the second channels] [e.g., the count of channels derived from a auxiliary sample array of a first one of the layers may differ from a count of channels derived from a auxiliary sample array of a second one of the layers, the auxiliary sample array of the second layer having the same type as the auxiliary sample array of the first layer (e.g., luma or chroma)].

[1170] 95. Apparatus according to embodiment 94, wherein the indication which indicates the manner of deriving the second channels indicates an order among the channels derived for the respective auxiliary picture.

[1171] 96. Apparatus according to embodiment 94 or 95, wherein the indication which indicates the manner of deriving the second channels differentiates between a plurality of modes including one or more or all of the following modes:

[1172] a first mode [e.g., nnpfc_auxiliary_inp_order_idc = 0] according to which the apparatus derives, for the respective auxiliary picture, one [e.g., exactly one]

[1173] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmof the channels comprising [or consisting of] an auxiliary luma sample array [e.g., derived from a luma sample array of the auxiliary picture],

[1174] a second mode [e.g., nnpfc_auxiliary_inp_order_idc = 1] according to which the apparatus derives, for the respective auxiliary picture, one [e.g., exactly one] of the channels comprising [or consisting of] a first auxiliary chroma sample array [e.g., derived from a first chroma sample array of the auxiliary picture], and a further one [e.g., exactly one] of the channels comprising [or consisting of] a second auxiliary chroma sample array [e.g., derived from a second chroma sample array of the auxiliary picture],

[1175] a third mode [e.g., nnpfc_auxiliary_inp_order_idc = 2] according to which the apparatus derives, for the respective auxiliary picture, one [e.g., exactly one] of the channels comprising [or consisting of] an auxiliary luma sample array [e.g., derived from a luma sample array of the auxiliary picture], a further one [e.g., exactly one] of the channels comprising [or consisting of] a first auxiliary chroma sample array [e.g., derived from a first chroma sample array of the auxiliary picture], and an even further one [e.g., exactly one] of the channels comprising [or consisting of] a second auxiliary chroma sample array [e.g., derived from a second chroma sample array of the auxiliary picture],

[1176] a fourth mode [e.g., nnpfc_auxiliary_inp_order_idc = 3] according to which the apparatus derives, for the respective auxiliary picture, four of the channels, each of which comprises [or consists of] an auxiliary luma sample array [e.g., derived from a luma sample array of the of the auxiliary picture], and a further one [e.g., exactly one] of the channels comprising [or consisting of] a first auxiliary chroma sample array [e.g., derived from a first chroma sample array of the auxiliary picture], and an even further one [e.g., exactly one] of the channels comprising [or consisting of] a second auxiliary chroma sample array [e.g., derived from a second chroma sample array of the auxiliary picture],

[1177] 97. Apparatus according to any of the embodiments 57 to 96, configured for

[1178] wherein the filter input data is to be obtained in form of a tensor comprising [or consisting of] a plurality of channels, each of which provides an input channel for the

[1179] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmneural network [e.g., the channels representing one dimension of the input tensor] by

[1180] obtaining one or more first channels of the plurality of channels based on the sample arrays of the reconstructed picture [e.g., obtaining each of one or more first channels of the plurality of channels based on one of the sample arrays of the reconstructed picture], and

[1181] obtaining one or more second channels of the plurality of channels based on one or more auxiliary pictures, each of the auxiliary pictures being represented by one or more of the one or more auxiliary sample arrays [e.g., a luma and two chroma sample arrays] [E.g., the apparatus is configured for reconstructing the auxiliary pictures from the data stream, e.g., from respective layers of the data stream], and

[1182] deriving, for each of the one or more auxiliary pictures, for each of one or more sample array types [e.g., luma and two chroma types], a number [e.g., a count, e.g., a predefined count or a count signaled individually for each of the auxiliary pictures] of channels of the plurality of channels [e.g., the predefined count equals the count of sample arrays of the reconstructed picture, e.g., three] by

[1183] if the respective auxiliary picture comprises an auxiliary sample array of the respective sample array type, deriving the number of channels based on [or from] the auxiliary sample array, and if the respective auxiliary picture misses [or does not comprise] an auxiliary sample array of the respective sample array type [e.g., if an auxiliary picture has a chroma format which is different from the chroma format of the reconstructed picture, e.g., if the auxiliary picture has no chroma components], deriving the number of channels using sample values of a predetermined value [e.g. deriving the channel using an auxiliary sample array consisting of sample values of the predefined value].

[1184] wherein the apparatus is configured for inserting an indication of the predetermined value into the data stream [e.g., deriving a syntax element from the data stream,

[1185] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmwhich indicates the predetermined value, e.g., nnpfc_aux_default_chroma_sample_mode_idc].

[1186] 98. Apparatus according to any of the embodiments 57 to 97, configured for

[1187] wherein the filter input data is to be obtained in form of a tensor comprising [or consisting of] a plurality of channels, each of which provides an input channel for the neural network [e.g., the channels representing one dimension of the input tensor] by

[1188] obtaining one or more first channels of the plurality of channels based on the sample arrays of the reconstructed picture, and

[1189] obtaining one or more second channels of the plurality of channels based on the one or more auxiliary sample arrays, and

[1190] deriving one of the second channels from one of the auxiliary sample arrays by converting [or recalculating] sample values of the one auxiliary sample array to obtain a converted auxiliary sample array and deriving the second channel from the converted auxiliary sample array [e.g., by including the auxiliary sample array or a portion thereof into the one second channel] [E.g., the conversation of the sample values is performed using a function, e.g., a heavyside function, or a mulit-step function],

[1191] wherein the apparatus is configured for inserting an indication of deriving a manner [e.g., a function or a mapping table] of converting the sample values of the one auxiliary array into the data stream [e.g., from an SEI message],

[1192] 99. Apparatus according to any of the embodiments 57 to 98, wherein [e.g., the apparatus is configured for guaranteeing that] sample arrays of the reconstructed picture and the one or more auxiliary sample arrays have equivalent sizes.

[1193] 100. Apparatus according to any of the embodiments 57 to 99, wherein [e.g., the apparatus is configured for guaranteeing that] the reconstructed picture and any [e.g., all] auxiliary picture [e.g., color picture] being represented by the one or more auxiliary sample array have the same chroma format.

[1194] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bm101. Apparatus according to any of the embodiments 57 to 100, wherein [e.g., the apparatus is configured for guaranteeing that] the reconstructed picture and any [e.g., all] auxiliary picture [e.g., color picture] being represented by the one or more auxiliary sample array have the same bit depth of luma and / or chroma.

[1195] 102. Apparatus according to any of the embodiments 57 to 101, wherein [e.g., the apparatus is configured for guaranteeing that] widths of the auxiliary sample arrays are a multiple of a width of the sample arrays of the reconstructed picture, and heights of the auxiliary sample arrays are a multiple of a height of the sample arrays of the reconstructed picture.

[1196] 103. Apparatus according to any of the embodiments 57 to 102, wherein the apparatus is configured for encoding the video into the data stream by block-based predictive coding and transform-based residual coding by

[1197] encoding prediction residual data of the residual block into the data stream

[1198] by use of context adaptive variable length coding by using

[1199] a first syntax element indicating a total number of non-zero transform coefficients in a transform block representing the residual block, and a trailing-one number, indicating a number of non-zero transform coefficients having an absolute value of one when traversing the coefficients along a scan order,

[1200] one or more second syntax elements indicating a sign of the non-zero transform coefficients having an absolute value of one when traversing the coefficients along the scan order, one or more third syntax elements indicating a value of the non-zero transform coefficients except for the number of non-zero transform coefficients having an absolute value of one when traversing the coefficients along the scan order,

[1201] a fourth syntax element indicating a total number of zero-valued transform coefficient levels in the transform block from a firstly- encountered non-zero transform coefficient in the scan order onwards, and

[1202] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmone or more fifth syntax elements indicting positions of the non-zero transform coefficients along the scan order by indicating a number of consecutive zero-valued transform coefficients in the scan order between in the scan order consecutively encountered non-zero transform coefficients, or

[1203] by use of context-adaptive binary arithmetic coding by

[1204] encoding a significance map which indicates positions of non-zero transform coefficients in a transform block representing the residual block by, in a forward scan traversing transform coefficients of the transform block, encoding a significance flag which indicates whether a non-zero transform coefficient is positioned at a current position, and, if so, and if the current position is not the last in the forward scan, encoding a last-significance flag which indicates whether the nonzero transform coefficient positioned at the current position is the last non-zero transform coefficient in the forward scan order, and encoding the non-zero transform coefficients’ values sequentially in a reverse scan order, reversing the forward scan order, or

[1205] encoding prediction residual data of a residual block into the data stream by use of context-adaptive binary arithmetic coding of quantization indices of transform coefficients of a transform block representing the residual block and sequential quantization of the transform coefficients to obtain the quantization indices, according to which a quantizer for quantizing a current transform coefficient depends on a parity of quantization indices of previous quantization indices.

[1206] 104. Method for reconstructing a video from a data stream 14, the method comprising:

[1207] reconstructing 21 a picture 12 of the video; and

[1208] filtering 61 the reconstructed picture 12’ using a neural network by obtaining 63 [e.g., deriving, forming] filter input data 65 [e.g., an input tensor] based on the reconstructed picture 12’ and based on one or more auxiliary sample arrays 15 [e.g., two-dimensional sample arrays, e.g., representing spatially sampled information], and subjecting [or inputting] the filter input data 65 to the neural network,

[1209] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmwherein the picture and the one or more auxiliary sample arrays belong to the same time frame 11* [e.g., access unit] of the video.

[1210] 105. Method for encoding a video 9 into a data stream 14, the method comprising:

[1211] encoding 81 a picture 12 of the video into the data stream; and

[1212] inserting 81, into the data stream, an indication 82 of one or more auxiliary sample arrays 15, the one or more auxiliary sample arrays being to be used for obtaining filter input data for filtering a reconstructed version of the picture, which filter input data is to be obtained based on the reconstructed picture and based on the one or more auxiliary sample arrays, and which filter input data is to be subjected to the neural network,

[1213] wherein the picture 12 and the one or more auxiliary sample arrays 15) belong to the same time frame of the video.

[1214] 106. A data stream 14 having encoded thereinto a video 9, wherein the data stream comprises:

[1215] a picture 12 of the video; and

[1216] an indication 82 [e.g., a syntax structure] of one or more auxiliary sample arrays 15, the one or more auxiliary sample arrays being to be used for obtaining filter input data for filtering a reconstructed version of the picture, which filter input data is to be obtained based on the reconstructed picture and based on the one or more auxiliary sample arrays, and which filter input data is to be subjected to the neural network,

[1217] wherein the picture 12 and the one or more auxiliary sample arrays 15) belong to the same time frame [e.g., access unit] of the video.

[1218] 107. A computer program for implementing the method of any of embodiments 104 or 105 when being executed on a computer or signal processor.

[1219]

[1220] alternatives

[1221] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmAlthough some aspects have been described as features in the context of an apparatus it is clear that such a description may also be regarded as a description of corresponding features of a method. Although some aspects have been described as features in the context of a method, it is clear that such a description may also be regarded as a description of corresponding features concerning the functionality of an apparatus. In particular, block diagrams illustrating the functionality of an apparatus may also be understood as illustration of a respective method comprising the functions described by the blocks of the block diagram as steps of the method.

[1222] The data signal or data stream provided by embodiments of the invention can be stored on a digital storage medium, e.g., a non-transitory or transitory digital storage medium, or can be transmitted on a transmission medium such as a wireless transmission medium or a wired transmission medium. In other words, further embodiments provide a computer product, e.g., a data stream product or bitstream product, e.g., a non-transitory digital storage medium, the computer product including, e.g., having stored thereon, the data signal or data stream according to any of the herein described embodiments.

[1223] Further embodiments provide a method for storing data, the method comprising a step of storing a data stream on a digital storage medium, e.g., a non-transitory digital storage medium, the data stream carrying the data. For example, the data stream is in accordance with any of the embodiments described herein. For example, has the data encoded thereinto according to any of the encoding methods described herein.

[1224] Further embodiments provide a method for transmitting a data stream of any of the embodiments described herein.

[1225] Features described with respect to an apparatus for receiving or processing a signal (e.g., receiver, decoder) are to be understood to serve as a description of a respective feature for an apparatus for providing the signal (e.g., an encoder) and vice versa, and as a feature of a respective signal, e.g., a data stream. In particular, the skilled person will understand that any information, e.g., a data type, structure, item, which is to be received by the receiver, or derived from the signal by the receiver, is inserted into the signal by a corresponding provider, and vice versa.

[1226] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmSome or all of the method steps may be executed by (or using) a hardware apparatus, like for example, a microprocessor, a programmable computer or an electronic circuit. In some embodiments, one or more of the most important method steps may be executed by such an apparatus.

[1227] The inventive encoded image signal can be stored on a digital storage medium or can be transmitted on a transmission medium such as a wireless transmission medium or a wired transmission medium such as the Internet. In other words, further embodiments provide a video bitstream product including the video bitstream according to any of the herein described embodiments, e.g. a digital storage medium having stored thereon the video bitstream.

[1228] Depending on certain implementation requirements, embodiments of the invention can be implemented in hardware or in software or at least partially in hardware or at least partially in software. The implementation can be performed using a digital storage medium, for example a floppy disk, a DVD, a Blu-Ray, a CD, a ROM, a PROM, an EPROM, an EEPROM or a FLASH memory, having electronically readable control signals stored thereon, which cooperate (or are capable of cooperating) with a programmable computer system such that the respective method is performed. Therefore, the digital storage medium may be computer readable.

[1229] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[1230] Generally, embodiments of the present invention can be implemented as a computer program product with a program code, the program code being operative for performing one of the methods when the computer program product runs on a computer. The program code may for example be stored on a machine readable carrier.

[1231] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[1232] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[1233] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmA further embodiment of the inventive methods is, therefore, a data carrier (or a digital storage medium, or a computer-readable medium) comprising, recorded thereon, the computer program for performing one of the methods described herein. The data carrier, the digital storage medium or the recorded medium are typically tangible and / or non-transitory.

[1234] A further embodiment of the inventive method is, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein. The data stream or the sequence of signals may for example be configured to be transferred via a data communication connection, for example via the Internet.

[1235] A further embodiment comprises a processing means, for example a computer, or a programmable logic device, configured to or adapted to perform one of the methods described herein.

[1236] A further embodiment comprises a computer having installed thereon the computer program for performing one of the methods described herein.

[1237] A further embodiment according to the invention comprises an apparatus or a system configured to transfer (for example, electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may, for example, be a computer, a mobile device, a memory device or the like. The apparatus or system may, for example, comprise a file server for transferring the computer program to the receiver.

[1238] In some embodiments, a programmable logic device (for example a field programmable gate array) may be used to perform some or all of the functionalities of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor in order to perform one of the methods described herein. Generally, the methods are preferably performed by any hardware apparatus.

[1239] The apparatus described herein may be implemented using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[1240] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bmThe methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

[1241] In the foregoing Detailed Description, it can be seen that various features are grouped together in examples for the purpose of streamlining the disclosure. This method of disclosure is not to be interpreted as reflecting an intention that the claimed examples require more features than are expressly recited in each claim. Rather, as the following claims reflect, subject matter may lie in less than all features of a single disclosed example. Thus the following claims are hereby incorporated into the Detailed Description, where each claim may stand on its own as a separate example. While each claim may stand on its own as a separate example, it is to be noted that, although a dependent claim may refer in the claims to a specific combination with one or more other claims, other examples may also include a combination of the dependent claim with the subject matter of each other dependent claim or a combination of each feature with other dependent or independent claims. Such combinations are proposed herein unless it is stated that a specific combination is not intended. Furthermore, it is intended to include also features of a claim to any other independent claim even if this claim is not directly made dependent to the independent claim.

[1242] The above described embodiments are merely illustrative for the principles of the present disclosure. It is understood that modifications and variations of the arrangements and the details described herein will be apparent to others skilled in the art. It is the intent, therefore, to be limited only by the scope of the pending patent claims and not by the specific details presented by way of description and explanation of the embodiments herein.

[1243] FH250102PEP-2025428381.DOCX PCT FILING VERSION / bm

Claims

1. Claims1. Apparatus (20) for reconstructing a video from a data stream (14), configured forreconstructing (21) a picture (12’) of the video; andfiltering (61) the reconstructed picture (12’) using a neural network by obtaining (63) filter input data (65) based on the reconstructed picture (12’) and based on one or more auxiliary sample arrays (15), and subjecting the filter input data (65) to the neural network,wherein the picture and the one or more auxiliary sample arrays belong to the same time frame (11*) of the video.

2. Apparatus according to claim 1, wherein the filter input data is represented by a tensor having the same dimension as an input of the neural network.

3. Apparatus according to claim 1 or 2, wherein the data stream is a multi-layer video data stream, and wherein the picture belongs to a first layer of the data stream and wherein the one or more auxiliary sample arrays belong to one of one or more further layers of the data stream.

4. Apparatus according to claim 3, wherein coded video sequences of the first layer and the one or more further layers start at the same time frame and / or end at the same time frame.

5. Apparatus according to claim 3, configured fordepending on an indication in the data stream, inferring that the coded video sequence of the first layer and coded video sequences of the one or more further layers start at the same time frame, and / ordepending on an indication in the data stream, inferring that the coded video sequence of the first layer and coded video sequences of the one or more further layers end at the same time frame.FH250102PEP-2025428381.DOCX PCT FILING VERSION / bm6. Apparatus according to any of claims 1 to 2, wherein the data stream has encoded thereinto a sequence of primary pictures including the picture, the sequence of primary pictures representing the video, and wherein the data stream further has encoded thereinto the one or more auxiliary sample arrays.

7. Apparatus according to any of claims 1 to 2 or 6, wherein the data stream has encoded thereinto a sequence of primary pictures including the picture, the sequence of primary pictures being encoded in payload packets of a first type, and wherein the data stream further has encoded thereinto the one or more auxiliary sample arrays in payload packets of a second type.

8. Apparatus according to any of the claims 1 to 7, wherein the picture belongs to a first time frame of the video, and wherein the apparatus is configured for reconstructing a further picture of the video, the further picture belonging to a second time frame of the video, and for obtaining the filter input data further based on the reconstructed further picture and based on one or more further auxiliary sample arrays, the one or more further auxiliary sample arrays belonging to the second time frame.

9. Apparatus according to any of the claims 1 to 8, configured for deriving, from the data stream, a syntax element which differentiates between a plurality of modes of obtaining the filter input data, wherein the plurality of modes includes a mode according to which the filter input data for the picture is to be obtained based on one or more auxiliary sample arrays belonging to the same time frame as the picture.

10. Apparatus according to any of the claims 1 to 9, wherein the apparatus is configured for obtaining the filter input data in form of a tensor comprising a plurality of channels, each of which provides an input channel for the neural network.

11. Apparatus according to any of the claims 1 to 10, configured forderiving, from the data stream, an indication of a set of auxiliary pictures, each of the auxiliary pictures being represented by one or more of the one or more auxiliary sample arrays.FH250102PEP-2025428381.DOCX PCT FILING VERSION / bm12. Apparatus according to claim 11 , wherein the set of auxiliary pictures is indicated by indicating a set of layers out of a plurality of layers of the data stream, each of the layers of the set of layers carrying one of the auxiliary pictures.

13. Apparatus according to claim 11 or 12, configured for deriving, from the data stream, an indication of the count of auxiliary pictures of the set of auxiliary pictures.

14. Apparatus according to any of the claims 1 to 13, configured forderiving, from the data stream, an indication of a set of layers out of a plurality of layers of the data stream, each of the layers of the set of layers carrying an auxiliary picture, the auxiliary picture being represented by one or more of the one or more auxiliary sample arrays.

15. Apparatus according to claim 14, configured for deriving the set or layers byderiving, from the data stream, an indication indicating the count of layers of the set of layers, andderiving, from the data stream, for each of the set of layers, an identification of the respective layer within the plurality of layers of the data stream.

16. Apparatus according to claim 15, configured for deriving, from the data stream, a layer identification syntax element which signals the identification of the respective layer within the plurality of layers, wherein each of the plurality of layers is associated with a respective layer identifier, and wherein the layer identification syntax element indicates the layer identifier of the respective layer.

17. Apparatus according to claim 15, configured for deriving, from the data stream, for each of the plurality of layers, a set of layer-specific property indicators, each of the layer-specific property indicators indicating a property of the respective layer by signaling a value out of a domain of the respective layer-specific property indicator, andwherein the apparatus is configured for deriving, from the data stream, a layer identification syntax element, which signals the identification of the respective layerFH250102PEP-2025428381.DOCX PCT FILING VERSION / bm129within the plurality of layers, wherein the layer identification syntax element indicates a value for each layer-specific property indicator of the set of layer-specific property indicators, and identifying the respective layer based on the combination of values of the layer-specific property indicators.

18. Apparatus according to claim 17, wherein, in the data stream, each of the layerspecific property indicators is represented by a predefined count of bits, andwherein the sum of the predefined counts of bits of the layer-specific property indicators of the set of layer-specific property indicators equals the count of bits with which the layer identification syntax element is signaled.

19. Apparatus according to claim 17, wherein the count of bits with which the layer identification syntax element is signaled in the data stream equals the sum of the counts of bits, which are required to represent the domains of the layer-specific property indicators of the set of layer-specific property indicators, wherein the apparatus is configured to infer that the counts of bits, which are required to represent the domains of the layer-specific property indicators are less than the counts of bits, which are used in the data stream to signal the layer-specific property indicators.

20. Apparatus according to any of claims 17 to 19, configured for deriving the set of layer-specific property indicators from an indication in the data stream.

21. Apparatus according to any of claims 17 to 20, configured for deriving the domains of the layer-specific property indicators based on an indication in the data stream.

22. Apparatus according to claim 16, configured for deriving the indication indicating the count of layers of the set of layers from a syntax element contained in a first payload packet of the data stream, and deriving, for each of the set of layers, an identification of the respective layer among the plurality of layers of the data stream from a second payload packet of the data stream.

23. Apparatus according to claim 22, configured forFH250102PEP-2025428381.DOCX PCT FILING VERSION / bm130deriving, from the second payload packet, an indication indicating a set of layerspecific property indicators, each of the layer-specific property indicators indicating a property of the respective layer by signaling a value out of a domain of the respective layer-specific property indicator,deriving, from the data stream, for each of the plurality of layers, the set of layerspecific property indicators, andfor each of the set of layers, deriving the identification of the respective layer among the plurality of layers byderiving, from the data stream, a layer identification syntax element, which signals the identification of the respective layer within the plurality of layers, wherein the layer identification syntax element indicates a value for each layer-specific property indicator of the set of layer-specific property indicators, and identifying the respective layer based on the combination of values of the layer-specific property indicators.

24. Apparatus according to claim 22, configured forfor each of the set of layers, deriving the identification of the respective layer among the plurality of layers byderiving, from the second payload packet, an indication indicating a set of layer-specific property indicators, each of the layer-specific property indicators indicating a property of the respective layer by signaling a value out of a domain of the respective layer-specific property indicator, andderiving, from the data stream, the set of layer-specific property indicators, andderiving, from the data stream, a layer identification syntax element, which signals the identification of the respective layer within the plurality of layers, wherein the layer identification syntax element indicates a value for each layer-specific property indicator of the set of layer-specific propertyFH250102PEP-2025428381.DOCX PCT FILING VERSION / bm131indicators, and identifying the respective layer based on the combination of values of the layer-specific property indicators.

25. Apparatus according to any of claims 17 to 24, wherein the indication indicating the set of layer-specific property indicators is signaled by a syntax element, which differentiates between a plurality of modes including one or more ofa first mode according to which the set of layer-specific property indicators comprises, or consists of, a layer identifier,a second mode according to which the set of layer-specific property indicators comprises, or consists of, a view identifier and / or a depth identifier, a third mode according to which the set of layer-specific property indicators comprises, or consists of, a layer identifier and one or more further layer-specific property indicators.

26. Apparatus according to claim 25, the apparatus is configured for deriving a further syntax element from the data steam, which indicates a bit-length of the syntax element in the data stream.

27. Apparatus according to any of claims 15 to 26, configured for deriving the identification of the respective layer within the plurality of layers of the data stream by deriving, from the data stream, a layer identification syntax element, which identifies the respective layer bysignaling a layer identifier which is unique to the respective layer, wherein each of the plurality of layers is associated with a respective layer identifier, and / orindicating a value for each of a set of layer-specific property indicators, each of the layer-specific property indicators indicating a property of the respective layer by signaling a value out of a domain of the respective layer-specific property indicator,wherein the apparatus is configured for deriving a further syntax element from the data steam, which indicates a bit-length of the layer-identification syntax element in the data stream.FH250102PEP-2025428381.DOCX PCT FILING VERSION / bm13228. Apparatus according to any of the claims 11 to 27, wherein the apparatus is configured for obtaining the filter input data in form of a tensor comprising a plurality of channels, each of which provides an input channel for the neural network bydetermining a first number of first channels of the plurality of channels based on sample arrays of the reconstructed picture, anddetermining, for each of the auxiliary pictures, a second number of second channels of the plurality of channels.

29. Apparatus according to claim 28, configured for determining a count of the second number of second channels based on the count of auxiliary pictures and based on an indication (64), which indicates a manner of deriving the second channels.

30. Apparatus according to claim 29, configured for deriving, from the data stream, the indication of the manner of deriving the second channels.

31. Apparatus according to claim 29 or 30,wherein the second number of second channels is equal for all of the auxiliary pictures, orwherein the apparatus is configured for deriving, from the data stream, for each of the auxiliary pictures a respective indication of a manner of deriving the second channels, and for deriving the second number of second channels individually for each of the auxiliary pictures.

32. Apparatus according to any of claims 11. to 29, wherein the apparatus is configured for obtaining the filter input data in form of a tensor comprising a plurality of channels, each of which provides an input channel for the neural network,wherein the apparatus is configured for deriving the plurality of channels to compriseFH250102PEP-2025428381.DOCX PCT FILING VERSION / bm133a first channel comprising a luma sample array derived from a luma sample array of the reconstructed picture and, for each of the auxiliary pictures, a second channel comprising an auxiliary luma sample array,ora first first channel comprising a first chroma sample array derived from a first chroma sample array of the reconstructed picture, a second first channel comprising a second chroma sample array derived from a second chroma sample array of the reconstructed picture, and, for each of the auxiliary pictures,a first second channel comprising a first auxiliary sample array, and a second second channel comprising a second auxiliary sample array,ora first channel comprising a luma sample array derived from a luma sample array of the reconstructed picture and, for each of the auxiliary pictures, a second channel comprising an auxiliary luma sample array, anda first first channel comprising a first chroma sample array derived from a first chroma sample array of the reconstructed picture, a second first channel comprising a second chroma sample array derived from a second chroma sample array of the reconstructed picture, and, for each of the auxiliary pictures,a first second channel comprising a first auxiliary sample array, and a second second channel comprising a second auxiliary sample array,orfour first channels, each comprising a luma sample array derived from a luma sample array of the reconstructed picture and, for each of the auxiliaryFH250102PEP-2025428381.DOCX PCT FILING VERSION / bm134picture, four second channels, each comprising an auxiliary luma sample array, anda first first channel comprising a first chroma sample array derived from a first chroma sample array of the reconstructed picture, a second first channel comprising a second chroma sample array derived from a second chroma sample array of the reconstructed picture, and, for each of the auxiliary pictures,a first second channel comprising a first auxiliary sample array, and a second second channel comprising a second auxiliary sample array.

33. Apparatus according to any of claims 10 to 32, wherein the input channels have equal dimensions.

34. Apparatus according to any of the claims 1 to 33, configured forobtaining the filter input data in form of a tensor comprising a plurality of channels, each of which provides an input channel for the neural network, wherein the apparatus is configured for, in obtaining the filter input data,obtaining one or more first channels of the plurality of channels based on the reconstructed picture and obtaining one or more second channels of the plurality of channels based on a set of auxiliary pictures, each of the auxiliary pictures being represented by one or more of the one or more auxiliary sample arrays,deriving, from the data stream, for each of the auxiliary pictures, an indication which indicates a manner of deriving one or more of the second channels based on the respective auxiliary picture.

35. Apparatus according to claim 34, configured for, for each of the auxiliary pictures, deriving one or more of the second channels based on a set of auxiliary sample arrays out of the auxiliary sample arrays of the respective auxiliary picture, which set of auxiliary sample arrays of the respective auxiliary picture is indicated to be used for the filter input data according to the indication which indicates the mannerFH250102PEP-2025428381.DOCX PCT FILING VERSION / bm135of deriving the one or more second channels based on the respective auxiliary picture.

36. Apparatus according to any of the claims 1 to 35, configured forobtaining one or more first channels of the plurality of channels based on the reconstructed picture and obtaining one or more second channels of the plurality of channels based on one or more auxiliary pictures, each of the auxiliary pictures being represented by one or more of the one or more auxiliary sample arrays,deriving, from the data stream, an indication of a set of layers out of a plurality of layers of the data stream, each of the set of layers carrying one of the auxiliary pictures, andderiving, from the data stream, for each of the layers of the set of layers, an indication which indicates a manner of deriving one or more of the second channels based on the auxiliary picture carried in the respective layer.

37. Apparatus according to claim 36, configured for, for each of the layers, deriving one or more of the second channels based on a set of auxiliary sample arrays out of the auxiliary sample arrays of the respective auxiliary picture, which set of auxiliary sample arrays of the respective auxiliary picture is indicated to be used for the filter input data according to the indication which indicates the manner of deriving the one or more second channels based on the auxiliary picture of the respective layer.

38. Apparatus according to any of the claims 1 to 37, configured forobtaining one or more first channels of the plurality of channels based on the reconstructed picture and obtaining one or more second channels of the plurality of channels based on one or more auxiliary pictures, each of the auxiliary pictures being represented by one or more of the one or more auxiliary sample arrays,deriving, from the data stream, an indication of a set of layers out of a plurality of layers of the data stream, each of the set of layers carrying one of the auxiliary pictures, andFH250102PEP-2025428381.DOCX PCT FILING VERSION / bm136deriving, from the data stream, an indication which indicates whether a manner of deriving the second channels based on the set of auxiliary pictures is selected individually for each of the auxiliary pictures, andif the indication which indicates that the manner of deriving the second channels is selected individual for each of the auxiliary pictures,deriving, from the data stream, for each of the auxiliary pictures, an indication which indicates a manner of deriving the filter input data based on the respective auxiliary picture, orderiving, from the data stream, for each of the auxiliary pictures, an indication which indicates a manner of deriving one or more of the second channels based on the respective auxiliary picture.

39. Apparatus according to claim 38, configured for,if the indication indicates that the manner of deriving the second channels based on the auxiliary pictures is not selected individually for each of the auxiliary pictures, deriving the manner of deriving the second channels from an indication in the data stream, which indicates a manner of deriving the first channels based on the reconstructed picture.

40. Apparatus according to any of claims 36 to 39, wherein the indication, which indicates the manner of deriving the second channels, indicates, for the respective auxiliary picture, for each of one or more sample array types, a number of channels to be derived for the plurality of channels of the input data.

41. Apparatus according to claim 40, configured for deriving an order among the channels derived for the respective auxiliary picture in dependence on the indication which indicates the manner of deriving the second channels.

42. Apparatus according to claim 40 or 41, wherein the indication which indicates the manner of deriving the second channels differentiates between a plurality of modes including one or more or all of the following modes:FH250102PEP-2025428381.DOCX PCT FILING VERSION / bm137a first mode according to which the apparatus derives, for the respective auxiliary picture, one of the channels comprising an auxiliary luma sample array,a second mode according to which the apparatus derives, for the respective auxiliary picture, one of the channels comprising a first auxiliary chroma sample array, and a further one of the channels comprising a second auxiliary chroma sample array,a third mode according to which the apparatus derives, for the respective auxiliary picture, one of the channels comprising an auxiliary luma sample array, a further one of the channels comprising a first auxiliary chroma sample array, and an even further one of the channels comprising a second auxiliary chroma sample array,a fourth mode according to which the apparatus derives, for the respective auxiliary picture, four of the channels, each of which comprises an auxiliary luma sample array, and a further one of the channels comprising a first auxiliary chroma sample array, and an even further one of the channels comprising a second auxiliary chroma sample array.

43. Apparatus according to any of the claims 1 to 42, wherein the picture belongs to a coded video sequence of a sequence of coded video sequences of the data stream, and wherein the apparatus is configured forderiving, from the data stream, an indication of a number of time frames for obtaining the filter input data for filtering the reconstructed picture, andif the indication of the number of time frames indicates to obtain the filter input data using, in addition to the time frame to which the picture belongs, a further picture of a further time frame, andif the further picture belongs to a preceding coded video sequence preceding the coded video sequence of the picture, in dependence on an indication signaled in the data stream, substitute the one of the one or more auxiliary sample arrays belonging to the respective time frame byFH250102PEP-2025428381.DOCX PCT FILING VERSION / bma corresponding one of the auxiliary sample arrays belonging to the first or the last time frame of the coded video sequence, ora default sample array,and / orif the further picture belongs to a succeeding coded video sequence succeeding the coded video sequence of the picture, in dependence on an indication signaled in the data stream, substitute the one of the one or more auxiliary sample arrays belonging the respective time frame bya corresponding one of the auxiliary sample arrays belonging to the first or the last time frame of the coded video sequence, ora default sample array.

44. Apparatus according to any of the claims 1 to 43, configured forobtaining the filter input data in form of a tensor comprising a plurality of channels, each of which provides an input channel for the neural network byobtaining one or more first channels of the plurality of channels based on the sample arrays of the reconstructed picture, andobtaining one or more second channels of the plurality of channels based on one or more auxiliary pictures, each of the auxiliary pictures being represented by one or more of the one or more auxiliary sample arrays, andwherein the apparatus is configured for deriving, for each of the one or more auxiliary pictures, for each of one or more sample array types, a number of channels of the plurality of channels byif the respective auxiliary picture comprises an auxiliary sample array of the respective sample array type, deriving the number of channels based on theFH250102PEP-2025428381.DOCX PCT FILING VERSION / bmauxiliary sample array, and if the respective auxiliary picture misses an auxiliary sample array of the respective sample array type, deriving the number of channels using sample values of a predetermined value.

45. Apparatus according to claim 44, configured for deriving the predetermined value from the data stream.

46. Apparatus according to any of the preceding claim 1 to 45, configured forobtaining the filter input data in form of a tensor comprising a plurality of channels, each of which provides an input channel for the neural network byobtaining one or more first channels of the plurality of channels based on the sample arrays of the reconstructed picture, the sample arrays of the reconstructed picture comprising a luma sample array and two chroma sample arrays, andobtaining one or more second channels of the plurality of channels based on one or more auxiliary pictures, each of the auxiliary pictures being represented by an auxiliary luma sample array and two auxiliary chroma sample array, wherein the apparatus is configured for deriving one of the second channels based on one of the auxiliary chroma sample arrays byif the count of sample values of the auxiliary chroma sample array is less than the count of sample values of the chroma sample array of the reconstructed picture, repeating sample values of the auxiliary chroma sample array, ordetermining a first ratio between the width of the one auxiliary chroma sample array and the width of the sample arrays of the reconstructed picture, and determining a second ratio between the height of the one auxiliary chroma sample array and the height of the sample arrays of the reconstructed picture, and repeating sample values of the auxiliary chroma sample array according to the first ratio and the second ratio.

47. Apparatus according to any of the preceding claim 1 to 46, configured forFH250102PEP-2025428381.DOCX PCT FILING VERSION / bmobtaining the filter input data in form of a tensor comprising a plurality of channels, each of which provides an input channel for the neural network byobtaining one or more first channels of the plurality of channels based on the reconstructed picture, the reconstructed picture comprising a luma sample array, andobtaining a second channel of the plurality of channels based on an auxiliary picture, the auxiliary picture comprising an auxiliary luma sample array, wherein the apparatus is configured for deriving the second channel based on the auxiliary luma sample array byif the count of sample values of the auxiliary luma sample array is less than the count of sample values of the luma sample array of the reconstructed picture, repeating sample values of the auxiliary luma sample array, ordetermining a first ratio between the width of the auxiliary luma sample array and the width of the luma sample array of the reconstructed picture, and determining a second ratio between the height of the auxiliary luma sample array and the height of the luma sample array of the reconstructed picture, and repeating sample values of the auxiliary luma sample array according to the first ratio and the second ratio.

48. Apparatus according to any of the claims 1 to 47, configured forobtaining the filter input data in form of a tensor comprising a plurality of channels, each of which provides an input channel for the neural network byobtaining one or more first channels of the plurality of channels based on the sample arrays of the reconstructed picture, andobtaining one or more second channels of the plurality of channels based on one or more auxiliary pictures, each of the auxiliary pictures being represented by one or more of the one or more auxiliary sample arrays,FH250102PEP-2025428381.DOCX PCT FILING VERSION / bm141wherein the apparatus is configured for deriving sample values of the first channels and the second channels at a common bit depth.

49. Apparatus according to any of the claims 1 to 48, configured forobtaining the filter input data in form of a tensor comprising a plurality of channels, each of which provides an input channel for the neural network byobtaining one or more first channels of the plurality of channels based on the sample arrays of the reconstructed picture, andobtaining one or more second channels of the plurality of channels based on the one or more auxiliary sample arrays,wherein the apparatus is configured for deriving one of the second channels from one of the auxiliary sample arrays by converting sample values of the one auxiliary sample array to obtain a converted auxiliary sample array and deriving the second channel from the converted auxiliary sample array.

50. Apparatus according to claim 49, configured for deriving a manner of converting the sample values of the one auxiliary array from the data stream.

51. Apparatus according to any of the claims 1 to 50, configured for deriving the filter input data by including into the filter input data one or more or all ofan information on quantization parameter used in reconstructing the picture, and / orone or more threshold values for an alpha maska smallest and / or a highest depth value for a depth mapan indication of a mapping function for sample values of a depth map, an indication of a mapping function for sample values of a disparity map.

52. Apparatus according to any of the claims 1 to 51 ,wherein the one or more auxiliary sample arrays comprise one or more of a depth map, a mask, and a transparency map; and / orFH250102PEP-2025428381.DOCX PCT FILING VERSION / bm142wherein the one or more auxiliary sample arrays comprise one or more sample arrays of a further picture, whereinthe data stream has encoded thereinto a multi-view video, wherein the picture belongs to a first view of the multi-view video, and wherein the further picture belongs to a second view of the multi-view video, orthe further picture has a different resolution than the picture.

53. Apparatus according to any of the claims 1 to 52, wherein sample arrays of the reconstructed picture and the one or more auxiliary sample arrays have equivalent sizes.

54. Apparatus according to any of the claims 1 to 53, wherein the reconstructed picture and any auxiliary picture being represented by the one or more auxiliary sample arrays have the same chroma format.

55. Apparatus according to any of the claims 1 to 54, wherein the reconstructed picture and any auxiliary picture being represented by the one or more auxiliary sample array have the same bit depth of luma and / or chroma.

56. Apparatus according to any of the claims 1 to 55, wherein the apparatus is configured for reconstructing the video from the data stream by block based predictive and transform based residual decoding bydecoding prediction residual data of a residual block from the data streamby use of context-adaptive variable length decoding by using a first syntax element indicating a total number of non-zero transform coefficients in a transform block representing the residual block, and a trailing-one number, indicating a number of non-zero transform coefficients having an absolute value of one when traversing the coefficients along a scan order,FH250102PEP-2025428381.DOCX PCT FILING VERSION / bm143one or more second syntax elements indicating a sign of the non-zero transform coefficients having an absolute value of one when traversing the coefficients along the scan order,one or more third syntax elements indicating a value of the non-zero transform coefficients except for the number of non-zero transform coefficients having an absolute value of one when traversing the coefficients along the scan order,a fourth syntax element indicating a total number of zero-valued transform coefficient levels in the transform block from a firstly- encountered non-zero transform coefficient in the scan order onwards, andone or more fifth syntax elements indicting positions of the non-zero transform coefficients along the scan order by indicating a number of consecutive zero-valued transform coefficients in the scan order between in the scan order consecutively encountered non-zero transform coefficients, orby use of context-adaptive binary arithmetic decoding bydecoding a significance map which indicates positions of non-zero transform coefficients in a transform block representing the residual block by, in a forward scan traversing transform coefficients of the transform block, decoding a significance flag which indicates whether a non-zero transform coefficient is positioned at a current position, and, if so, and if the current position is not the last in the forward scan, decoding a last-significance flag which indicates whether the nonzero transform coefficient positioned at the current position is the last non-zero transform coefficient in the forward scan order, and decoding the non-zero transform coefficients’ values sequentially in a reverse scan order, reversing the forward scan order, ordecoding prediction residual data of a residual block from the data stream by use of context-adaptive binary arithmetic decoding of quantization indices of transform coefficients of a transform block representing the residual block and sequential dequantization of the quantization indices according to which a value of a current transform coefficient depends on a parity of quantization indices of previous quantization indices.FH250102PEP-2025428381.DOCX PCT FILING VERSION / bm14457. Apparatus (10) for encoding a video (9) into a data stream, configured forencoding (81) a picture (12) of the video into the data stream; andinserting (81), into the data stream, an indication (82) of one or more auxiliary sample arrays (15), the one or more auxiliary sample arrays being to be used for obtaining filter input data for filtering a reconstructed version of the picture, which filter input data is to be obtained based on the reconstructed picture and based on the one or more auxiliary sample arrays, and which filter input data is to be subjected to a neural network,wherein the picture (12) and the one or more auxiliary sample arrays (15) belong to the same time frame of the video.

58. Apparatus according to claim 57, wherein the filter input data is represented by a tensor having the same dimension as an input of the neural network.

59. Apparatus according to claim 57 or 58, wherein the data stream is a multi-layer video data stream, and wherein picture belongs to a first layer of the data stream and wherein the one or more auxiliary sample arrays belong to one of one or more further layers of the data stream.

60. Apparatus according to claim 60, wherein coded video sequences of the first layer and the one or more further layers start at the same time frame and end at the same time frame.

61. Apparatus according to any of claims 57 to 58, configured for encoding into the data streama sequence of primary pictures including the picture, the sequence of primary pictures representing the video, andthe one or more auxiliary sample arrays.FH250102PEP-2025428381.DOCX PCT FILING VERSION / bm14562. Apparatus according to any of claims 57 to 58 or 61, configured for encoding, into the data stream, a sequence of primary pictures including the picture by encoding the sequence of primary pictures in payload packets of a first type, and encoding the one or more auxiliary sample arrays into payload packets of a second type.

63. Apparatus according to any of the claims 57 to 62, wherein the picture belongs to a first time frame of the video, and wherein the filter input data is further to be obtained based on a further picture, the further picture belonging to a second time frame of the video, and based on one or more further auxiliary sample arrays, the one or more further auxiliary sample arrays belonging to the second time frame.

64. Apparatus according to any of the claims 57 to 63, configured for inserting, into the data stream, a syntax element which differentiates between a plurality of modes of obtaining the filter input data, wherein the plurality of modes includes a mode according to which the filter input data for the picture is to be obtained based on one or more auxiliary sample arrays belonging to the same time frame as the picture.

65. Apparatus according to any of the claims 57 to 64, wherein the filter input data is to be obtained in form of a tensor comprising a plurality of channels, each of which provides an input channel for the neural network.

66. Apparatus according to any of the claims 57 to 65, configured forinserting, into the data stream, an indication of a set of auxiliary pictures, each of the auxiliary pictures being represented by one or more of the one or more auxiliary sample arrays.

67. Apparatus according to claim 66, wherein the set of auxiliary pictures is indicated by indicating a set of layers out of a plurality of layers of the data stream, each of the layers of the set of layers carrying one of the auxiliary pictures.

68. Apparatus according to claim 66 or 67, configured for inserting, into the data stream, and indication of the count of auxiliary pictures of the set of auxiliary pictures.

69. Apparatus according to any of the claims 57 to 68, configured forFH250102PEP-2025428381.DOCX PCT FILING VERSION / bm146inserting, into the data stream, an indication of a set of layers out of a plurality of layers of the data stream, each of the layers of the set of layers carrying an auxiliary picture, the auxiliary picture being represented by one or more of t the one or more auxiliary sample arrays.

70. Apparatus according to claim 69, configured forinserting, into the data stream, an indication indicating the count of layers of the set of layers, andinserting, into the data stream, for each of the set of layers, an identification of the respective layer within the plurality of layers of the data stream.

71. Apparatus according to claim 70, configured for inserting, into the data stream, a layer identification syntax element which signals the identification of the respective layer within the plurality of layers, wherein each of the plurality of layers is associated with a respective layer identifier, and wherein the syntax element indicates the layer identifier of the respective layer.

72. Apparatus according to claim 70, configured for inserting, into the data stream, for each of the plurality of layers, a set of layer-specific property indicators, each of the layer-specific property indicators indicating a property of the respective layer by signaling a value out of a domain of the respective layer-specific property indicator, andwherein the apparatus is configured for inserting, into the data stream, a layer identification syntax element, which signals the identification of the respective layer within the plurality of layers, wherein the syntax element indicates a value for each layer-specific property indicator of the set of layer-specific property indicators, and identifying the respective layer based on the combination of values of the layerspecific property indicators.

73. Apparatus according to claim 72, wherein, in the data stream, each of the layerspecific property indicators is represented by a predefined count of bits domain of the respective layer-specific property indicator, andFH250102PEP-2025428381.DOCX PCT FILING VERSION / bm147wherein the sum of the predefined counts of bits of the layer-specific property indicators of the set of layer-specific property indicators equals the count of bits with which the layer identification syntax element.

74. Apparatus according to claim 72, wherein the count of bits with which the layer identification syntax element is signaled in the data stream equals the sum of the counts of bits, which are required to represent the domains of the layer-specific property indicators of the set of layer-specific property indicators, wherein the the counts of bits, which are required to represent the domains of the layer-specific property indicators are less than the counts of bits, which are used in the data stream to signal the layer-specific property indicators.

75. Apparatus according to any of claims 72 to 74, configured for inserting the set of layer-specific property indicators into an indication in the data stream.

76. Apparatus according to any of claims 72 to 75, configured for inserting an indication of the domains of the layer-specific property indicators in the data stream.

77. Apparatus according to claim 71 , configured for inserting a syntax element indicating the count of layers of the set of layers from a syntax element contained into a first payload packet of the data stream, and inserting, for each of the set of layers, an identification of the respective layer among the plurality of layers of the data stream into a second payload packet of the data stream.

78. Apparatus according to claim 77, configured forinserting, into the second payload packet, an indication indicating a set of layerspecific property indicators, each of the layer-specific property indicators indicating a property of the respective layer by signaling a value out of a domain of the respective layer-specific property indicator,inserting, into the data stream, for each of the plurality of layers, the set of layerspecific property indicators, andfor each of the set of layers, inserting, into the data stream, a layer identification syntax element, which signals the identification of the respective layer within theFH250102PEP-2025428381.DOCX PCT FILING VERSION / bm148plurality of layers, wherein the syntax element indicates a value for each layerspecific property indicator of the set of layer-specific property indicators, and identifying the respective layer based on the combination of values of the layerspecific property indicators.

79. Apparatus according to claim 77, configured forfor each of the set of layers,inserting, into the second payload packet, an indication indicating a set of layer-specific property indicators, each of the layer-specific property indicators indicating a property of the respective layer by signaling a value out of a domain of the respective layer-specific property indicator, andinserting, into the data stream, the set of layer-specific property indicators, andinserting, into the data stream, a layer identification syntax element, which signals the identification of the respective layer within the plurality of layers, wherein the syntax element indicates a value for each layer-specific property indicator of the set of layer-specific property indicators, and identifying the respective layer based on the combination of values of the layer-specific property indicators.

80. Apparatus according to any of claims 72 to 79, wherein the indication indicating the set of layer-specific property indicators is signaled by a syntax element, which differentiates between a plurality of modes including one or more ofa first mode according to which the set of layer-specific property indicators comprises, or consists of, a layer identifier,a second mode according to which the set of layer-specific property indicators comprises, or consists of, a view identifier and / or a depth identifier, a third mode according to which the set of layer-specific property indicators comprises, or consists of, a layer identifier and one or more further layer-specific property indicators.FH250102PEP-2025428381.DOCX PCT FILING VERSION / bm14981. Apparatus according to claim 80, wherein the apparatus is configured for inserting a further syntax element into the data steam, which indicates a bit-length of the syntax element in the data stream.

82. Apparatus according to any of claims 70 to 81 , configured for inserting, into the data stream, a layer identification syntax element, which identifies the respective layer bysignaling a layer identifier which is unique to the respective layer, wherein each of the plurality of layers is associated with a respective layer identifier, and / orindicating a value for each of a set of layer-specific property indicators, each of the layer-specific property indicators indicating a property of the respective layer by signaling a value out of a domain of the respective layer-specific property indicator,wherein the apparatus is configured for inserting a further syntax element into the data steam, which indicates a bit-length of the layer-identification syntax element in the data stream.

83. Apparatus according to any of the claims 68 or 70, wherein the filter input data is to be obtained in form of a tensor comprising a plurality of channels, each of which provides an input channel for the neural network bydetermining a first number of first channels of the plurality of channels based on sample arrays of the reconstructed picture, anddetermining, for each of the auxiliary pictures, a second number of second channels of the plurality of channels.

84. Apparatus according to claim 84, configured for inserting, into the data stream, the indication of the manner of deriving the second channels, wherein the indication of the manner of deriving the second channels.

85. Apparatus according to claim 84,FH250102PEP-2025428381.DOCX PCT FILING VERSION / bm150wherein the second number of second channels is equal for all of the auxiliary pictures, orwherein the apparatus is configured for inserting, into the data stream, for each of the auxiliary pictures a respective indication of a manner of deriving the second channels, and for deriving the second number of second channels individually for each of the auxiliary pictures.

86. Apparatus according to any of claims 66 to 84, wherein the filter input data is to be obtained in form of a tensor comprising a plurality of channels, each of which provides an input channel for the neural network,wherein the apparatus is configured for inserting, into the data stream a syntax element which indicates a mode for obtaining the filter input data out of a plurality of modes, the plurality of modes comprising one or more of all of the following modes, according to which the plurality of channels to comprisea first channel comprising a luma sample array derived from a luma sample array of the reconstructed picture and, for each of the auxiliary pictures, a second channel comprising an auxiliary luma sample array,ora first first channel comprising a first chroma sample array derived from a first chroma sample array of the reconstructed picture, a second first channel comprising a second chroma sample array derived from a second chroma sample array of the reconstructed picture, and, for each of the auxiliary pictures,a first second channel comprising a first auxiliary chroma sample array, and a second second channel comprising a second auxiliary chroma sample array,orFH250102PEP-2025428381.DOCX PCT FILING VERSION / bm151a first channel comprising a luma sample array derived from a luma sample array of the reconstructed picture and, for each of the auxiliary pictures, a second channel comprising an auxiliary luma sample array, anda first first channel comprising a first chroma sample array derived from a first chroma sample array of the reconstructed picture, a second first channel comprising a second chroma sample array derived from a second chroma sample array of the reconstructed picture, and, for each of the auxiliary picture,a first second channel comprising a first auxiliary chroma sample array, and a second second channel comprising a second auxiliary chroma sample array derived from a second chroma sample array of the auxiliary picture,orfour first channels, each comprising a luma sample array derived from a luma sample array of the reconstructed picture and, for each of the set of layers, four second channels, each comprising an auxiliary luma sample array, anda first first channel comprising a first chroma sample array derived from a first chroma sample array of the reconstructed picture, a second first channel comprising a second chroma sample array derived from a second chroma sample array of the reconstructed picture, and, for each of the auxiliary pictures,a first second channel comprising a first auxiliary chroma sample array, and a second second channel comprising a second auxiliary chroma sample array.

87. Apparatus according to any of claims 65 to 86, wherein the input channels have equal dimensions.

88. Apparatus according to any of the claims 57 to 87,FH250102PEP-2025428381.DOCX PCT FILING VERSION / bm152wherein the filter input data is to be obtained in form of a tensor comprising a plurality of channels, each of which provides an input channel for the neural network byobtaining one or more first channels of the plurality of channels based on the reconstructed picture and obtaining one or more second channels of the plurality of channels based on a set of auxiliary pictures, each of the auxiliary pictures being represented by one or more of the one or more auxiliary sample arrays,wherein the apparatus is configured for inserting, into the data stream, for each of the auxiliary pictures, an indication which indicates a manner of deriving one or more of the second channels based on the respective auxiliary picture.

89. Apparatus according to claim 88, wherein the indication which indicates the manner of deriving the one or more second channels based on the respective auxiliary picture indicates, for each of the auxiliary pictures, a set of auxiliary sample arrays out of the auxiliary sample arrays of the respective auxiliary picture, which set of auxiliary sample arrays of the respective auxiliary picture is to be used for the filter input data.

90. Apparatus according to any of the claims 57 to 89,wherein the filter input data is to be obtained by obtaining one or more first channels of the plurality of channels based on the reconstructed picture and obtaining one or more second channels of the plurality of channels based on one or more auxiliary pictures, each of the auxiliary pictures being represented by one or more of the one or more auxiliary sample arrays,wherein the apparatus is configured forinserting, into the data stream, an indication of a set of layers out of a plurality of layers of the data stream, each of the set of layers carrying one of the auxiliary pictures, andFH250102PEP-2025428381.DOCX PCT FILING VERSION / bm153inserting, into the data stream, for each of the layers of the set of layers, an indication which indicates a manner of deriving one or more of the second channels based on the auxiliary picture carried in the respective layer.

91. Apparatus according to claim 90, wherein the indication which indicates the manner of deriving the one or more second channels based on the auxiliary picture of the respective layer indicates, for each of the layers, a set of auxiliary sample arrays out of the auxiliary sample arrays of the respective auxiliary picture, which set of auxiliary sample arrays of the respective auxiliary picture is to be used for the filter input data.

92. Apparatus according to any of the claims 57 to 91 , configured forwherein the filter input data is to be obtained by obtaining one or more first channels of the plurality of channels based on the reconstructed picture and obtaining one or more second channels of the plurality of channels based on one or more auxiliary pictures, each of the auxiliary pictures being represented by one or more of the one or more auxiliary sample arrays,wherein the apparatus is configured forinserting, into the data stream, an indication of a set of layers out of a plurality of layers of the data stream, each of the set of layers carrying one of the auxiliary pictures, andinserting, into the data stream, an indication which indicates whether a manner of deriving the second channels based on the set of auxiliary pictures is selected individual for each of the auxiliary pictures, andif the indication which indicates that the manner of deriving the second channels is selected individual for each of the auxiliary pictures,inserting, into the data stream, for each of the auxiliary pictures, an indication which indicates a manner of deriving the filter input data based on the respective auxiliary picture, orFH250102PEP-2025428381.DOCX PCT FILING VERSION / bm154inserting, into the data stream, for each of the auxiliary pictures, an indication which indicates a manner of deriving one or more of the second channels based on the respective auxiliary picture.

93. Apparatus according to claim 92, configured for,if the indication indicates that the manner of deriving the second channels based on the set of layers is not individual for each of the layers, inserting an indication in the data stream, which indicates a manner of deriving the first channels based on the reconstructed picture.

94. Apparatus according to any of claims 90 to 93, wherein the indication, which indicates the manner of deriving the second channels, indicates, for the respective auxiliary picture, for each of one or more sample array types, a number of channels to be derived for the plurality of channels of the input data.

95. Apparatus according to claim 94, wherein the indication which indicates the manner of deriving the second channels indicates an order among the channels derived for the respective auxiliary picture.

96. Apparatus according to claim 94 or 95, wherein the indication which indicates the manner of deriving the second channels differentiates between a plurality of modes including one or more or all of the following modes:a first mode according to which the apparatus derives, for the respective auxiliary picture, one of the channels comprising an auxiliary luma sample array,a second mode according to which the apparatus derives, for the respective auxiliary picture, one of the channels comprising a first auxiliary chroma sample array, and a further one of the channels comprising a second auxiliary chroma sample array,a third mode according to which the apparatus derives, for the respective auxiliary picture, one of the channels comprising an auxiliary luma sample array, a further one of the channels comprising a first auxiliary chromaFH250102PEP-2025428381.DOCX PCT FILING VERSION / bm155sample array, and an even further one of the channels comprising a second auxiliary chroma sample array,a fourth mode according to which the apparatus derives, for the respective auxiliary picture, four of the channels, each of which comprises an auxiliary luma sample array, and a further one of the channels comprising a first auxiliary chroma sample array, and an even further one of the channels comprising a second auxiliary chroma sample array.

97. Apparatus according to any of the claims 57 to 96, configured forwherein the filter input data is to be obtained in form of a tensor comprising a plurality of channels, each of which provides an input channel for the neural network byobtaining one or more first channels of the plurality of channels based on the sample arrays of the reconstructed picture, andobtaining one or more second channels of the plurality of channels based on one or more auxiliary pictures, each of the auxiliary pictures being represented by one or more of the one or more auxiliary sample arrays, andderiving, for each of the one or more auxiliary pictures, for each of one or more sample array types, a number of channels of the plurality of channels byif the respective auxiliary picture comprises an auxiliary sample array of the respective sample array type, deriving the number of channels based on the auxiliary sample array, and if the respective auxiliary picture misses an auxiliary sample array of the respective sample array type, deriving the number of channels using sample values of a predetermined value.wherein the apparatus is configured for inserting an indication of the predetermined value into the data stream.

98. Apparatus according to any of the claims 57 to 97, configured forFH250102PEP-2025428381.DOCX PCT FILING VERSION / bm156wherein the filter input data is to be obtained in form of a tensor comprising a plurality of channels, each of which provides an input channel for the neural network byobtaining one or more first channels of the plurality of channels based on the sample arrays of the reconstructed picture, andobtaining one or more second channels of the plurality of channels based on the one or more auxiliary sample arrays, andderiving one of the second channels from one of the auxiliary sample arrays by converting sample values of the one auxiliary sample array to obtain a converted auxiliary sample array and deriving the second channel from the converted auxiliary sample array.wherein the apparatus is configured for inserting an indication of deriving a manner of converting the sample values of the one auxiliary array into the data stream.

99. Apparatus according to any of the claims 57 to 98, wherein sample arrays of the reconstructed picture and the one or more auxiliary sample arrays have equivalent sizes.

100. Apparatus according to any of the claims 57 to 99, wherein the reconstructed picture and any auxiliary picture being represented by the one or more auxiliary sample array have the same chroma format.

101. Apparatus according to any of the claims 57 to 100, wherein the reconstructed picture and any auxiliary picture being represented by the one or more auxiliary sample array have the same bit depth of luma and / or chroma.

102. Apparatus according to any of the claims 57 to 101 , wherein widths of the auxiliary sample arrays are a multiple of a width of the sample arrays of the reconstructed picture, and heights of the auxiliary sample arrays are a multiple of a height of the sample arrays of the reconstructed picture.FH250102PEP-2025428381.DOCX PCT FILING VERSION / bm157103. Apparatus according to any of the claims 57 to 102, wherein the apparatus is configured for encoding the video into the data stream by block-based predictive coding and transform-based residual coding byencoding prediction residual data of the residual block into the data streamby use of context adaptive variable length coding by usinga first syntax element indicating a total number of non-zero transform coefficients in a transform block representing the residual block, and a trailing-one number, indicating a number of non-zero transform coefficients having an absolute value of one when traversing the coefficients along a scan order,one or more second syntax elements indicating a sign of the non-zero transform coefficients having an absolute value of one when traversing the coefficients along the scan order, one or more third syntax elements indicating a value of the non-zero transform coefficients except for the number of non-zero transform coefficients having an absolute value of one when traversing the coefficients along the scan order,a fourth syntax element indicating a total number of zero-valued transform coefficient levels in the transform block from a firstly- encountered non-zero transform coefficient in the scan order onwards, andone or more fifth syntax elements indicting positions of the non-zero transform coefficients along the scan order by indicating a number of consecutive zero-valued transform coefficients in the scan order between in the scan order consecutively encountered non-zero transform coefficients, orby use of context-adaptive binary arithmetic coding byencoding a significance map which indicates positions of non-zero transform coefficients in a transform block representing the residual block by, in a forward scan traversing transform coefficients of the transform block, encoding a significance flag which indicates whetherFH250102PEP-2025428381.DOCX PCT FILING VERSION / bm158a non-zero transform coefficient is positioned at a current position, and, if so, and if the current position is not the last in the forward scan, encoding a last-significance flag which indicates whether the nonzero transform coefficient positioned at the current position is the last non-zero transform coefficient in the forward scan order, and encoding the non-zero transform coefficients’ values sequentially in a reverse scan order, reversing the forward scan order, orencoding prediction residual data of a residual block into the data stream by use of context-adaptive binary arithmetic coding of quantization indices of transform coefficients of a transform block representing the residual block and sequential quantization of the transform coefficients to obtain the quantization indices, according to which a quantizer for quantizing a current transform coefficient depends on a parity of quantization indices of previous quantization indices.

104. Method for reconstructing a video from a data stream (14), the method comprising:reconstructing (21) a picture (12) of the video; andfiltering (61) the reconstructed picture (12’) using a neural network by obtaining (63) filter input data (65) based on the reconstructed picture (12’) and based on one or more auxiliary sample arrays (15), and subjecting the filter input data (65) to the neural network,wherein the picture and the one or more auxiliary sample arrays belong to the same time frame (11*) of the video.

105. Method for encoding a video (9) into a data stream (14), the method comprising:encoding (81) a picture (12) of the video into the data stream; andinserting (81), into the data stream, an indication (82) of one or more auxiliary sample arrays (15), the one or more auxiliary sample arrays being to be used for obtaining filter input data for filtering a reconstructed version of the picture, which filter input data is to be obtained based on the reconstructed picture and based on the one orFH250102PEP-2025428381.DOCX PCT FILING VERSION / bm159more auxiliary sample arrays, and which filter input data is to be subjected to the neural network,wherein the picture (12) and the one or more auxiliary sample arrays 15) belong to the same time frame of the video.

106. A data stream (14) having encoded thereinto a video (9), wherein the data stream comprises:a picture (12) of the video; andan indication (82) of one or more auxiliary sample arrays (15), the one or more auxiliary sample arrays being to be used for obtaining filter input data for filtering a reconstructed version of the picture, which filter input data is to be obtained based on the reconstructed picture and based on the one or more auxiliary sample arrays, and which filter input data is to be subjected to the neural network,wherein the picture (12) and the one or more auxiliary sample arrays 15) belong to the same time frame of the video.

107. A computer program for implementing the method of claim 104 or 105, when being executed on a computer or signal processor.FH250102PEP-2025428381.DOCX PCT FILING VERSION / bm

Images