Methods, systems, and apparatuses for usage restrictions of videos

Abstract

Methods, apparatuses, and systems provide recovery for usage restrictions of videos. In the context of a method, the method includes generating first information indicative of a video; generating second information indicative of at least one rule for usage of the video; and outputting at least one signal comprising a representation of the first information and the second information.

Description

METHODS, SYSTEMS, AND APPARATUSES FOR USAGE RESTRICTIONS OF VIDEOS TECHNOLOGICAL FIELD

[0001] Embodiments of the present disclosure relate generally to communication technology and, more particularly, relate to methods, systems, and apparatuses for usage restrictions of videos.BACKGROUND

[0002] In order to facilitate storage and / or transmission of videos in an efficient manner, frames of a video may be encoded in order to compress the video information prior to storage and / or transmission. The encoded information may then be decoded and, as a result, decompressed by a decoder in order to reconstruct the video for display or the like.BRIEF SUMMARY

[0003] Methods, apparatuses, and systems are disclosed for usage restrictions of videos. In this regard, the methods, apparatuses, and systems are configured to support a framework for an encoder to signal, in or along a bitstream, and for a decoder to decode, from or along the bitstream, information indicative of how a video may be used, where the video or a signal from which the video is derived may be included in the bitstream. The framework provides for an increase in the granularity of usage restrictions that apply to a video, thereby reducing or removing ambiguities associated with usage of the video at a decoder and, as such, improving the performance of the decoder.

[0004] In at least one example embodiment, an apparatus is provided comprising at least one processor and at least one memory including computer program code (e.g., instructions) configured to, with the at least one processor, cause the apparatus at least to: generate first information indicative of a video; generate second information indicative of at least one rule for usage of the video; and output at least one signal comprising a representation of the first information and the second information.

[0005] In at least one example embodiment, generating the first information comprises: generating a bitstream comprising the first information, wherein the bitstream comprises encoded data from which the video is derived.

[0006] In at least one example embodiment, the video comprises: an output of a processor, wherein the output is based at least in part on a decoded video input into the processor, or an output of a processing chain comprising at least two processors, wherein theoutput is based at least in part on an input to the processing chain, and wherein the input comprises a decoded video.

[0007] In at least one example embodiment, the processor comprises a neural network, or the at least two processors each comprise a neural network.

[0008] In at least one example embodiment, the at least one rule comprises a rule for usage of the video in accordance with at least one operation to train at least one artificial intelligence model.

[0009] In at least one example embodiment, the at least one rule is for usage of the video to determine a ground-truth for the at least one operation to train the at least one artificial intelligence model.

[0010] In at least one example embodiment, the at least one rule is for usage of the video to determine an input for the at least one operation to train the at least one artificial intelligence model.

[0011] In at least one example embodiment, the at least one signal comprises a supplemental enhancement information (SEI) message including the second information.

[0012] In at least one example embodiment, the SEI message comprises an artificial intelligence usage restrictions (AUR) SEI message or a neural network post filter (NNPF) SEI message.

[0013] In at least one example embodiment, the SEI message comprises a text description information SEI message.

[0014] In at least one example embodiment, the SEI message comprises an artificial intelligence usage restrictions (AUR) SEI message, and wherein a value of an indicator included in the AUR SEI message indicates the at least one rule by indicating, to a receiver of the at least one signal, at least one of the following: to refrain from using the video as input to an artificial intelligence model for training of the artificial intelligence model; or to refrain from using the video to determine a ground-truth for training an artificial intelligence model.

[0015] In at least one example embodiment, an AUR SEI message type associated with the AUR SEI message is associated with an SEI processing order (SPO) SEI message to indicate that the AUR SEI message is descriptive of at least one of the following: characteristics of a processing chain indicated in the SPO SEI message, pictures output by a processing chain indicated in the SPO SEI message, or a result of a processing chain indicated in the SPO SEI message.

[0016] In at least one example embodiment, the AUR SEI message is included in a processing order nesting (PON) SEI message and is associated with a processing order value.

[0017] In at least one example embodiment, the processing order value associated with the AUR SEI message indicates that the AUR SEI message is descriptive of an output of a NNPF for post-process filtering associated with the video.

[0018] In at least one example embodiment, the SEI message comprises a neural network post filter (NNPF) SEI message indicating at least one of characteristics or activation of a NNPF for post-process filtering associated with the video, and wherein the NNPF SEI message includes at least one flag or at least one indicator that indicates the at least one rule.

[0019] In at least one example embodiment, the NNPF SEI message includes the at least one flag, and wherein the at least one flag indicates the at least one rule by indicating, to a receiver of the at least one signal, at least one of the following: to use one or more pictures output by the NNPF as input to an artificial intelligence model for training of the artificial intelligence model, to refrain from using one or more pictures output by the NNPF as input to an artificial intelligence model for training of the artificial intelligence model, to use one or more pictures output by the NNPF to determine a ground-truth for training an artificial intelligence model, or to refrain from using one or more pictures output by the NNPF to determine a ground-truth for training an artificial intelligence model.

[0020] In at least one example embodiment, the NNPF SEI message includes an indicator, and wherein a value of the indicator indicates the at least one rule by indicating, to a receiver of the at least one signal, at least one of the following: to refrain from using one or more pictures output by the NNPF both as input to an artificial intelligence model for training of the artificial intelligence model and to determine a ground-truth for training the artificial intelligence model, to use one or more pictures output by the NNPF as input to an artificial intelligence model for training of the artificial intelligence model, to use one or more pictures output by the NNPF to determine a ground-truth for training an artificial intelligence model, or to use one or more pictures output by the NNPF both as input to an artificial intelligence model for training of the artificial intelligence model and to determine a ground-truth for training the artificial intelligence model.

[0021] In at least one example embodiment, the NNPF SEI message includes an indicator, and wherein a value of the indicator indicates the at least one rule by indicating, to a receiver of the at least one signal, at least one of the following: to use one or more pictures output by the NNPF as input to a machine vision model, to refrain from using one or more pictures output by the NNPF as input to a machine vision model, to use one or more pictures output by the NNPF for training a machine vision model, to refrain from using one or more pictures output by the NNPF for training a machine vision model, to use one or more picturesoutput by the NNPF as input to a machine vision model for training of the machine vision model, to refrain from using one or more pictures output by the NNPF as input to a machine vision model for training of the machine vision model, to use one or more pictures output by the NNPF to determine a ground-truth for training a machine vision model, or to refrain from using one or more pictures output by the NNPF to determine a ground-truth for training a machine vision model.

[0022] In at least one example embodiment, the NNPF SEI message includes a plurality of indicators associated with a plurality of processors in a processing chain.

[0023] In at least one example embodiment, the NNPF is comprised in a processing chain indicated via a SEI processing order (SPO) SEI message.

[0024] In at least one example embodiment, the NNPF is associated with distortions of images for stylistic or artistic functions.

[0025] In at least one example embodiment, the NNPF SEI message is indicative of one or more generative artificial intelligence characteristics associated with the NNPF, and wherein the at least one rule is based at least in part on the one or more generative artificial intelligence characteristics.

[0026] In at least one example embodiment, the NNPF SEI message indicates that the NNPF comprises a capability to output a plurality of pictures for each activation or inference of the NNPF, and wherein the at least one rule pertains to the plurality of pictures.

[0027] In at least one example embodiment, the NNPF SEI message comprises a neural network post filter characteristics (NNPFC) SEI message indicating characteristics of the NNPF or a neural network post filter activation (NNPF A) SEI message indicating activation of the NNPF for post-process filtering associated with the video.

[0028] In at least one example embodiment, the NNPF SEI message comprises an indication of whether an output of the NNPF comprises information pertaining to one or more pictures input to or output by the NNPF.

[0029] In at least one example embodiment, the second information comprises content information pertaining to respective content of one or more pictures associated with the video, and wherein the content information comprises semantic information or image quality information.

[0030] In at least one example embodiment, the at least one rule pertains to one or more operations performed using the one or more pictures associated with the video.

[0031] In at least one example embodiment, the at least one rule is based at least in part on whether the one or more pictures include faces.

[0032] In at least one example embodiment, the second information is indicative of a quality metric threshold, and wherein application of the at least one rule to the one or more pictures is based at least in part on whether a respective quality of the one or more pictures satisfies the quality metric threshold.

[0033] In at least one example embodiment, the at least one signal comprises third information indicative of a spatial region, within the one or more pictures, to which the at least one rule is applicable.

[0034] In at least one example embodiment, the third information indicates the spatial region by including a region type indication that indicates at least one of the following: whether the at least one rule is applicable to complete pictures, whether the at least one rule is applicable to one or more spatial regions associated with the region type indication, or whether the at least one rule is applicable to leftover areas, wherein a leftover area comprises a picture without one or more spatial regions associated with the region type indication.

[0035] In at least one example embodiment, the region type indication further indicates at least one of the following: one or more spatial regions indicated by at least one SEI message, one or more spatial regions indicated by an annotated regions SEI message, one or more regions of an annotated regions SEI message that are indicated in an AUR SEI message, or one or more spatial regions indicated through spatial coordinates within an AUR SEI message.

[0036] In at least one example embodiment, the at least one signal comprises fourth information indicating that the at least one rule is applicable to one or more types of data that is output by a neural network post filter (NNPF) and is associated with the one or more pictures, wherein the one or more types of data include at least one of the following: spatially extrapolated data, temporally extrapolated data, or temporally interpolated data.

[0037] In at least one example embodiment, the second information is indicative of one or more machine learning tasks, one or more machine vision types, or one or more artificial intelligence model types.

[0038] In at least one example embodiment, the at least one signal comprises third information indicative of the one or more machine vision types or the one or more artificial intelligence model types, and wherein the one or more machine vision types or the one or more artificial intelligence model types are based at least in part on a predetermined vocabulary.

[0039] In at least one example embodiment, the at least one rule indicates whether data associated with the video is usable for one or more operations associated with an artificialintelligence model, and wherein the one or more operations comprise at least one of the following: training, validation, or inference.

[0040] In at least one example embodiment, the at least one rule pertains to usage of the video with at least one of the following: artificial intelligence, machine learning, machine analysis, a neural network, human viewing or displaying, one or more derivative operations, one or more types of software applications, content authentication, or digital rights management.

[0041] In at least one example embodiment, an apparatus is provided comprising at least one processor and at least one memory including computer program code (e.g., instructions) configured to, with the at least one processor, cause the apparatus at least to: receive at least one signal comprising a representation of first information indicative of a video and second information indicative of at least one rule for usage of the video; and decode the at least one signal to obtain the first information and the second information.

[0042] In at least one example embodiment, the video comprises an output of a processor, wherein the output is based at least in part on a decoded video input into the processor, or an output of a processing chain comprising at least two processors, wherein the output is based at least in part on an input to the processing chain, and wherein the input comprises a decoded video.

[0043] In at least one example embodiment, the processor comprises a neural network, or the at least two processors each comprise a neural network.

[0044] In at least one example embodiment, the at least one rule comprises a rule for usage of the video in accordance with at least one operation to train at least one artificial intelligence model.

[0045] In at least one example embodiment, the at least one rule comprises a rule for usage of the video in accordance with at least one operation to train at least one artificial intelligence model.

[0046] In at least one example embodiment, the at least one rule is for usage of the video to determine a ground-truth for the at least one operation to train the at least one artificial intelligence model.

[0047] In at least one example embodiment, the at least one rule is for usage of the video to determine an input for the at least one operation to train the at least one artificial intelligence model.

[0048] In at least one example embodiment, the at least one signal comprises a supplemental enhancement information (SEI) message including the second information.

[0049] In at least one example embodiment, the SEI message comprises an artificial intelligence usage restrictions (AUR) SEI message or a neural network post filter (NNPF) SEI message.

[0050] In at least one example embodiment, the SEI message comprises a text description information SEI message.

[0051] In at least one example embodiment, the SEI message comprises an artificial intelligence usage restrictions (AUR) SEI message, and wherein a value of an indicator included in the AUR SEI message indicates the at least one rule by indicating, to the apparatus, at least one of the following: to refrain from using the video as input to an artificial intelligence model for training of the artificial intelligence model; or to refrain from using the video to determine a ground-truth for training an artificial intelligence model.

[0052] In at least one example embodiment, an AUR SEI message type associated with the AUR SEI message is associated with an SEI processing order (SPO) SEI message to indicate that the AUR SEI message indicates at least one of the following: characteristics of a processing chain indicated in the SPO SEI message, pictures output by a processing chain indicated in the SPO SEI message, or a result of a processing chain indicated in the SPO SEI message.

[0053] In at least one example embodiment, the AUR SEI message is included in a processing order nesting (PON) SEI message and is associated with a processing order value.

[0054] In at least one example embodiment, the processing order value associated with the AUR SEI message indicates that the AUR SEI message is descriptive of an output of a NNPF for post-process filtering associated with the video.

[0055] In at least one example embodiment, the instructions, when executed by the at least one processor, cause the apparatus at least to: decode the AUR SEI message; and determine that the at least one rule is applicable to a result of a processing chain based at least in part on the decoded AUR SEI message.

[0056] In at least one example embodiment, the SEI message comprises a neural network post filter (NNPF) SEI message indicating at least one of characteristics or activation of a NNPF for post-process filtering associated with the video, and wherein the NNPF SEI message includes at least one flag or at least one indicator that indicates the at least one rule.

[0057] In at least one example embodiment, the NNPF SEI message includes the at least one flag, and wherein the at least one flag indicates the at least one rule by indicating, to the apparatus, at least one of the following: to use one or more pictures output by the NNPF as input to an artificial intelligence model for training of the artificial intelligence model, torefrain from using one or more pictures output by the NNPF as input to an artificial intelligence model for training of the artificial intelligence model, to use one or more pictures output by the NNPF to determine a ground-truth for training an artificial intelligence model, or to refrain from using one or more pictures output by the NNPF to determine a ground-truth for training an artificial intelligence model.

[0058] In at least one example embodiment, the NNPF SEI message includes an indicator, and wherein a value of the indicator indicates the at least one rule by indicating, to the apparatus, at least one of the following: to refrain from using one or more pictures output by the NNPF both as input to an artificial intelligence model for training of the artificial intelligence model and to determine a ground-truth for training the artificial intelligence model, to use one or more pictures output by the NNPF as input to an artificial intelligence model for training of the artificial intelligence model, to use one or more pictures output by the NNPF to determine a ground-truth for training an artificial intelligence model, or to use one or more pictures output by the NNPF both as input to an artificial intelligence model for training of the artificial intelligence model and to determine a ground-truth for training the artificial intelligence model.

[0059] In at least one example embodiment, the NNPF SEI message includes a plurality of indicators associated with a plurality of processors in a processing chain.

[0060] In at least one example embodiment, the NNPF is comprised in a processing chain indicated via a SEI processing order (SPO) SEI message.

[0061] In at least one example embodiment, the NNPF is associated with distortions of images for stylistic or artistic functions.

[0062] In at least one example embodiment, the NNPF SEI message is indicative of one or more generative artificial intelligence characteristics associated with the NNPF, and wherein the at least one rule is based at least in part on the one or more generative artificial intelligence characteristics.

[0063] In at least one example embodiment, the NNPF SEI message indicates that the NNPF comprises a capability to output a plurality of pictures for each activation or inference of the NNPF, and wherein the at least one rule pertains to the plurality of pictures.

[0064] In at least one example embodiment, the NNPF SEI message comprises a neural network post filter characteristics (NNPFC) SEI message indicating characteristics of the NNPF or a neural network post filter activation (NNPF A) SEI message indicating activation of the NNPF for post-process filtering associated with the video.

[0065] In at least one example embodiment, the NNPF SEI message comprises an indication of whether an output of the NNPF comprises information pertaining to one or more pictures input to or output by the NNPF.

[0066] In at least one example embodiment, the second information comprises content information pertaining to respective content of one or more pictures associated with the video, and wherein the content information comprises semantic information or image quality information.

[0067] In at least one example embodiment, the at least one rule pertains to one or more operations performed using the one or more pictures associated with the video.

[0068] In at least one example embodiment, the at least one rule is based at least in part on whether the one or more pictures include faces.

[0069] In at least one example embodiment, the second information is indicative of a quality metric threshold, and wherein application of the at least one rule to the one or more pictures is based at least in part on whether a respective quality of the one or more pictures satisfies the quality metric threshold.

[0070] In at least one example embodiment, the at least one signal comprises third information indicative of a spatial region, within the one or more pictures, to which the at least one rule is applicable.

[0071] In at least one example embodiment, the third information indicates the spatial region by including a region type indication that indicates at least one of the following: whether the at least one rule is applicable to complete pictures, whether the at least one rule is applicable to one or more spatial regions associated with the region type indication, or whether the at least one rule is applicable to leftover areas, wherein a leftover area comprises a picture without one or more spatial regions associated with the region type indication.

[0072] In at least one example embodiment, the region type indication further indicates at least one of the following: one or more spatial regions indicated by at least one SEI message, one or more spatial regions indicated by an annotated regions SEI message, one or more regions of an annotated regions SEI message that are indicated in an AUR SEI message, or one or more spatial regions indicated through spatial coordinates within an AUR SEI message.

[0073] In at least one example embodiment, the at least one signal comprises fourth information indicating that the at least one rule is applicable to one or more types of data that is output by a neural network post filter (NNPF) and is associated with the one or morepictures, wherein the one or more types of data include at least one of the following: spatially extrapolated data, temporally extrapolated data, or temporally interpolated data.

[0074] In at least one example embodiment, the second information is indicative of one or more machine learning tasks, one or more machine vision types, or one or more artificial intelligence model types.

[0075] In at least one example embodiment, the at least one signal comprises third information indicative of the one or more machine vision types or the one or more artificial intelligence model types, and wherein the one or more machine vision types or the one or more artificial intelligence model types are based at least in part on a predetermined vocabulary.

[0076] In at least one example embodiment, the at least one rule indicates whether data associated with the video is usable for one or more operations associated with an artificial intelligence model, and wherein the one or more operations comprise at least one of the following: training, validation, or inference.

[0077] In at least one example embodiment, the at least one rule indicates a permission associated with an operation, and wherein, the instructions, when executed by the at least one processor, cause the apparatus at least to: perform the operation in accordance with the permission.

[0078] In at least one example embodiment, the at least one rule pertains to usage of the video with at least one of the following: artificial intelligence, machine learning, machine analysis, a neural network, human viewing or displaying, one or more derivative operations, one or more types of software applications, content authentication, or digital rights management.

[0079] In at least one example embodiment, a method is provided comprising: generating first information indicative of a video; generating second information indicative of at least one rule for usage of the video; and outputting at least one signal comprising a representation of the first information and the second information.

[0080] In at least one example embodiment, a method is provided comprising: receiving at least one signal comprising a representation of first information indicative of a video and second information indicative of at least one rule for usage of the video; and decoding the at least one signal to obtain the first information and the second information.

[0081] In at least one example embodiment, a non-transitory computer readable storage medium is provided. The non-transitory computer readable storage medium comprises computer instructions that, when executed by an apparatus, cause the apparatus to: generatefirst information indicative of a video; generate second information indicative of at least one rule for usage of the video; and output at least one signal comprising a representation of the first information and the second information.

[0082] In at least one example embodiment, an apparatus is provided comprising at least one processor and at least one memory including computer program code (e.g., instructions) configured to, with the at least one processor, cause the apparatus at least to: generate first information indicative of a video; generate second information indicative of at least one rule for usage of the video; and output at least one signal comprising a representation of the first information and the second information.

[0083] In at least one example embodiment, an apparatus is provided that comprises means for: generating first information indicative of a video; generating second information indicative of at least one rule for usage of the video; and outputting at least one signal comprising a representation of the first information and the second information.

[0084] In at least one example embodiment, an apparatus is provided that comprises means for: receiving at least one signal comprising a representation of first information indicative of a video and second information indicative of at least one rule for usage of the video; and decoding the at least one signal to obtain the first information and the second information.

[0085] In an embodiment, an apparatus comprises means for performing one or more methods as described in any of the previous paragraphs.

[0086] In an embodiment, a computer readable medium comprises instructions that, when executed by an apparatus, cause the apparatus at least to perform one or more methods as described in any of the previous paragraphs. In an embodiment, the computer readable medium comprises a non-transitory computer readable medium.

[0087] The above summary is provided merely for purposes of summarizing at least some example embodiments to provide a basic understanding of some aspects of the disclosure. Accordingly, it will be appreciated that the above-described embodiments are merely examples and should not be construed to narrow the scope of the disclosure in any way. It will also be appreciated that the scope of the disclosure encompasses many potential embodiments in addition to those summarized here, some of which will be further described below.BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Having thus described certain example embodiments of the present disclosure in general terms, reference will hereinafter be made to the accompanying drawings, which are not necessarily drawn to scale, and wherein:

[0089] FIG. 1 illustrates an example communication system to which one or more examples disclosed herein may be applied;

[0090] FIG. 2 illustrates an example block diagram of an apparatus to which one or more examples disclosed herein may be applied;

[0091] FIG. 3 illustrates an example block diagram of a communication system to which one or more examples disclosed herein may be applied;

[0092] FIG. 4 illustrates a flowchart according to an exemplary method to which one or more examples disclosed herein may be applied ; and

[0093] FIG. 5 illustrates a flowchart according to an exemplary method to which one or more examples disclosed herein may be applied.DETAILED DESCRIPTION

[0094] The following embodiments are exemplary. Although the specification may refer to “an”, “one”, or “some” embodiment(s) in several locations of the text, this does not necessarily mean that each reference is made to the same embodiment(s), or that a particular feature only applies to a single embodiment. Single features of different embodiments may also be combined to provide other embodiments. Further, when a particular feature, structure, or characteristic is described in connection of an embodiment, it is within the knowledge of one skilled in the art to apply such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described. It shall be understood that although the terms “first,” “second” and the like may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another.

[0095] For the purposes of the present disclosure, the phrases “at least one of A or B”, “at least one of A and B”, and “A and / or B” means (A), (B), or (A and B). For the purposes of the present disclosure, the phrase “A, B, and / or C” means (A), (B), (C), (A and B), (A and C), (B and C), or (A, B, and C).

[0096] Embodiments described may be implemented in a communications system (e.g., a communication network), such as any of the following radio access technologies (RATs): wireless fidelity (Wi-Fi), BLUETOOTH, Worldwide Interoperability for Micro-wave Access (WiMAX), Global System for Mobile communications (GSM, 2G), GSM EDGE radio accessNetwork (GERAN), General Packet Radio Service (GRPS), Universal Mobile Telecommunications system (UMTS, 3G) based on basic wideband-code division multiple access (W-CDMA), high-speed packet access (HSPA), Long Term Evolution (LTE), LTE-Advanced, and enhanced LTE (eLTE), 5G (also called NR), or any future radio access technology (RAT) such as 6G. Moreover, communication within the communication network may utilize any suitable wireless communication technology, comprising but not limited to: Code Division Multiple Access (CDMA), Frequency Division Multiple Access (FDMA), Time Division Multiple Access (TDMA), Frequency Division Duplex (FDD), Time Division Duplex (TDD), Multiple-Input Multiple-Output (MIMO), Orthogonal Frequency Division Multiplexing (OFDM), and / or Discrete Fourier Transform spread OFDM (DFT-s-OFDM).

[0097] The term “terminal device” refers to any end device that may be capable of wireless communication. By way of example, a terminal device may be referred to as a communication device, user equipment (UE), a Subscriber Station (SS), a Mobile Station (MS). The terminal device may include a mobile phone, a cellular phone, a smart phone, voice over IP (VoIP) phones, wireless local loop phones, a tablet, a wearable terminal device, a personal digital assistant (PDA), portable computers, desktop computers, image capture terminal devices such as digital cameras, gaming terminal devices, music storage and playback appliances, vehicle-mounted wireless terminal devices, universal serial bus (USB) USB dongles, an Internet of Things (loT) device, a watch or other wearable, a head-mounted display (HMD), a vehicle, a drone, a medical device and applications (e.g., remote surgery), an industrial device and applications (e.g., a robot and / or other wireless devices operating in an industrial and / or an automated processing chain contexts), a consumer electronics device, a device operating on commercial and / or industrial wireless networks, and the like.

[0098] FIG. 1 illustrates an example communication system to which one or more examples disclosed herein may be applied. In the example of FIG. 3, an encoder 30 may communicate with a decoder 32 and / or a memory 34. Video images may be encoded and therefore compressed by an encode prior to being stored and / or transmitted, such as to a decoder that then decodes and decompresses the video images. In this regard, FIG. 1 depicts an encoder 30 configured to encode frames of a video image so as to compress the frames prior to transmission to, for example, a decoder 32 and / or storage, such as in memory 34. The decoder of FIG. 1 is configured to receive the encoded frames of the video image from the memory device, such as following storage of the encoded frame, or from the encoder, either directly or via one or more intermediary devices. The decoder is then configured to decodeand, therefore, decompress the frames in order to reconstruct a representation of the video image.

[0099] The encoding and decoding of video images may be done in accordance with one or more video coding standards. For example, versatile video coding (VVC, H.266) is an international video coding standard and an enhanced compression model (ECM) that is built on top of VVC is being developed as a future video coding standard. Both VVC and ECM are block-based video coding standards in which an input picture is divided into coding tree units (CTUs) and each CTU may be further divided into coding units (CUs) or blocks. A CU may be coded in accordance with either an inter-coding mode or an intra-coding mode. For interceding in VVC and ECM, a sub-block transform (SBT) is a coding mode in which a coding unit that is to be encoded includes multiple transform units (TUs). Pursuant to the SBT, only one of the TUs is associated with a residual block and, as a result, is encoded, while the other TUs of the coding unit have no residual and, as a result, need not be encoded.

[0100] In the inter-coding mode, an encoder identifies, for a coding unit, a temporal prediction block in a reference picture of the video image. In the SBT coding mode, the encoder determines the residual block for a first TU of the coding unit in order to define the difference between that portion of the original block that corresponds positionally to the first TU and the corresponding portion of the temporal prediction block. However, in the SBT coding mode, the other TUs of the coding unit do not have an associated residual and, as a result, are not considered to be different than the corresponding portions of the temporal prediction block associated with the coding unit. In addition to determining the temporal prediction block associated with the coding unit and the residual block for the first TU of the coding unit, the encoder also determines motion information, such as motion vectors, reference pictures, reference lists or the like, in order to identify the temporal prediction block relative to the coding unit of the current image being encoded.

[0101] Another example of a video coding standard is H.265, also known as High-Efficiency Video Coding (HEVC). The H.265 (HEVC) standard is a video compression standard designed to improve compression efficiency relative to H.264. In some examples, video compression in the H.265 (HEVC) standard includes a series of operations to transform and encode video data. The operations may include block partitioning (or frame division) in which video frames are divided into smaller coding units (CUs), prediction units (PUs), and transform units (TUs), which may vary in size and may be used to apply one or more compression techniques.

[0102] In some examples, the operations may include application of intra-prediction to one or more CUs (e.g., each CU). Intra-prediction involves predicting the pixel values within a block based on neighboring pixels within the same frame. The difference between the actual pixels and the predicted pixels (residuals) may then be calculated.

[0103] In some examples, the operations may include inter-prediction, which involves using previously encoded frames as references to predict the current frame's pixels. Motion compensation may be applied to align the reference frame with the current frame, and residuals are calculated.

[0104] In some examples, the operations may include transform and quantization in which residuals obtained from intra- and inter-predictions are transformed using various transform sizes (e.g., 4x4, 8x8, 16x16) and using methods such as a Discrete Cosine Transform (DCT). The resulting transform coefficients are then quantized to reduce their precision and remove fine details.

[0105] In some examples, the operations may include inverse transform and quantization in which quantized transform coefficients from both intra and inter prediction are first inversely quantized, which scales the quantized values back to an original range. In some such examples, an inverse discrete cosine transform (inverse DCT) may be applied to convert the frequency coefficients back into pixel values.

[0106] In some examples, the operations may include in-loop filtering, which includes deblocking filtering and sample adaptive offset (SAO) filtering, which may be applied to smooth block boundaries and reduce artifacts caused by quantization. In some examples, inloop filtering leads to an improved visual quality of the reconstructed image.

[0107] In some examples, the operations may include the use of a reference frame. For example, a reference frame, which is a complete image (frame) in a video sequence, may be used as a guide to predict other frames. In other words, reference frames may act as anchor points for predicting the content of subsequent frames.

[0108] In some examples, the operations may include entropy coding in which quantized transform coefficients are entropy encoded using context-adaptive binary arithmetic coding (CABAC). In some examples, of CABAC, shorter codes may be assigned to more frequent coefficients and longer codes to less frequent coefficients.

[0109] In some examples, the operations may include bitstream generation in which encoded data, along with information about coding modes, motion vectors, and other parameters, is assembled into a structured bitstream. This bitstream includes information for decoding the compressed video. The compressed bitstream may be stored or transmitted to bedecoded at a decoder (or receiver). During decoding, the compressed data undergoes a reverse process to reconstruct the original video frames.

[0110] The phrase “along the bitstream,” and the like (e.g. indicating along the bitstream), may be defined to refer to out-of-band transmission, signaling, or storage in a manner that the out-of-band data is associated with the bitstream. The phrase “decoding along the bitstream,” and the like, refers to decoding the referred out-of-band data (which may be obtained from out-of-band transmission, signaling, or storage) that is associated with the bitstream. For example, an indication along the bitstream may refer to metadata in a container file that encapsulates the bitstream.

[0111] A neural network is a computation graph comprising several layers of computation. Each layer comprises one or more units, where each unit performs an elementary computation. A unit is connected to one or more other units, and the connection may have associated a weight. The weight may be used for scaling the signal passing through the associated connection. Weights are usually learnable parameters, (e.g., values which may be learned from training data). There may be other learnable parameters, such as those of batch-normalization layers.

[0112] As used herein, the terms “model,” “machine learning environment,” “deep learning framework,” “neural network,” “neural net,” and “network” are used interchangeably, and may refer to a framework or environment for algorithms and the like to be implemented to process complex data inputs. In some implementations of the embodiments described herein, a “neural network” and the like may comprise a computation graph comprising several layers of computation.

[0113] As used herein, the weights or weighting values used for neural networks are sometimes referred to as “learnable parameters,” “learned parameters,” or simply as “parameters,” terms which are used interchangeably herein.

[0114] Two examples of architectures for neural networks are feed-forward and recurrent architectures. Feed-forward neural networks are such that there is no feedback loop: each layer takes input from one or more of the layers before and provides its output as the input for one or more of the subsequent layers. Also, units included in certain layers take input from units in one or more of preceding layers and provide output to one or more of following layers.

[0115] Initial layers (those close to the input data) extract semantically low-level features such as edges and textures in images, and intermediate and final layers extract more high-level features. After the feature extraction layers there may be one or more layers performinga certain task, such as classification, semantic segmentation, object detection, denoising, style transfer, super-resolution, etc. In recurrent neural networks, there is a feedback loop, so that the network becomes stateful, e.g., it is able to memorize information or a state.

[0116] Neural networks may be utilized in an increasing number of applications for many different types of device, such as mobile phones. Examples include image and video analysis and processing, social media data analysis, device usage data analysis, etc. Some neural networks (and other machine learning tools) are able to learn properties from input data, either in a supervised way or in an unsupervised way. Such learning is a result of a training algorithm, or of a meta-level neural network providing the training signal.

[0117] A neural network may be generated, trained, tested, implemented, and otherwise controlled or affected through the use of any suitable computing device or other such apparatus or device, such as any of those described herein, and the like. As an example, a neural network may be generated by a processor and stored on a memory, such as a server or the like. In this regard, the encoder 30 (or a device including the encoder 30), the decoder 32 (or a device including the decoder 32) may be an example of an apparatus and system including at least a processor for generating a neural network, which may be stored on the memory 34 (or other memory).

[0118] ITU-T Recommendation H.274, which is equivalent to ISO / IEC 23002-7, may be called "versatile supplemental enhancement information messages for coded video bitstreams" and be referred to as "versatile supplemental enhancement information" or VSEI. The VSEI standard specifies the syntax and semantics of video usability information (VUI) parameters and supplemental enhancement information (SEI) messages. The VUI parameters and SEI messages defined in the VSEI standard are designed to be conveyed within coded video bitstreams in a manner specified in a video coding specification or to be conveyed by other means determined by the specifications for systems that make use of such coded video bitstreams. The VSEI standard is intended for use with H.266 coded video bitstreams, although it is specified in a manner intended to be sufficiently generic that it may also be used with other types of coded video bitstreams.

[0119] In some examples, VVC may be improved through an artificial intelligence (Al) extension that enhances picture quality and provides functionality. For example, a versatile supplemental enhancement information (VSEI) standard may be used with VVC to (among other things) enable a device to enhance image quality after decoding the content. The VSEI standard defines information for tasks beyond traditional video coding, such as adapting decoded video to display. VSEI includes the syntax and semantics of supplementalenhancement information (SEI) messages, which allow decoders to interpret the SEI messages consistently across different systems.

[0120] In some examples of the VSEI standard, two SEI messages are defined for neural-network-based post-processing: the neural-network post-filter characteristics (NNPFC) SEI message and the neural-network post-filter activation (NNPFA) SEI message. These messages enable the flexible application of neural networks as post-filters, which improves decoded pictures and / or creates additional interpolated pictures.

[0121] In some examples, the NNPFC SEI message allows an encoder to define a neural network for post-processing after decoding. The NNPFC may also specifies the input and output of the neural network and describes the neural networks complexity. Additionally, the NNPFC SEI message may indicate the intended purpose of the neural network, which may include, for example, enhancing visual quality, changing spatial resolution (e.g., from high-definition decoded video to ultra-high definition), upsampling picture rate (e.g., from 30 Hz to 60 Hz), upsampling bit depth to increase the dynamic range of pixel values, and / or colorization to convert monochrome video to full colors. By implementing neural-network filters as a post-processing step, utilization of the neural-network filters may be introduced to services without negatively impacting older systems that lack the ability to execute neural networks.

[0122] The syntax structure specifying the NNPFC SEI message may be referred to as nn_post_filter_characteristics. The syntax structure specifying the NNPFA SEI message may be referred to as nn_post_filter_activation.

[0123] The NNPFC SEI message comprises the nnpfc_id syntax element, which includes an identifying number that may be used to identify a post-processing filter. The NNPFC SEI message syntax comprises nnpfc_base_flag. nnpfc_base_flag equal to 1 specifies that the SEI message specifies the base NNPF. The nnpfc_base_flag equal to 0 specifies that the SEI message specifies an update relative to the base NNPF.

[0124] One or more constraints may be applied to the value of nnpfc_base_flag. Such constraints may specify that when an NNPFC SEI message is the first NNPFC SEI message, in decoding order, that has a particular nnpfc_id value within the current CLVS, the value of nnpfc_base_flag is required to be equal to 1. Additionally, or alternatively, such constraints may specify that NNPFC SEI messages (e.g., all NNPFC SEI messages) in a CLVS that have a particular nnpfc_id value and nnpfc_base_flag equal to 1 are required to have identical SEI payload content. Additionally, or alternatively, such constraints may specify that when nnpfc_base_flag is equal to 0, the following applies: this SEI message defines an updaterelative to the preceding base NNPF in decoding order with the same nnpfc_id value; updates are not cumulative but rather each update is applied on the base NNPF, which is the NNPF specified by the first NNPFC SEI message, in decoding order, that has a particular nnpfc_id value within the current CLVS; and / or the NNPF defined by this SEI message is obtained by applying the update defined by this SEI message relative to the base NNPF with the same nnpfc_id value. Additionally, or alternatively, such constraints may specify that his SEI message pertains to the current decoded picture and all subsequent decoded pictures of the current layer, in output order, until the end of the current CLVS or up to but excluding the decoded picture that follows the current decoded picture in output order within the current CLVS and is associated with a subsequent NNPFC SEI message, in decoding order, having nnpfc_base_flag equal to 0 and that particular nnpfc_id value within the current CLVS, whichever is earlier.

[0125] In some examples, the NNPFC SEI message comprises the nnpfc_mode_idc syntax element. In some such examples, the semantics may be defined as follows: nnpfc_mode_idc equal to 1 specifies that the base post-processing filter or the update relative to the base post-processing filter associated with the nnpfc_id value is a neural network identified by the Uniform Resource Identifier (URI) nnpfc_uri with the format identified by the tag URI nnpfc_tag_uri; and nnpfc_mode_idc equal to 0 indicates that this SEI message contains an ISO / IEC 15938-17 bitstream that specifies the base post-processing filter or updates relative to the base post-processing filter with the same nnpfc_id value.

[0126] In some examples, when nnpfc_mode_idc is equal to 0 and nnpfc_base_flag is equal to 0, the update may be obtained by decoding the coded neural network bitstream included in the NNPFC SEI message.

[0127] In some examples, the NNPFC SEI message may also comprise a purpose of the post-processing filter, which may comprise, but may not be limited to, one or more of the following: visual quality improvement; chroma upsampling from the 4:2:0 chroma format to the 4:2:2 or 4:4:4 chroma format, or from the 4:2:2 chroma format to the 4:4:4 chroma format; increasing the width or height of the input picture; frame rate upsampling; bit depth upsampling; colorization; temporal extrapolation (e.g., generating one or more future pictures); or spatial extrapolation (e.g., extending one or more pictures, possibly additionally including cropping). Additionally, or alternatively, the NNPFC SEI message may also comprise formatting of the input tensors that are given as input to the neural network inference, formatting of the output tensors that are resulting from the neural network inference, and / or characterization of the complexity of the neural network.

[0128] The NNPFC SEI message syntax may include the nnpfc_num_input_pics_minusl syntax element. nnpfc_num_input_pics_minusl plus 1 specifies the number of pictures used as input for the NNPF. The variable numlnputPics may be set equal to nnpfc_num_input_pics_minusl + 1.

[0129] In some examples, when nnpfc_num_input_pics_minusl is greater than 0, the NNPFC SEI message syntax includes nnpfc_input_pic_filtering_flag[ i ] for each value of i in the range of 0 to nnpfc_num_input_pics_minusl, inclusive. nnpfc_input_pic_filtering_flag[ i ] equal to 1 indicates that for the i-th input picture the NNPF generates a corresponding output picture. nnpfc_input_pic_filtering_flag[ i ] equal to 0 indicates that for the i-th input picture the NNPF does not generate a corresponding output picture. Each NNPF-generated picture is stored in the output tensor of the NNPF. When nnpfc_num_input_pics_minusl is equal to 0, nnpfc_input_pic_filtering_flag

[0000] is inferred to be equal to 1.

[0130] In some examples, a frame rate upsampling filter may interchangeably be called a picture rate upsampling filter. Such a filter generates or interpolates one or more pictures between a pair of pictures given as input to the filter. It is also possible to have a frame rate upsampling filter where the number of input pictures may be greater than 2. Such a frame rate upsampling filter may generate pictures between more than one pair of input pictures. A frame rate upsampling filter may comprise a neural network, in which case the generation of the interpolated pictures between a pair of input pictures is performed by the inference of the neural network. It is possible to have a frame rate upsampling filter that extrapolates a picture before input picture(s) or after input picture(s), instead of or in addition to between input pictures.

[0131] In some examples, when the filtering purpose comprises frame rate upsampling, the NNPFC SEI message includes nnpfc_interpolated_pics[ i ] syntax elements for the values of i in the range of 0, inclusive, to nnpfc_num_input_pics_minusl, exclusive. nnpfc_interpolated_pics[ i ] specifies the number of interpolated pictures generated by the NNPF between the i-th and the ( i + 1 )-th picture used as input for the NNPF.

[0132] In some examples, when the filtering purpose comprises temporal extrapolation, the NNPFC SEI message includes the nnpfc_extrapolated_pics_minusl syntax element, in which the nnpfc_extrapolated_pics_minusl plus 1 specifies the number of extrapolated pictures generated by the NNPF subsequent to all input pictures for the NNPF in output order.

[0133] In some examples, the NNPFC SEI message syntax may comprise an indication, which may be referred to as nnpfc_absent_input_pic_zero_flag, which indicates how pictures that would not originate from the current bitstream are expected to be replaced in the input tensor. In some examples, the nnpfc_absent_input_pic_zero_flag equal to 1 indicates that the NNPF expects an input picture that is not present in the current bitstream to be represented sample arrays with sample values equal to 0. In some examples, the nnpfc_absent_input_pic_flag equal to 0 indicates that the NNPF expects an input picture that is not present in the current bitstream to be represented by the closest input picture in output order within the current bitstream.

[0134] The NNPFC SEI message may include the nnpfc_inp_order_idc syntax element, which indicates the method of ordering the sample arrays of an input picture to form an input tensor to the NNPF. In some examples, of the NNPFC SEI message, values of nnpfc_inp_order_idc may be described as follows: when nnpfc_inp_order_idc is equal to 0, one luma matrix is present in the input tensor for each input picture; when nnpfc_inp_order_idc is equal to 1 , two chroma matrices are present in the input tensor; when nnpfc_inp_order_idc is equal to 2, one luma and two chroma matrices are present in the input tensor; and when nnpfc_inp_order_idc is equal to 3, four luma matrices and two chroma matrices are present in the input tensor and the luma channels are derived in an interleaved manner. In some examples, nnpfc_inp_order_idc equal to 3 may (only) be used when the input chroma format is 4:2:0.

[0135] The NNPFC SEI message provides an extensible auxiliary input mechanism, which enables to input data other than the input pictures to the neural network inference. The NNPFC SEI message syntax may comprise an indication, which may be called nnpfc_auxiliary_inp_idc, that indicates if auxiliary input data in addition to sample array(s) of input picture(s) is present in the input tensor of the NNPF. nnpfc_auxiliary_inp_idc greater than 0 indicates that auxiliary input data is present in the input tensor of the NNPF. Specific semantics may be specified for specific non-zero values of nnpfc_auxiliary_inp_idc. nnpfc_auxiliary_inp_idc equal to 0 indicates that auxiliary input data is not present in the input tensor.

[0136] In some examples, the value range of nnpfc_auxiliary_inp_idc is specified to be up to 255, inclusive. In some examples of VSEI, one auxiliary input type, namely the QP-based StrengthControlValf i ], is defined, which is indicated by nnpfc_auxiliary_inp_idc equal to 1.

[0137] The NNPFA SEI message may specify the neural-network post-processing filter (NNPF) that may be used for post-processing filtering for a current picture, or for postprocessing filtering for the current picture and one or more other pictures. The NNPFA SEI message comprises the nnpfa_target_id syntax element, which indicates that the neural-network post-processing filter with nnpfc_id equal to nnpfa_target_id may be used for postprocessing filtering for the indicated persistence. The indicated persistence may be the current picture (e.g., indicated by nnpfa_persistence_flag equal to 0). Alternatively, the NNPF activation may be indicated to be persistent by nnpfa_persistence_flag equal to 1 , in which case the persistence of the NNPF activation may last until the end of the current CEVS or the next picture, in output order, in the current layer associated with a NNPFA SEI message with the same nnpfa_target_id as the current SEI message.

[0138] The NNPFA SEI message syntax may comprise a syntax element indicative of the base post-processing filter or the latest post-processing filter is activated, where the latest post-processing filter is defined by the base post-processing filter relative to which the latest filter update, if any, has been applied. The syntax element may be referred to as nnpfa_target_base_flag. In some examples, nnpfa_target_base_flag equal to 1 specifies that the target NNPF is the base NNPF with nnpfc_id equal to nnpfa_target_id. nnpfa_target_base_flag equal to 0 specifies that the target NNPF is the NNPF specified by the last NNPFC SEI message with nnpfc_id equal to nnpfa_target_id that precedes the first VCL NAE unit of the current picture in decoding order and is not a repetition of the NNPFC SEI message that contains the base NNPF.

[0139] The NNPFA SEI message syntax may indicdate which ones of the filtered pictures corresponding to the input pictures are output by the NNPF process. For the i-th input picture that is filtered by the NNPF, the NNPFA SEI message syntax may comprise nnpfa_output_flag[ i ] syntax element, which when equal to 0, specifies that the filtered picture is not output by the NNPF process, and when equal to 1, specifies that the filtered picture is output by the NNPF process.

[0140] In relation to an NNPFA SEI message, two sets of pictures may be defined, namely nnpfcTargetPictures and nnpfaTargetPictures. In some examples, nnpfcTargetPictures may be defined to be the set of pictures to which the last NNPFC SEI message with nnpfc_id equal to nnpfa_target_id that precedes the current NNPFA SEI message in decoding order pertains. In some examples, nnpfaTargetPictures may be defined to be the set of pictures for which the target NNPF is activated by the current NNPFA SEImessage. In some examples, for a conforming bitstream, a picture included in nnpfaTargetPictures may also be included in nnpfcTargetPictures.

[0141] An NNPF process may comprise performing the NNPF inference for given input pictures. The NNPF inference may be performed in a patch-wise manner so that the entire picture area is filtered. The NNPF inference may be followed by outputting NNPF-generated pictures in increasing index order, in which the NNPF-generated pictures that were interpolated by the NNPF are output and those NNPF-generated pictures that correspond to any input pictures to the NNPF are output as specified in the semantics of the NNPFA SEI message.

[0142] A general post-processing filtering process using NNPFs may be described as follows. Input to this process is a bitstream BitstreamToFilter. Output of this process is a list of NNPF output pictures ListNnpfOutputPics. First, BitstreamToFilter is decoded, and the list CroppedDecodedPictures is set to be the list of the cropped decoded pictures in output order resulted from decoding BitstreamToFilter. Second, the filtering process for one picture, as described below, is repeatedly invoked, in output order, for each cropped decoded picture that is in CroppedDecodedPictures and for which one or more NNPFs are activated. The order of the pictures in ListNnpfOutputPics is in output order. In some examples, within ListNnpfOutputPics, there may be no more than one picture pertaining to any particular output time instance. In some examples, when for any particular picture in CroppedDecodedPictures there are multiple NNPFs activated and one the NNPFs is allowed to be chosen to be applied (although any of the NNPFs may be chosen), the above constraint may apply irrespective of which NNPF is chosen to be applied to the particular picture.

[0143] A filtering process for one picture using an NNPF may be described as follows. The filtering process for one picture using an NNPF may be applied to each cropped decoded picture, referred to as the current picture, that is in CroppedDecodedPictures and for which one or more NNPFs are activated. When applying an NNPF to the current picture, the filtered and / or interpolated pictures are generated by the NNPF by applying the NNPF process to the current picture. When applying an NNPF to the current picture, the order of the pictures generated by the NNPF by applying the NNPF process being stored into the output tensor of the NNPF is in output order. When the applied NNPF is the last NNPF that is applied to the current picture, the pictures generated by the NNPF and output by the NNPF process are included into ListNnpfOutputPics, in the same order as when the pictures are stored into the output tensor of the NNPF.

[0144] The use of NNPFC and NNPFA SEI messages for VVC may be described in a standard, such as the VVC standard. It is to be understood that NNPFC and NNPFA SEI message may be similarly used for any other video coding specification.

[0145] In some examples, when NNPFC and NNPFA SEI messages are used for VVC, a decoder selects input pictures for the NNPF. The input pictures may be selected in reverse output order starting from a picture for which the NNPF is activated through an NNPFA SEI message. The input pictures may be indexed, starting from index 0 that is assigned for the picture for which the NNPF is activated through an NNPFA SEI message. In at least one example, the decoder selects the input picture with index i, where i is greater than 0, to be the latest cropped decoded output picture, in output order, that precedes the input picture with index i-1 in output order. When there is no cropped decoded output picture, in output order, that precedes the input picture with index i-1 in output order as a result of decoding the bitstream, it may be considered that the input picture with index i is not present in the current bitstream (e.g., missing) and the subsequent input pictures, when any, with index i+1 to numlnputPics-l, inclusive, are likewise missing. A missing input picture may be treated as described above in relation to nnpfc_absent_input_pic_zero_flag syntax element.

[0146] In some examples in which NNPFC and NNPFA SEI messages are used for VVC and a picture rate upsampling NNPF that interpolates pictures between a single pair of input pictures is activated persistently until the end of the bitstream, the NNPF is applied repeatedly at the end of the bitstream for different sets of input pictures up to but excluding a set of input pictures that would cause creation of any interpolated picture after the last picture of the bitstream in output order. In these sets of input pictures, some of the pictures may be missing and may be, for example, replaced by the last picture within the bitstream in output order.

[0147] In some examples, a standard may specify one or more types of SEI processing order (SPO) SEI messages and / or one or more types of processing order nesting (PON) SEI messages. A SPO SEI message may carry information indicating the preferred processing order, as determined by the encoder (e.g., the content producer), for a group of types of SEI messages that may be present in a CLVS. The processing order is indicated by indicating processing order values for each SEI message type included in the processing chain. A processing chain comprises a list of types of SEI messages identified by an SPO SEI message in the preferred processing order indicated in the SPO SEI message. A PON SEI message may include one or more SEI messages that may be applied only as parts of the processing chain identified by an associated SEI processing order SEI message and may not be appliedin a manner that would contradict with the processing chain identified by the associated SEI processing order SEI message. The PON SEI message may include a processing order value for the SEI message included in the PON SEI message.

[0148] In some examples, a standard may specify one or more types of text description SEI messages. Terms of a syntax table and semantics for a text description SEI message may be defined in accordance with the following Table 1.<

[0149] In the example of Table 1, tdi_descr_purpose indicates the purpose of the text description SEI as specified in the following Table 2.

[0150] The value of text_descr_purpose may be in the range of 0 to 5, inclusive. Values in the range of 6 to 255, inclusive, for text_descr_purpose may be reserved for future use (e.g., by ITU-T I ISO / IEC) and may not be present in a bitstream, such as a bitstream conforming to a standard. In some examples, decoders conforming to the standard may allow a value of text_descr_purpose in the range of 0 to 255, inclusive. In some examples, tdi_purpose_cancel_flag equal to 1 indicates that the text description information SEI message cancels the persistence of any previous text description information SEI message with the same tdi_descr_purpose in output order that applies to the current layer. tdi_purpose_cancel_flag equal to 0 indicates that text description information follows.

[0151] In some examples, tdi_descr_id indicates the identifier value of this text description information SEI message. The value of tdi_descr_id may be in the range of 1 to 8191, inclusive. In some examples, value 0 and values 8064-8191 are reserved. Text description SEI messages with different values for tdi_descr_purpose may use separate values spaces for tdi_descr_id.

[0152] In some examples, tdi_id_cancel_flag equal to 1 indicates that the text description information SEI message cancels the persistence of any previous text description information SEI message with the same tdi_descr_id and tdi_descr_purpose values as those in the current SEI in output order that applies to the current layer. In some examples, tdi_id_cancel_flag equal to 0 indicates that text description information syntax elements (tdi_persistence_flag, tdi_num_strings_minusl, tdi_descr_string_lang[ i ], tdi_descr_string[ i ] ) follow.

[0153] In some exampels, tdi_persistence_flag specifies the persistence of the text description information SEI message with identifier equal to tdi_descr_id and purpose equal to tdi_descr_purpose for the current layer. In some examples, tdi_persistence_flag equal to 0 specifies that the text description information SEI message with identifier equal to tdi_descr_id and purpose equal to tdi_descr_purpose applies to the current decoded picture only. In some exampels, tdi_persistence_flag equal to 1 specifies that the text description information SEI message with identifier equal to tdi_descr_id and purpose equal to tdi_descr_purpose applies to a current decoded picture and persists for all subsequent pictures of the current layer in output order until one or more of the following conditions are true: a new CLVS of the current layer begins; the bitstream ends; and a picture in the current layer in an AU associated with a text description information SEI message with the same values of tdi_descr_id and tdi_descr_purpose is output that follows the current picture in output order.

[0154] In some examples, tdi_num_strings_minusl plus 1 indicates the number of entries for tdi_descr_string_lang[ i ] and tdi_descr_s tring [ i ] that follow. In some examples,tdi_descr_string_lang[ i ] specifies the language of the tdi_descr_s tring [ i ]. The language of the tdi_descr_s tring [ i ] may be given by a language tag as defined, for example, by IETF RFC 5646. The length of tdi_descr_string_lang[ i ] corresponding to the stringLength variable for the st(v) parsing process may be in the range of 0 to 49 bytes, inclusive. The value of tdi_descr_string_lang[m] may not be equal to the value of tdi_descr_string_lang[ n ] when m is not equal to n, for any values of m and n in the range from 0 to tdi_num_strings_minusl , inclusive.

[0155] In some examples, tdi_descr_string[ i ] specifies i-th text description information string whose value is interpreted as specified by the tdi_descr_purpose. In some such examples, when tdi_descr_purpose is equal to 0, the interpretation of what information is conveyed in the tdi_descr_string is application-defined. When tdi_descr_purpose is equal to 1 , the tdi_descr_string[ i ] specifies copyright information that pertains to the picture(s) in the persistence scope of this SEI message. When tdi_descr_purpose is equal to 2, the tdi_descr_s tring [ i ] specifies, when not a null string, Al marking information that pertains to the picture(s) within the persistence scope of this SEI message. In some examples, when tdi_descr_purpose is equal to 2 the string may include information about machine-learning-based processing, intended use of the decoded pictures, or other aspects relevant to the associated pictures. When tdi_descr_purpose is equal to 3, the tdi_descr_string[ i ] specifies a general text label description that pertains to the picture(s) in the persistence scope of this SEI message. When tdi_descr_purpose is equal to 4, the tdi_descr_string[ i ] may specify content advisory rating information conforming to US and Canadian Rating Region Tables (RRT) as defined, for example, by CTA-766-D that pertains to the picture(s) in the persistence scope of this SEI message. When tdi_descr_purpose is equal to 5, tdi_descr_s tring [ i ] may contain a tag URI with syntax and semantics as specified, for example, in IETF RFC 4151 identifying the CLVS. When tdi_descr_purpose is equal to 6, the tdi_descr_s tring [ i ] specifies a description of the encoder used to produce the coded picture(s) in the persistence scope of this SEI.

[0156] In some examples, a standard may specify one or more types of Al usage restrictions (AUR) SEI message. Terms of a syntax table and semantics for an AUR SEI message may be defined in accordance with the following Table 3.<

[0157] An AUR SEI message may signal one or more rules (e.g., restrictions and optional context information) for usage by Al applications. In some examples, aur_cancel_flag equal to 1 specifies that the SEI message cancels the persistence of any previous Al usage restrictions SEI message in output order. In some such examples, aur_cancel_flag equal to 0 specifies that Al usage restriction information follows.

[0158] In some examples, aur_persistence_flag specifies the persistence of the Al usage restrictions SEI message for the current layer. In some such examples, aur_persistence_flag equal to 0 specifies that the Al usage restrictions SEI message applies to the current decoded picture and aur_persistence_flag equal to 1 specifies that the Al usage restrictions SEI message applies to the current decoded picture and persists for all subsequent pictures of the current layer in output order until one or more of the following conditions are true: a new CL VS of the current layer begins; the bitstream ends, and a picture in the current layer in an AU associated with an Al usage restrictions SEI message is output that follows the current picture in output order.

[0159] In some examples, aur_num_restrictions_minusl plus one specifies the number of restriction entries that follow. In some examples, aur_restriction[ i ] specifies the i-th restriction as specified in Table 2. The value of aur_restriction[ i ] may be in the range of 0 to 2, inclusive. In some examples, values greater than 2 for aur_restriction[ i ] are reserved for future use (e.g., by ITU-T I ISO / IEC) and may be ignored in bitstreams (e.g., bitstreams conforming the standard). Interpretations for values of aur_restriction[ i ] may be defined in accordance with the following table 4.

[0160] aur_context_present_flag[ i ] equal to 1 specifies that the context information for aur_restriction[ i ] is present. aur_context_present_flag[ i ] equal to 0 specifies that no context information is present for aur_restriction[ i ]. When aur_context_present_flag[ i ] is equal to 0, aur_restriction [ i ] applies regardless of the context.

[0161] In some examples, aur_context[ i ], when present, indicates the context for the i-th aur_restriction[ i ] as specified in the following Table 5, where (aur_context & bitMask ) not equal to zero indicates that the i-th restriction applies to the context associated with the bitMask value in Table 5.

[0162] In some examples, when aur_context[ i ] is greater than 0 and ( aur_context[ i ] & bitMask ) is equal to zero, the application of the i-th restriction is undefined in the context associated with the bitMask value in Table 5. The value of aur_context[ i ] may be in the range of 1 to 15, inclusive, in a bitstream (e.g., a bitstream conforming to the standard). In some examples, values greater than 15 are reserved for future use (e.g., by ITU-T I ISO / IEC) and may be ignored in a bitstream (e.g., a bitstream conforming to the standard). In some examples, when the value of aur_context[ i ] is greater than 15 decoders (e.g., decoders conforming to the standard) may ignore aur_context[ i ]. In some examples, aur_context[ i ] may not be equal to 0.

[0163] In some examples, a standard may specify one or more quality metrics SEI message. In some such examples, a quality metrics SEI message may signal (e.g., include information indicating) one or more quality metric values. In some examples, a quality metric value may indicate the quality of a single picture, the mean quality of all the pictures corresponding to a CLVS, the quality gain of a single picture (which is difference betweenthe quality of a single picture relative to the quality of a gain reference picture), and / or the mean quality gain of all the pictures corresponding to a CLVS. In some examples, a quality metrics SEI message may provide for indicating the quality in one or more indicated quality metric types. In some such examples, the quality metric types include, for example, the peak signal-to-noise ratio (PSNR) and the structured similarity index measure (SSIM).

[0164] In some examples, a standard (e.g., VSEI standard (ISO / IEC 23002-7 I ITU-T H.274), HEVC (ISO / IEC 23008-2 I ITU-T H.265)) may specify one or more annotated regions SEI message. In some such examples, an annotated regions SEI message may carry one or more parameters that identify annotated regions using bounding boxes representing the size and location of identified objects. The annotated regions SEI message may comprise one or more types of information. The one or more types of information may include a ar_not_optimized_for_viewing_flag, in which ar_not_optimized_for_viewing_flag equal to 1 indicates that the decoded pictures that the annotated regions SEI message applies to are not optimized for user viewing, but rather are optimized for some other purpose such as algorithmic object classification performance and ar_not_optimized_for_viewing_flag equal to 0 indicates that the decoded pictures that the annotated regions SEI message applies to may or may not be optimized for user viewing.

[0165] Additionally, or alternatively, the one or more types of information may include a ar_true_motion_flag, in which ar_true_motion_flag equal to 1 indicates that the motion information in the coded pictures that the annotated regions SEI message applies to was selected with a goal of accurately representing object motion for objects in the annotated regions and ar_true_motion_flag equal to 0 indicates that the motion information in the coded pictures that the annotated regions SEI message applies to may or may not have been selected with a goal of accurately representing object motion for objects in the annotated regions.

[0166] Additionally, or alternatively, the one or more types of information may include a ar_occluded_object_flag, in which ar_occluded_object_flag equal to 1 indicates that each of the bounding boxes represents the size and location of an object or a portion of an object that may not be visible or may be partially visible within the cropped decoded picture that the annotated regions SEI message applies to. ar_occluded_object_flag equal to 0 indicates that each of the bounding boxes represents the size and location of an object that is entirely visible within the cropped decoded picture that the annotated regions SEI message applies to.

[0167] Additionally, or alternatively, the one or more types of information may include at least one of the following: one or more textual labels (ar_label[ ar_label_idx[ i ] ]), which are assigned indices ar_label_idx[ i ]; a language used in the labels; a mapping of an object to alabel index; or a bounding box of an object. In some examples, the one or more types of information may include an indication of whether the bounding box represents the size and location of an object that is only partially visible within the cropped decoded picture and / or a degree of confidence associated with an object.

[0168] The terms “mask”, “map”, “mask image”, “mask information image”, and “map” image are used interchangeably herein. In these terms, image, picture, and frame may be interchangeably used. Likewise, terms mask video, mask information video, and map video are used interchangeably herein. These terms may be prefixed by the type of the mask or map.

[0169] Masks may be binary or may have multiple distinct sample value levels or value ranges.

[0170] Masks may be pixel-accurate or their granularity may be coarser, such as 8x8 block granularity.

[0171] An object mask may identify one or more objects of a picture. An object mask may result from an instance segmentation or object detection algorithm, for example. In other words, an object mask may map mask sampling grid locations to identified objects.

[0172] In some cases, an object mask may be a binary matrix or image, wherein a value, such as “0”, may represent background and another value, such as “1”, may represent the foreground.

[0173] In some cases, an object mask may be a monochrome (grayscale) image. An object mask may be represented with a luma sample array, having bit-depth equal to BitDepthY. Thus, there are l«BitDepthY different values for each luma sample (where x « y is arithmetic left shift of a two's complement integer representation of x by y binary digits). To distinguish different masks within one picture, a selected sample value or a selected sample value range may represent a single object of the object mask.

[0174] Other terms that may be used similarly to object include annotated region and region of interest (ROI). An annotated region or an ROI does not necessarily represent a single object but may be more generally defined to identify region(s) having some annotation and / or of different interest or importance. A region-of-interest mask may indicate one or more regions of interest in an associated image. A region of interest may be perceptually more important to human viewers or may be more important to improve a computer vision task accuracy when compared to areas outside of the regions of interest. Annotated region and / or ROI masks may be represented similarly to object masks.

[0175] A saliency map may be defined as an image that highlights the region(s) on which viewers are likely to focus. Alternatively, a saliency may reflect the likely most relevant regions for computer vision tasks. Sample values of a saliency map may correspond to a degree of importance of the samples to the human visual system or to computer vision task(s).

[0176] In some instances, use cases associated with one or more types of SEI messages may provide insufficient coverage for use cases associated with videos. For example, an AUR SEI message may provide relatively coarse usage restrictions and lack coverage for various uses of a video or picture (e.g., video image). In some examples, an AUR SEI message may lack the capability to distinguish whether a video is usable as input for Al training or as ground-truth for Al training. Additionally, or alternatively, an AUR SEI message may lack Al usage restrictions for post-processed video, such as video that has been processed by a neural-network post-processing filter (NNPF) or a processing chain controlled by SEI messages, an AUR SEI message may lack Al usage restrictions related to auxiliary outputs resulting by neural-network post-filtering, where the auxiliary outputs may comprise, for example, semantic information about the content of pictures. Additionally, or alternatively, an AUR SEI message may lack Al usage restrictions depending on information about content of pictures. Additionally, or alternatively, an AUR SEI message may lack Al usage restrictions depending on picture quality. Additionally, or alternatively, an AUR SEI message may lack Al usage restrictions specific for spatial regions.

[0177] Various aspects of the present disclosure provide a framework for increasing the granularity of usage restrictions that apply to a video (or picture), thereby reducing or removing ambiguities associated with usage of the video at a decoder (e.g., the decoder 32) and, as such, improving the performance of the decoder. In accordance with the framework for usage restrictions of videos described herein, the encoder 30 may signal, in or along a bitstream, and the decoder 32 to decode, from or along the bitstream, information indicative of how a video may be used, where the video or a signal from which the video is derived may be included in the bitstream. In other words, the framework for usage restrictions of videos described herein may enable an encoder to indicate various usage restrictions (e.g., Al usage restrictions) for a video. In some examples, the various usage restrictions may include one or more Al usage restrictions for either or both usage as input for Al training and / or as groundtruth for Al training. Additionally, or alternatively, the various usage restrictions may include one or more Al usage restrictions for post-processed video, such as video that has been processed by a neural-network post-processing filter (NNPF) or a processing chain controlledby SEI messages. Additionally, or alternatively, the various usage restrictions may include one or more Al usage restrictions related to auxiliary outputs resulting by neural-network post- filtering, in which the auxiliary outputs may comprise, for example, semantic information about the content of pictures. Additionally, or alternatively, the various usage restrictions may include one or more Al usage restrictions depending on information about content of pictures. Additionally, or alternatively, the various usage restrictions may include one or more Al usage restrictions depending on picture quality. Additionally, or alternatively, the various usage restrictions may include one or more Al usage restrictions specific for spatial regions.

[0178] In some examples of the framework, the encoder 30 may generate first information indicative of a video (e.g., may generate the video itself or a signal from which the video is derived) and second information indicative of at least one rule for usage of the video (e.g., information indicative of how the video may be used). In some such examples, the encoder 30 may output (e.g., to the decoder 32 or memory 34), at least one signal comprising a representation of the first information and the second information. The representation may include the first information and second information or an encoded version of the first information and second information.

[0179] Additionally, or alternatively, in some examples of the framework, the decoder 32 may receive (e.g., from the encoder 30) or otherwise obtain (e.g., from the memory 34) at least one signal comprising a representation of first information indicative of a video and second information indicative of at least one rule for usage of the video. Additionally, in some examples, the decoder 32 may decode the at least one signal to obtain the first information and the second information.

[0180] FIG. 2 illustrates an example block diagram of an apparatus 40 to which one or more examples disclosed herein may be applied. The apparatus 40 comprises, for example, at least one processor 42 and at least one memory 44 storing instructions 46 that, when executed by the at least one processor, cause the apparatus 40 at least to perform the method or methods as disclosed herein, and any of the embodiments thereof. In an example, the at least one memory and the instructions (e.g. a computer program code, software), are configured, with the at least one processor, to cause the apparatus 40 to perform the method or methods as disclosed herein, and any of the embodiments thereof.

[0181] A processor 42 may comprise circuitry, or be constituted as circuitry or circuitries, the circuitry or circuitries being configured to perform phases of methods in accordance with example embodiments described herein. As used in this application, the term “circuitry” mayrefer to one or more or all of the following: (a) hardware-only circuit implementations, such as implementations in only analog and / or digital circuitry, and (b) combinations of hardware circuits and software, such as, as applicable: (i) a combination of analog and / or digital hardware circuit(s) with software / firmware and (ii) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a user equipment, to perform various functions) and (c) hardware circuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation. This definition of circuitry applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.

[0182] The memory 44 may be implemented using any suitable data storage technology. The memory may comprise a database for storing data. The memory 44 may be at least in part external to apparatus 40 but accessible to apparatus 40.

[0183] The instructions 46 may be comprised in a computer readable medium or a non-transitory computer readable medium. A term non-transitory, as used herein, is a limitation of the medium itself (e.g., tangible, not a signal) as opposed to a limitation on data storage persistency (e.g., random access memory, RAM, vs. read only memory, ROM).

[0184] For example, the apparatus 40 is embodied by an encoder 30 and / or a decoder 32 as shown in Figure 3. The apparatus 40 as embodied by an encoder 30 and / or a decoder 32, e.g., as a chipset configured to control the encoder and / or decoder, respectively, may be caused or configured to perform at least the methods of Figures 4 and 5, respectively.

[0185] The apparatus 40 comprises a radio interface 49. The radio interface 49 may provide the apparatus 40 with communication capabilities. The radio interface 49 may comprise a receiver configured to receive information in accordance with at least one cellular or non-cellular standard. The radio interface 49 may comprise a transmitter configured to transmit information in accordance with at least one cellular or non-cellular standard. The receiver may comprise more than one receiver. The transmitter may comprise more than one transmitter. The radio interface 49 may comprise a transceiver configured to receive andtransmit information in accordance with at least one cellular or non-cellular standard. The transceiver may comprise more than one transceiver.

[0186] The apparatus 40 may comprise a user interface 48 comprising, for example, at least one of a keypad, a microphone, a touch display, a display, a speaker, etc. The user interface 48 may be used to control the apparatus by the user. The user interface 48 may be external to the apparatus 10. For example, the apparatus 40 may be connected to another device, such as a computer, either via wireless or wired connection, and the apparatus 40 is controlled by the user via the computer.

[0187] In some examples, the apparatus 40 may be an encoder. In some such examples, in accordance with the framework for usage restrictions of videos, described herein, the apparatus 40 may generate first information indicative of a video (e.g., may generate the video itself or a signal from which the video is derived) and second information indicative of at least one rule for usage of the video (e.g., information indicative of how the video may be used). Additionally, the apparatus 40 may output, at least one signal comprising a representation of the first information and the second information. The representation may include the first information and second information or an encoded version of the first information and second information.

[0188] In some examples, the apparatus 40 may be a decoder. In some such examples, in accordance with the framework for usage restrictions of videos, described herein, the apparatus 40 may receive at least one signal comprising a representation of first information indicative of a video and second information indicative of at least one rule for usage of the video. Additionally, in some examples, the apparatus 40 may decode the at least one signal to obtain the first information and the second information.

[0189] FIG. 3 illustrates an example block diagram of a communication system to which one or more examples disclosed herein may be applied. As shown in FIG. 3, a data source 50 provides a source signal to an encoder in an analog, uncompressed digital, or compressed digital format, or any combination of these formats. In some examples, the data source includes one or more video capturing devices. An encoder 51 encodes the source signal into a bitstream (e.g., a coded media bitstream). In some examples, the encoder 51 may be capable of encoding more than one media type, such as audio and video, or more than one encoder 51 may be used to code different media types of the source signal. The encoder 51 may also receive synthetically produced input, such as graphics and text, or it may be capable of producing coded bitstreams of synthetic media. In some examples described herein, processing of one bitstream of one media type may be considered to simplify the description.

[0190] In various embodiments, the encoder 51 may encode a video or a signal from the video is derived into a bitstream. Additionally, the encoder 51 may encode (in the same bitstream or another bit stream) information indicative of how the video may be used. In some examples, the bitstream is transferred to a storage 52. The storage 52 may comprise any type of mass memory to store the bitstream. The format of the bitstream in the storage 52 may be an elementary self-contained bitstream format, or one or more bitstreams may be encapsulated into a container file. The bitstream may then be transferred to the sender 53 (e.g., a server). In some other examples, the communication system may operate “live” and, as such, may omit storage. That is, in some examples, the encoder 51 may directly transfer (e.g., signal) the bitstream to the sender 53. The sender 53 may transmit the bitstream. The format used in the transmission of the bitstream may be an elementary self-contained bitstream format, a packet stream format, or one or more bitstreams may be encapsulated into a container file. The encoder 51, the storage 52, and / or the sender 53 may reside in the same physical device or they may be included in separate devices. The encoder 51 and sender 53 may operate with live real-time content, in which case the bitstream may not be stored permanently, but rather buffered for relatively short periods of time in the encoder 51 and / or in the sender 53 to smooth out variations in processing delay, transfer delay, and / or coded media bitrate.

[0191] The sender 53 may send the bitstream using a communication protocol stack. The stack may include but is not limited to Real-Time Transport Protocol (RTP), User Datagram Protocol (UDP), and Internet Protocol (IP). When the communication protocol stack is packet-oriented, the sender 53 encapsulates the bitstream into packets. For example, when RTP is used, the sender 53 encapsulates the coded media bitstream into RTP packets according to an RTP payload format. In some examples, each media type has a dedicated RTP payload format.

[0192] If the media content is encapsulated in a container file for the storage 52 or for inputting the data to the sender 53, the sender 53 may comprise or be operationally attached to a "sending file parser" (not shown in the figure). In particular, if the container file is not transmitted as such but at least one of the contained coded media bitstream is encapsulated for transport over a communication protocol, a sending file parser locates appropriate parts of the coded media bitstream to be conveyed over the communication protocol. The sending file parser may also help in creating the correct format for the communication protocol, such as packet headers and pay loads. The multimedia container file may contain encapsulation instructions, such as hint tracks in the ISO Base Media File Format, for encapsulation of theat least one of the contained media bitstream on the communication protocol. The sender may follow one or more hint tracks in encapsulating a packet stream or it may form packets on the basis of media tracks and samples therein. Furthermore, the sender may use the information in the file format on switch points and redundant pictures in responding to view switching requests by receivers.

[0193] The sender 53 may or may not be connected to a gateway 54 through a communication network, sometimes also referred to as the transmission channel. The gateway 54 may perform different types of functions, such as translation of a packet stream according to one communication protocol stack to another communication protocol stack, merging and forking of data streams, and manipulation of data stream according to the downlink and / or receiver capabilities, such as controlling the bit rate of the forwarded stream according to prevailing downlink network conditions. Examples of gateways 54 include Multipoint Conference Units (MCUs), gateways between circuit-switched and packet-switched video telephony, Push-to-talk over Cellular (PoC) servers, IP encapsulators in digital video broadcasting-handheld (DVB-H) systems, and set-top boxes that forward broadcast transmissions locally to home wireless networks. When RTP is used, the gateway 54 is called an RTP mixer or an RTP translator and typically acts as an endpoint of an RTP connection. It should be noted that the system may include multiple gateways.

[0194] In some examples, the gateway 54 may receive a bitstream or a packet stream including both primary and respective redundant pictures. The system may include one or more receivers 55, which may be capable of receiving, de-modulating, and de-capsulating the transmitted signal into a bitstream. The bitstream may then be transferred to a recording storage 56. The recording storage 56 may comprise a type of mass memory to store the coded media bitstream. The recording storage 56 may alternatively or additively comprise computation memory, such as random access memory. The format of the coded media bitstream in the recording storage 56 may be an elementary self-contained bitstream format, or one or more bitstreams may be encapsulated into a container file. In some systems, the receiver 55 may not decapsulate the transmitted signal, but the encapsulated packet stream may be stored in the recording storage 56. If there are multiple coded media bitstreams, such as an audio stream and a video stream, associated with each other, a container file is typically used and the receiver 55 comprises or is attached to a container file generator producing a container file from input streams. Some systems operate “live,” (e.g., omit the recording storage 56) and transfer the bitstream from the receiver 55 directly to the decoder 57.

[0195] The bitstream may be transferred from the recording storage 56 to the decoder 57. If there are multiple bitstreams, such as an audio stream and a video stream, associated with each other and encapsulated into a container file, a file parser (not shown in the figure) may be used to decapsulate each coded media bitstream from the container file. The recording storage 56 or the decoder 57 may comprise the file parser, or the file parser is attached to either recording storage 56 or the decoder 57. The decoder 57 may decode the bitstream to obtain the video (or the signal from the video is derived) and the information indicative of how the video may be used. In some examples, a Tenderer 58 may reproduce the video (e.g., via a display). The receiver 55, recording storage 56, decoder 57, and Tenderer 58 may reside in the same physical device or they may be included in separate devices. In some examples, transmission via one or more gateways 54 is omitted, and a file is transferred by other means from the encoder 51 to the decoder 57.

[0196] As illustrated in the example of FIG. 3, the encoder 51 signals, in or along the bitstream, and the decoder 57 decodes, from or along the bitstream, information indicative of how a video (e.g., a decoded video or a decoded and then post-processed video) may be used, where an encoded video from which the video is derived may be comprised in the bitstream. In other words, the encoder 51 may generate first information indicative of a video and second information indicative of at least one rule for usage of the video. Additionally, the encoder 51 may output at least one signal comprising a representation of the first information and the second information.

[0197] In some examples, the bitstream may comprise an encoded video from which the video is derived. For example, the encoder 51 may generate a bitstream comprising encoded data from which the video is derived. In one example, the encoded video is decoded to obtain a decoded video that represents the video. In another example, the encoded video is decoded to obtain a decoded video, and then the decoded video is processed or post-processed by one or more processors or post-processors to obtain a processed video.

[0198] In some examples, the video is a processed video. For example, the video may be an output of a processor when an input to the processor is a decoded video. The processor may be a task or module, which may be implemented in software or in hardware. In some examples, the processor may be a neural network, such as a neural network post processing filter (also referred to as neural network post-filter (NNPF)). In some other examples, the video may be an output of a processing chain. In some such examples, the processing chain may comprise two or more processors, in which the two or more processors may comprise zero, one, or multiple neural network based processors, and in which an input to theprocessing chain may comprise a decoded video. In other words, the video may comprise an output of a processor (e.g., a task or module, such as a neural network), where the output is based on a decoded video input into the processor. Alternatively, the video may comprise an output of a processing chain comprising at least two processors (e.g., at least two tasks or modules, such as neural networks), where the output is based at least in part on an input to the processing chain, and where the input comprises a decoded video. The terms “processing chain” and “post-processing chain” may be used interchangeably.

[0199] In some examples, the information indicative of how a video (e.g., a decoded video or a decoded and then post-processed video) may be used may be generic to multiple uses (e.g., any use) of the video or may be specific to one or more usage types of the video, which may include, but are not necessarily limited to, one or more of the following: artificial intelligence (such as training of neural networks), machine learning (which may comprise machine learning with and without artificial intelligence means), machine analysis (e.g., computer vision tasks, such as face detection, object detection, object tracking, anomaly detection, action recognition, etc.), input to a neural network, human viewing or displaying, derivative works or subsequent workflows (such as video editing), usage in one or more types of applications (such as social networking), content authentication (e.g., the video may be indicated to be only used when content authentication means are present), digital rights management (e.g., the video may be indicated to be only used when digital rights management means are present).

[0200] In at least one example embodiment, the information indicative of how the video may be used comprises information indicative of one or more usage types, which may, for example, be among those listed above.

[0201] In at least one example embodiment, the information indicative of how the video may be used, may include information indicative of how the video may be used with artificial intelligence. For example, the at least one rule may comprise a rule for usage of the video in accordance with at least one operation to train at least one artificial intelligence model.Additionally, or alternatively, the at least one rule may be for usage of the video to determine a ground-truth for the at least one operation to train the at least one artificial intelligence model. In some examples, the at least one rule is for usage of the video to determine an input for the at least one operation to train the at least one artificial intelligence model. A groundtruth for the at least one operation to train the at least one artificial intelligence model may be used for computing a training or validation loss during or after a training process of the at least one artificial intelligence model. In one example, during a training process of a neuralnetwork, an input is provided to the neural network, an inference of the neural network is performed to obtain an output of the neural network, a training loss, such as a mean-squared error (MSE), is computed based on the output of the neural network and a ground-truth that is associated with the input that is provided to the neural network. The training loss is then used to compute updates to one or more parameters or weights of the neural network.

[0202] In some examples, the information indicative of how a video may be used may be comprised in an SEI message, such as an SEI message that is specified in a VSEI standard (or another type of standard). In other words, in some examples, the at least one signal may comprise an SEI message including the second information.

[0203] In some examples, the information indicative of how a video may be used may be comprised in an NNPFC SEI message. Additionally, or alternatively, in at least one example embodiment, the information indicative of how a video may be used may be comprised in an AUR SEI message. In other words, in some examples, the SEI message comprises an AUR SEI message and / or an NNPF SEI message. Additionally, or alternatively, the SEI message may comprise a text description information SEI message.

[0204] It is to be understood that a machine vision model (or Al model, or machine learning model, or machine vision process, and the like) may comprise any type of task performed based on an input picture, such as a discriminative task (e.g., classification, detection), a regression task (e.g., filtering, processing), a generative task (e.g., generative new content, extrapolation, editing).

[0205] In some examples, the at least one rule may distinguish whether a video is usable as input for training or as ground-truth for training. For example, in at least one example embodiment, the information indicative of how a video may be used may comprise an indication of whether the video is usable in a training process, where the video is used as an input to a machine learning model to be trained by means of the training process.

[0206] In some examples, the information indicative of how a video may be used may comprise an indication of whether the video is usable in a training process, where the video is used as ground-truth or to derive ground-truth, where the ground-truth is used to compute a loss or objective for training a machine learning model by means of the training process. For example, the SEI message may comprise an AUR SEI message where a value of an indicator included in the AUR SEI message indicates the at least one rule by indicating, to a receiver of the at least one signal (e.g., the decoder 57), whether to refrain from using the video as input to an artificial intelligence model for training of the artificial intelligence model or whether torefrain from using the video to determine a ground-truth for training an artificial intelligence model.

[0207] In some examples in which the information indicative of how a video may be used is comprised in an AUR SEI message, the AUR SEI message comprises an indicator aur_restriction[i], where possible values and interpretations of each possible value are specified in the following Table 6:

[0208] In some such examples, the video may be used for a training process but not as an input to an Al model being trained by the training process, a value of aur_restriction[i] may be set to 3. In some examples, when the video may be used for a training process but not as ground-truth for computing a training loss to be used in the training process, a value of aur_restriction[i] may be set to 4. Additionally, or alternatively, when the video may be used for a training process but not as an input to an Al model being trained by the training process and not as ground-truth for computing a training loss to be used in the training process, aur_restriction may comprise at least two values, where a first value (e.g., aur_restriction[i]) may be set to 3 and a second value (e.g., aur_restriction[j]) may be set to 4.

[0209] In some examples, when the video may be used for a training process but not as an input to an Al model being trained by the training process and not as ground-truth for computing a training loss to be used in the training process, the video may be derived or inferred that the video may be used for a validation process that is part of the training process, such as for computing a validation loss, where the validation process or the validation loss are used for determining one or more of the following: model selection (e.g., determining which of a set of models is best according to one or more criteria), or hyper-parameter tuning (e.g., determining a value of a learning rate, determining an architecture of the model).

[0210] In some examples, the video may be of relatively low quality and, as such, the encoder 51 may indicate to a receiver that the video should not be used as ground-truth, or for deriving ground-truth, thus a value of aur_restriction[i] may be set to 4. In some examples where an encoder indicates to a receiver that the video should not be used as ground-truth orto derive ground-truth for training an Al model, a ground-truth may be obtained by other means, such as may be separately indicated by the encoder, or may be provided to the receiver by external means.

[0211] In an example embodiment, the information indicative of how the video may be used may include or may be associated with information about a reason or motivation or explanation for the information indicative of how the video may be used. In one example, the information indicative of how the video may be used may include a reason for at least one rule. In another example, the information indicative of how the video may be used may include a reason for at least one restriction on how the video may be used. In yet another example, a reason may comprise (but may not be limited to) one or more of the following: (i) the video comprises a too low visual quality; (ii) the video comprises one or more faces; (iii) the video comprises personal identification data; (iv) the video comprises sensitive data; (v) the video was generated by means of a generative Al model; (vi) the video was processed by means of a generative Al model; (vii) the video was processed by means of an Al model; (viii) the video comprises age-inappropriate content; (ix) the video is protected by copyrights; (x) the video was tagged by a user (e.g., a content creator) as indicated in the information indicative of how the video may be used. In yet another example, the information indicative of how the video may be used may be associated with a Text Description Information SEI message that indicates a reason for the information indicative of how the video may be used. In yet another example, the information indicative of how the video may be used may include or may be associated with one or more reasons for respective one or more rules or restrictions comprised in the information indicative of how the video may be used.

[0212] In some examples in which the at least one rule is indicated via an AUR SEI message, an AUR SEI message type associated with the AUR SEI message may be associated with an SEI processing order (SPO) SEI message to indicate that the AUR SEI message indicates at least one of the following: characteristics or rules or restrictions of a processing chain indicated in the SPO SEI message, characteristics or rules or restrictions of pictures output by a processing chain indicated in the SPO SEI message, or characteristics or rules or restrictions of a result of a processing chain indicated in the SPO SEI message. In some examples, the AUR SEI message may be included in a processing order nesting (PON) SEI message and is associated with a processing order value. In some such examples, the processing order value associated with the AUR SEI message indicates that the AUR SEI message may be descriptive of an output of a NNPF for post-process filtering associated with the video.

[0213] In some examples, the SEI message comprises a neural network post filter (NNPF) SEI message indicating at least one of characteristics or activation of a NNPF for post-process filtering associated with the video. In some such examples, the NNPF SEI message includes at least one flag or at least one indicator that indicates the at least one rule. In some such examples, the NNPF SEI message comprises a NNPFC SEI message indicating characteristics of the NNPF or a NNPFA SEI message indicating activation of the NNPF for post-process filtering associated with the video.

[0214] In some examples, the NNPF SEI message comprises an indication of whether an output of the NNPF comprises information pertaining to one or more pictures input to or output by the NNPF. For example, an NNPFC SEI message that specifies a NNPF comprises a flag, such as nnpfc_input_for_training_flag, that when it is equal to 1 , the flag indicates that output pictures of the NNPF may be used as input to a machine learning model during a training process, and when it is equal to 0 it indicates that output pictures of the NNPF should not be used as input to a machine learning model during a training process. In another example, an NNPFC SEI message that specifies a NNPF comprises a flag, such as nnpfc_use_for_ai_training_flag, that when it is equal to 1 , the flag indicates that output pictures of the NNPF may be used for training an Al model, and when it is equal to 0 the flag indicates that output pictures of the NNPF may not be used for training an Al model. In yet another example, an NNPFC SEI message that specifies a NNPF comprises a flag, such as nnpfc_use_for_ai_inference_flag, that when it is equal to 1 , the flag indicates that output pictures of the NNPF may be used as input to an Al model for an inference of the Al model, and when it is equal to 0, the flag indicates that output pictures of the NNPF may not be used as input to an Al model for an inference of the Al model. In yet another example, an NNPFC SEI message that specifies a NNPF comprises a flag, such as nnpfc_use_for_generative_ai_flag, that when it is equal to 1 , the flag indicates that output pictures of the NNPF may be used as input to a generative Al model, and when it is equal to 0, the flag indicates that output pictures of the NNPF may not be used as input to a generative Al model. In yet another example, an NNPFC SEI message that specifies a NNPF comprises a flag, such as nnpfc_use_in_ai_applications_flag, that when it is equal to 1 , the flag indicates that output pictures of the NNPF may be used in any Al application, and when it is equal to 0, the flag indicates that output pictures of the NNPF may not be used in any Al application. In yet another example, when one of the flags in some of the above examples is not present, its value may be inferred to be equal to a predetermined value, such as 0 or 1. In yet another example, for the above examples, instead of using a flag (thus, a binary indication), anindicator with more than two possible values may be used, such as nnpfc_use_in_ai_applications_idc that, when it is equal to 0 it indicates that output pictures of the NNPF may or may not be used for any Al application (e.g., it is not known, thus a receiver may choose whether to use the video for any Al application), when it is equal to 1 it indicates that output pictures of the NNPF may be used in any Al application, and when it is equal to 2 it indicates that output pictures of the NNPF may not be used in any Al application.

[0215] In some examples, a NNPFA SEI message that activates a NNPF comprises a flag, such as nnpfa_input_for_training_flag, when the flag is equal to 1 , the flag indicates that output pictures of the NNPF may be used as input to a machine learning model during a training process, and when the flag is equal to 0, the flag indicates that output pictures of the NNPF should not be used as input to a machine learning model during a training process. The semantics of the flag may apply to NNPF output pictures that result when applying the NNPF to the pictures where the NNPFA SEI message persists.

[0216] In some examples, a NNPFC SEI message that specifies a NNPF comprises a flag, such as nnpfc_gt_for_training_flag, that when the flag is equal to 1 , the flag indicates that output pictures of the NNPF may be used as ground-truth or to derive ground-truth for training a machine learning, and when the flag is equal to 0, the flag indicates that the output pictures of the NNPF should not be used as ground-truth or to derive ground-truth for training a machine learning model.

[0217] In some examples, a NNPFA SEI message that activates an NNPF comprises a flag, such as nnpfa_gt_for_training_flag, that when it is equal to 1 it indicates that output pictures of the NNPF may be used as ground-truth or to derive ground-truth for training a machine learning, and when it is equal to 0 it indicates that the output pictures of the NNPF should not be used as ground-truth or to derive ground-truth for training a machine learning model. The semantics of the flag may apply to the NNPF output pictures that are resulting when applying the NNPF to the pictures where the NNPFA SEI message persists.

[0218] In some examples, a NNPFC SEI message that specifies a NNPF comprises an indicator, such as nnpfc_input_gt_for_training_idc, or a NNPFA SEI message that activates a NNPF comprises an indicator, such as nnpfa_input_gt_for_training_idc, that, when the flag is equal to 0, the flag indicates that output pictures of the NNPF should not be used as input to a machine learning model during a training process and should not be used as ground-truth or to derive ground-truth for training a machine learning model. In some such examples, when the flag is equal to 1, the flag indicates that output pictures of the NNPF may be used as inputto a machine learning model during a training process. In some such examples, when the flag is equal to 2, the flag indicates that output pictures of the NNPF may be used as ground-truth or to derive ground-truth for training a machine learning model. In some such examples, when the flag is equal to 3, the flag indicates that output pictures of the NNPF may be used as input to a machine learning model during a training process and as ground-truth or to derive ground-truth for training the machine learning model. In some examples, the semantics of the indicator in a NNPFA SEI message may apply to the NNPF output pictures that are resulting when applying the NNPF to the pictures where the NNPFA SEI message persists.

[0219] In other words, in some examples, the NNPF SEI message includes the at least one flag, in which the at least one flag indicates the at least one rule by indicating, to a receiver of the at least one signal, at least one of the following: to use one or more pictures output by the NNPF as input to an artificial intelligence model for training of the artificial intelligence model, to refrain from using one or more pictures output by the NNPF as input to an artificial intelligence model for training of the artificial intelligence model, to use one or more pictures output by the NNPF to determine a ground-truth for training an artificial intelligence model, or to refrain from using one or more pictures output by the NNPF to determine a ground-truth for training an artificial intelligence model.

[0220] In some examples, the at least one rule may indicate restrictions of output pictures. For example, an NNPF SEI message may be defined as an NNPFC SEI message that specifies a NNPF and / or an NNPFA SEI message that activates a NNPF. In some embodiments in which the at least one rule is indicated via an NNPF SEI message, the embodiments may be applied using either or both a NNPFC SEI message that specifies a NNPF and / or a NNPFA SEI message that activates a NNPF. In some embodiments in which the at least one rule is indicated via an NNPF SEI message that activates an NNPF, the embodiments may concern the NNPF output pictures that are resulting when applying the NNPF to the pictures where the NNPFA SEI message persists. In some examples, when at least one rule is indicated via both an NNPFC SEI message and an NNPFA SEI message, the indication in the NNPFA SEI message updates or overrides the indication in the NNPFC SEI message for the pictures where the NNPFA SEI message persists. In some examples, when at least one rule is indicated via both an AUR SEI message and an NNPFA SEI message, the indication in the NNPFA SEI message updates or overrides the indication in the AUR SEI message for the pictures where the NNPFA SEI message persists. In one example, an AUR SEI message indicates that the video can be used in any Al application, and an NNPFA SEI message indicates that the pictures where the NNPFA SEI message persists cannot be used inany Al application; the indication in the NNPFA SEI message has priority over the indication in the AUR SEI message for the pictures where the NNPFA SEI message persists, thus the pictures where the NNPFA SEI message cannot be used in any Al application.

[0221] In some examples, a NNPF SEI message may comprise an indication of whether pictures output by the NNPF, or data derived from pictures output by the NNPF, may be used as input to a machine vision process, such as an Al model.

[0222] In some examples, a NNPF SEI message may comprise an indication of whether pictures output by the NNPF, or data derived from pictures output by the NNPF, may be used for training a machine vision process, such as an Al model.

[0223] In some examples, an NNPF SEI message may comprise an indication of whether pictures output by the NNPF, or data derived from pictures output by the NNPF, may be used as input to a machine vision process, such as an Al model, during a training process of that machine vision process.

[0224] In some examples, an NNPF SEI message may comprise an indication of whether pictures output by the NNPF, or data derived from pictures output by the NNPF, may be used as ground-truth or to derive ground-truth for training a machine vision process, such as an Al model.

[0225] In some examples, an NNPF SEI message may comprise an indicator that indicates whether pictures output by the NNPF, or data derived from pictures output by the NNPF, may be used as input to a machine vision process, such as an Al model. In some examples, an NNPF SEI message may comprise an indicator that indicates whether pictures output by the NNPF, or data derived from pictures output by the NNPF, may be used for training a machine vision process, such as an Al model. In some examples, an NNPF SEI message may comprise an indicator that indicates whether pictures output by the NNPF, or data derived from pictures output by the NNPF, may be used as input to a machine vision process, such as an Al model, during a training process of that machine vision process. In some examples, an NNPF SEI message may comprise an indicator that indicates whether pictures output by the NNPF, or data derived from pictures output by the NNPF, may be used as ground-truth or to derive ground-truth for training a machine vision process, such as an Al model.

[0226] In other words, in some examples, the NNPF SEI message includes an indicator, in which a value of the indicator indicates the at least one rule by indicating, to a receiver of the at least one signal, at least one of the following: to refrain from using one or more pictures output by the NNPF both as input to an artificial intelligence model for training ofthe artificial intelligence model and to determine a ground-truth for training the artificial intelligence model, to use one or more pictures output by the NNPF as input to an artificial intelligence model for training of the artificial intelligence model, to use one or more pictures output by the NNPF to determine a ground-truth for training an artificial intelligence model, or to use one or more pictures output by the NNPF both as input to an artificial intelligence model for training of the artificial intelligence model and to determine a ground-truth for training the artificial intelligence model.

[0227] Additionally, or alternatively, in some examples, the NNPF SEI message includes an indicator, in which a value of the indicator indicates the at least one rule by indicating, to a receiver of the at least one signal, at least one of the following: to use one or more pictures output by the NNPF as input to a machine vision model, to refrain from using one or more pictures output by the NNPF as input to a machine vision model, to use one or more pictures output by the NNPF for training a machine vision model, to refrain from using one or more pictures output by the NNPF for training a machine vision model, to use one or more pictures output by the NNPF as input to a machine vision model for training of the machine vision model, to refrain from using one or more pictures output by the NNPF as input to a machine vision model for training of the machine vision model, to use one or more pictures output by the NNPF to determine a ground-truth for training a machine vision model, or to refrain from using one or more pictures output by the NNPF to determine a ground-truth for training a machine vision model.

[0228] In some examples, the information indicative of how a video may be used, such as information comprised in a NNPFC SEI message, may apply to a video that is output by a processing chain, where at least one of the processors comprised in the processing chain is associated with the information indicative of how a video may be used.

[0229] In some examples, the NNPF SEI message includes a plurality of indicators associated with a plurality of processors in a processing chain. In some examples, when two or more processors in a processing chain are associated with respective two or more indications of how pictures output by the respective two or more processors may be used, the more restrictive indication(s) may apply to pictures that are output by the processing chain. In some examples, a processing chain comprises two neural network post processing filters specified by respective two NNPFC SEI messages, where a first NNPFC SEI message specifying a first NNPF, or a first AUR SEI message associated to a first NNPFC SEI message specifying a first NNPF, indicates that pictures output by the first NNPF should not be used as an input to a machine learning model during a training process, and where asecond NNPFC SEI message specifying a second NNPF, or a second AUR SEI message associated to a second NNPFC SEI message specifying a second NNPF, indicates that pictures output by the second NNPF may not be used as ground-truth for training a machine learning model, it may be inferred that pictures output by the processing chain should not be used as an input to a machine learning model during a training process and may not be used as ground-truth for training a machine learning model.

[0230] In some examples, the NNPF is comprised in a processing chain (e.g., a postprocessing chain) indicated via an SPO SEI message. For example, an NNPF specified by a NNPFC SEI message and / or activated by a NNPFA SEI message may be comprised in a post-processing chain specified by a SEI processing order (SPO) SEI message, and the NNPFC SEI message and / or the NNPFA SEI message comprises an indication that pictures output by the NNPF should not be used for Al training. Thus, a picture that is output by the chain comprising the NNPF may not be used for Al training, if not otherwise indicated.

[0231] In some examples, the NNPF is associated with distortions of images for stylistic or artistic functions. For example, an NNPF specified by a NNPFC SEI message distorts images for stylistic or artistic purposes (e.g., for a style transfer purpose). The NNPFC SEI message or a NNPFA SEI message activating the NNPF may indicate that output pictures may not be used for training a machine vision model.

[0232] In some examples, the NNPF SEI message is indicative of one or more generative artificial intelligence characteristics associated with the NNPF, where the at least one rule is based on the one or more generative artificial intelligence characteristics. For example, an NNPFC SEI message that specifies an NNPF may comprise an indication of whether the NNPF has generative capabilities, or is a generative Al model, or the NNPFC may comprise an indication from which it can be inferred that the NNPF has generative capabilities, or is a generative Al model, and when it is indicated or inferred that the NNPF has generative capabilities, or is a generative Al model, output pictures of the NNPF should not be used for training Al models. The motivation may be that generative Al models may introduce new information or new content that may not be reliable and / or of sufficient quality for being used as input to an Al model being trained and / or as ground-truth for training an Al model.

[0233] In some examples, an NNPFC SEI message that specifies an NNPF may comprise information that may be used to infer that the NNPF has generative capabilities or is a generative model. In one example, when an NNPFC SEI message indicates that the NNPF takes as an input (e.g., as an auxiliary input) a prompt (such as a text prompt), it may be inferred that the NNPF is a generative model, and thus pictures output by the NNPF shouldnot be used in a training process, such as an input to a machine learning model or as groundtruth for training the machine learning model.

[0234] In some examples, the NNPF SEI message indicates that the NNPF comprises a capability to output a plurality of pictures for each activation or inference of the NNPF, and wherein the at least one rule pertains to the plurality of pictures. For example, an NNPFC SEI message that specifies an NNPF may indicate that the NNPF may output two or more pictures for each activation or inference of the NNPF, where at least one of the two or more pictures may be used in a training process, such as an input to a machine learning model or as ground-truth for training a machine learning model, and where at least another one of the two or more pictures may be used for display.

[0235] In some examples, where an NNPF comprises a capability to output two or more pictures for each activation or inference of the NNPF, one or more of the two or more pictures may be associated with at least one rule.

[0236] In some examples, where an NNPF comprises a capability to output two or more pictures for each activation or inference of the NNPF, at least two of the two or more pictures may be associated with different rules. In one example, where an NNPF outputs two pictures for each activation, one of the two pictures is indicated to be usable in any Al application and another one of the two pictures is indicated not to be usable for Al training. In another example, where an NNPF outputs two pictures for each activation, one of the two pictures is indicated to be usable in any Al application and another one of the two pictures is indicated not to be usable in any Al application.

[0237] In some examples, an NNPFC SEI message that specifies an NNPF may indicate that the NNPF may output two or more pictures for each activation or inference of the NNPF, where at least one of the two or more pictures may be used as an input to an Al model (e.g., for a machine vision task) and where at least another one of the two or more pictures may be used for display.

[0238] In at least one example embodiment, the at least one rule indicates one or more Al usage restrictions for a processing chain. For example, in at least one example embodiment, an AUR SEI message type may be included in or associated with an SPO SEI message to indicate that the associated AUR SEI message(s) describe characteristics or restrictions of a post-processing chain or of pictures output by a post-processing chain that is specified in the SEI processing order SEI message.

[0239] In some examples, an encoder includes an AUR SEI message type in an SEI processing order (SPO) SEI message to indicate that a result of the processing chain definedin the SPO SEI message is restricted for Al usage as indicated by the AUR SEI message(s) associated with the processing chain.

[0240] In some examples, an encoder specifies a processing order value (po_processing_order[ i ]) for an AUR SEI message type in an SPO SEI message to be the next greater value or equal to the processing order value of a process included in a processing chain defined by the SPO SEI message, which indicates that the AUR SEI message describes the Al usage restrictions for the output of the process within the processing chain.

[0241] In some examples, the decoder 57 decodes an AUR SEI message type from an SEI processing order (SPO) SEI message defining a processing chain, decodes AUR SEI message(s) associated with the processing chain, and determines that a result of the processing chain is restricted for Al usage as indicated by the decoded AUR SEI message(s).

[0242] In some examples, an AUR SEI message is included in a PON SEI message and associated with a particular processing order value (pon_processing_order[ i ]). In some examples, the at least one rule (e.g., one or more restrictions) specified in the AUR SEI message apply to the processing chain associated with the PON SEI message at the processing stage identified by the processing order value. For example, if the AUR SEI message is associated with the last processing stage of the processing chain, it describes the Al usage restrictions of the output of the processing chain.

[0243] In some examples, an encoder includes an AUR SEI message in a PON SEI message, when the signaled usage restrictions apply to post-processed video and do not apply to decoded video.

[0244] In some examples, an encoder includes an AUR SEI message type in a SPO SEI message but refrains from including the AUR SEI message in a PON SEI message, when the signaled usage restrictions apply to post-processed video and to decoded video.

[0245] In some examples, an encoder indicates a first set of usage restrictions in a first AUR SEI message that is not included in a PON SEI message and applies to decoded video and a second set of usage restrictions in a second AUR SEI message that is included in a PON SEI message and applies to post-processed video.

[0246] In some examples, an AUR SEI message and an NNPF SEI message are included in a PON SEI message and the AUR and NNPF SEI messages have such processing order values that indicates the AUR SEI message to describe the output of the NNPF. In some such embodiments, the restrictions specified in the AUR SEI message may apply to output pictures of the NNPF.

[0247] In some examples, when two or more rules are comprised in the at least one rule, the two or more rules may be associated with respective two or more ranks or order values that indicate a level of restriction. For example, a first rule may be associated with a higher level of restriction than a second rule, thus the first rule may be more restrictive than the second rule. The two or more ranks or order values may be specified in a standard specification, such as a version of the VSEI standard, or may be indicated in an SEI message such as in an AUR SEI message. When a processing chain includes two or more processors, and at least two of the two or more processors are associated with respective two rules and respective two order values, an output of the processing chain (e.g., a picture) is inferred to be associated with the more restrictive rule according to the two order values.

[0248] In at least one example embodiment, the at least one rule pertains to auxiliary output of NNPF usable for training. For example, in at least one example embodiment, an NNPF SEI message may comprise an indication of whether an output of the NNPF may comprise information about the picture input to or output by the NNPF, such as semantic information about the content of the input / output picture (e.g., detected faces), or information about the capturing process of the input / output picture (e.g., camera parameters). In one additional embodiment, the output of the NNPF that may comprise information about the picture input to or output by the NNPF may be an auxiliary or additional output with respect to an output picture. In other words, in some examples, the information indicative of the at least one rule comprises content information pertaining to respective content of one or more pictures associated with the video, where the content information comprises semantic information or image quality information.

[0249] In some examples, an NNPF SEI message may comprise an indication of whether an output of the NNPF that comprises information about the picture input to or output by the NNPF may be used as ground-truth for training an Al model.

[0250] In some examples, semantic information about the content of pictures may be indicated based on a predetermined vocabulary. In one example, the predetermined vocabulary may be based on the Dublin Core metadata vocabulary. In another example, the predetermined vocabulary may be based on a metadata registry that is compliant with a standard, such as the ISO / IEC 11179 standard. In yet another example, the predetermined vocabulary may be based on a schema (e.g., a schema specified in Schema.org). In yet another example, the predetermined vocabulary may be based on Wikidata.

[0251] In some examples, information about a picture, such as semantic information about the content of the input / output picture (e.g., detected faces), or information about thecapturing process of the input / output picture (e.g., camera parameters), may be comprised in an SEI message, such as an essential content description SEI message.

[0252] In some examples, the vocabulary or schema used for the semantic information is indicated, for example, in the same SEI message that also comprises the semantic information. In some examples, the vocabulary or schema used for the semantic information may be indicated through an index to a list of vocabularies or schemas, such as the ones in a table below. In some other examples, the vocabulary or schema used for the semantic information may be indicated by a URI, e.g. https: / / www.dublincore.org / indicates the Dublin Core metadata vocabulary and https: / / www.wikidata.org indicates Wikidata.

[0253] In some examples, a text description information (TDI) SEI message may comprise a syntax element, such as txt_descr_schema_uri[i], that identifies a schema that is used in this TDI SEI message and to which the text string comprised in this TDI SEI message (e.g., txt_descr_string[i]) refers. In some examples, values and interpretations of txt_descr_schema_uri[i] may be specified in a table, such as the following Table 7.

[0254] In some examples, the information indicative of how a video may be used, or the information about a picture such as semantic information about the content of a picture, may be comprised in a TDI SEI message. In one example, the syntax element txt_descr_purpose of the TDI SEI message may take a value that indicates that a purpose of the TDI SEI message comprises a semantic description of a picture associated with the TDI SEI message, or a content description of a picture associated with the TDI SEI message.

[0255] In at least one example embodiment, the at least one rule pertains to the content of pictures. For example, the at least one signal (e.g., the second information indicating the at least one rule or other information) may include information about content of pictures. In some examples, the second information may comprise content information pertaining to respective content of one or more pictures associated with the video. In some such examples, the content information comprises semantic information or image quality information. In other words, in at least one example embodiment, the information indicative of how a videomay be used may comprise information about the content of pictures, such as semantic information or quality of the pictures. In some examples, the information about the content of pictures may comprise one or more of the following: whether a picture comprises a particular type of content (e.g., persons, faces, etc.); where, within the picture, a particular type of content is present (e.g., where persons or faces are located within the picture); whether a picture is too dark; or whether a picture is of too low quality.

[0256] In some examples, the at least one rule pertains to one or more operations performed using the one or more pictures associated with the video. For example, in at least one example embodiment, the information about the content of pictures may be used for determining whether and / or how to perform some operations or processes on those pictures, such as training an Al model by using those pictures or performing inference of an Al model on those pictures.

[0257] In at least on embodiment, the at least one rule is based on whether the one or more pictures include faces. For example, the at least one rule may indicate that pictures that are indicated to contain faces may be used for training a face detection model. In another example, the at least one rule may indicate that pictures that are indicated to contain faces may not be used for training an Al model. In some examples, the at least one rule may indicate that pictures that are indicated not to contain faces may not be further analyzed by an Al model that performs one or more of the following: face detection, face localization, face expression analysis, face tracking, activity recognition, or anomaly detection.

[0258] In some examples, the at least one rule may indicate that pictures that are indicated to be too dark may not be used for training an Al model or as input to an Al model inference. In some examples, a receiver may determine that the video does not include any faces, thus it would not use the video for training a face detector.

[0259] In some examples, semantic information about the content of a picture, such as object classes present in a picture, may be comprised in a AUR SEI message that is associated with the picture. In some such examples, when semantic information about the content of a particular picture is comprised in a AUR SEI message associated with the particular picture, and when aur_restriction[i] is equal to 1 (do not use for training), the particular picture may not be used for training an Al model that should analyze one or more aspects related to the indicated semantic information.

[0260] In some examples, semantic information about the content of a picture, such as object classes present in a picture, may be comprised in a AUR SEI message that is associated with the picture. In some such examples, when semantic information about thecontent of a particular picture is comprised in a AUR SEI message associated with the particular picture, and when aur_restriction[i] is equal to 1 (do not use for training), the particular picture should not be used for training an Al model except when the Al model may analyze one or more aspects related to the indicated semantic information.

[0261] In some examples, semantic information about the content of a picture, such as object classes present in a picture, may be comprised in a AUR SEI message that is associated with the picture. In some such examples, when semantic information about the content of a particular picture is comprised in a AUR SEI message associated with the particular picture, and when aur_restriction[i] is equal to 0 (do not use for any Al application), the particular picture should not be used for any Al application (e.g., inference, training) that may analyze one or more aspects related to the indicated semantic information.

[0262] In some examples, semantic information about the content of a picture, such as object classes present in a picture, may be comprised in a AUR SEI message that is associated with the picture. In some such examples, when semantic information about the content of a particular picture is comprised in a AUR SEI message associated with the particular picture, and when aur_restriction[i] is equal to 0 (do not use for any Al application), the particular picture may not be used for any Al application (e.g., inference, training) except when the Al application analyzes one or more aspects related to the indicated semantic information.

[0263] In some examples, when semantic information about the content of a picture, or about a machine vision task, is indicated or comprised in a TDI SEI message, a AUR SEI message may be associated with a TDI SEI message so that any Al restrictions indicated in the AUR SEI message are applied (or, alternatively, are not applied) to pictures comprising the semantic information indicated in the TDI SEI message. In one example, an AUR SEI message may comprise an identifier of a TDI SEI message, such as aur_txt_sei_id that points to a value of txt_descr_id.

[0264] In some examples, the information indicative of how a video may be used may comprise an indication of one or more tasks or machine vision types or Al model types that may be trained by using pictures comprised in the video as training data (e.g., as input to an Al model, and / or as ground-truth for computing a training loss). In other words, in some examples, the information indicative of the at least one rule (or other information included in the one or more signals) may be indicative of one or more machine learning tasks, one or more machine vision types, or one or more artificial intelligence model types. Examples of tasks or machine vision types may include (but may not be limited to) the following: imageclassification, face detection, person detection, person tracking, anomaly detection, object detection, semantic segmentation, or instance segmentation. In some examples, the information indicative of how a video may be used may comprise an indication that pictures in the video may be used for training a face detection model.

[0265] In some examples, the information indicative of how a video may be used may comprise an indication of one or more tasks or machine vision types or Al model types that may take as input pictures comprised in the video. In one example, the information indicative of how a video may be used may comprise an indication that pictures in the video may be used as input to a face analysis model, such as a face expression recognition model.

[0266] In some examples, the at least one signal comprises third information indicative of the one or more machine vision types or the one or more artificial intelligence model types, in which the one or more machine vision types or the one or more artificial intelligence model types are based on a predetermined vocabulary. For example, in at least one example embodiment, information about a type of machine vision (or a type of Al model) may be indicated based on a predetermined vocabulary. In one example, the predetermined vocabulary may be based on the Dublin Core metadata vocabulary (https: / / www.dublincore.org / ). In another example, the predetermined vocabulary may be based on a metadata registry that is compliant with a standard, such as the ISO / IEC 11179 standard. In some examples, the predetermined vocabulary may be based on a schema, such as the one specified in Schema.org. In some other examples, the predetermined vocabulary may be based on Wikidata (https: / / www.wikidata.org).

[0267] In some examples, the syntax element txt_descr_purpose of the TDI SEI message may take a value that indicates that a purpose of the TDI SEI message comprises a description of a machine vision task that may be run or applied on a picture associated with the TDI SEI message. Additionally, or alternatively, the syntax element txt_descr_purpose of the TDI SEI message may take a value that indicates that a purpose of the TDI SEI message comprises a description of a machine vision task that may be trained by using a picture associated with the TDI SEI message.

[0268] In at least one example embodiment, the at least one rule indicates whether one or more aspects of the video may be used for Al models. For example, in at least one example embodiment, the information indicative of how a video may be used may comprise an indication of whether a particular data item derived from or associated with the video or pictures of the video may be used for one or more of the following: for training an Al model; as an input to an Al model during a training process of that Al model; as ground-truth forcomputing a training or validation loss for training or validating an Al model; or as input to an Al mode inference. In other words, in at least one example embodiment, the at least one rule indicates whether data associated with the video is usable for one or more operations associated with an artificial intelligence model, and wherein the one or more operations comprise at least one of the following: training, validation, or inference.

[0269] In some examples, a permission may be given by the Al usage restriction (e.g., the at least one rule). In some such examples, the decoder 57 may perform an operation according to the given permission.

[0270] In some examples, a data item derived from or associated with the video or pictures of the video may include text, such as text overlaid on top of visual content, or subtitles, or text present in the visual data (e.g., street signs, etc.), or text extracted from a speech or audio track associated with the video. Additionally, or alternatively, a data item derived from or associated with the video or pictures of the video may include visual content. Additionally, or alternatively, a data item derived from or associated with the video or pictures of the video may include features extracted from the visual content (or from any other data associated with or extracted from the visual content). Additionally, or alternatively, a data item derived from or associated with the video or pictures of the video may include higher-level information, such as human motion trajectories, human skeleton information, etc.

[0271] In at least one example embodiment, the at least one signal (e.g., the second information or other information) may indicate a quality metric threshold to which the at least one rule applies. For example, the second information may be indicative of a quality metric threshold, and application of the at least one rule to the one or more pictures may be based on whether a respective quality of the one or more pictures satisfies the quality metric threshold. In other words, in at least one example embodiment, the information indicative of how a video may be used may comprise a quality metric threshold such that when a picture quality is worse than the indicated threshold (e.g., fails to satisfy the indicated threshold), the usage restrictions (e.g., the at least one rule) apply, and when a picture quality is better than the indicated threshold (e.g., satisfies the indicated threshold), the usage restrictions (e.g., the at least one rule) do not apply.

[0272] In some examples, a quality metric threshold may be included in an AUR SEI message. For example, a AUR SEI message may comprise a quality metric threshold and / or an indication of a quality metric. In some examples, a AUR SEI message may comprise a reference data item that may be used for computing a quality metric value. For example,when the quality metric is Mean Squared Error (MSE), the reference data may be a reconstructed intra frame. In some examples, picture quality may be indicated by an SEI message, such as the quality metrics SEI message, whose syntax may be defined with the quality_metric( ) syntax structure.

[0273] In some examples, a quality metric threshold may be included in an AUR SEI message using essentially the same syntax as that used for indicating the picture quality. For example, the AUR SEI message may comprise quality_metric( ) syntax structure and the picture quality may be indicated by the quality metrics SEI message, whose syntax may be defined with the quality_metric( ) syntax structure. It may be required that the same quality metric type is used in both the quality metric threshold and the picture quality indication.

[0274] In some examples, the information indicative of how a video may be used may comprise or be associated with an indication that the information indicative how a video may be used or part of may apply to a subset of two or more temporal prediction layers of a video decoding pipeline. For example, a AUR SEI message may comprise an indication that Al restrictions specified in the AUR SEI message apply to the last (top-most) temporal prediction layer (e.g., the layer including the pictures that are not used as reference frame for inter-frame prediction). In some examples, the information indicative how a video may be used may apply to a subset of two or more temporal prediction layers of a video decoding pipeline, where such subset may be specified in a standard specification.

[0275] In some examples, the quality metric threshold may comprise a Quantization Parameter (QP) threshold. In one example, the second information may be indicative of a QP threshold, and application of the at least one rule to the one or more pictures may be based on whether a frame QP or slice QP associated to the one or more pictures satisfies the QP threshold. In another example, when a picture is associated to a slice QP higher that a QP threshold equal to 32 (or another suitable value), the picture may not be used for Al training.

[0276] In at least one example embodiment, the at least one signal (e.g., the second information or other information) may indicate a spatial region to which the at least one rule applies. For example, the at least one signal may comprise third information indicative of a spatial region, within one or more pictures, to which the at least one rule is applicable.

[0277] In some examples, an AUR SEI message comprises an indication, such as a flag or an indication (e.g., a region type indication), indicating if the Al usage restrictions apply to complete picture(s). In some examples, an AUR SEI message comprises an indication, such as a flag, indicating if the Al usage restrictions apply to indicated region(s). In some examples, an AUR SEI message comprises an indication, such as a flag, indicating whetherthe Al usage restrictions apply to complete picture(s) or to indicated region(s). In some examples, an AUR SEI message comprises an indication, such as a flag, indicating if the Al usage restrictions apply to leftover area(s), where a leftover area comprises a picture without the indicated region(s). In other words, the third information may indicate the spatial region by including a flag that indicates at least one of the following: whether the at least one rule is applicable to complete pictures, whether the at least one rule is applicable to one or more spatial regions associated with the flag, or whether the at least one rule is applicable to leftover areas, where a leftover area comprises a picture without one or more spatial regions associated with the flag.

[0278] In some examples, when the Al usage restrictions apply to leftover area(s) or indicated region(s), it is inferred that an annotated regions SEI message comprises the indicated region(s). In some examples, when the Al usage restrictions apply to leftover area(s) or indicated region(s), the AUR SEI message further comprises an indication of a type of indicating region(s). The potential types of indicating region(s) may comprise one or more of the following: regions indicated by an SEI message, in which case the AUR SEI message may further comprise a payload type value of the SEI message that indicates the region(s), regions indicated by an annotated regions SEI message, regions of an annotated regions SEI message that are indicated in the AUR SEI message, or spatial region(s) indicated through spatial coordinates within the AUR SEI message, such as top-left and bottom-right coordinates of a rectangle. In other words, in at least one example, embodiment, the at least one signal comprises fourth information indicative of a type of spatial region, and wherein the type of spatial region comprises at least one of the following: one or more spatial regions indicated by at least one SEI message, one or more spatial regions indicated by an annotated regions SEI message, one or more regions of an annotated regions SEI message that are indicated in an AUR SEI message, or one or more spatial regions indicated through spatial coordinates within an AUR SEI message.

[0279] In some examples, when the Al usage restrictions (or the at least one rule) apply to leftover area(s) or indicated region(s), the indicated region(s) are determined based at least on QP information of one or more blocks in the one or more pictures and on a QP threshold. In one example, a QP threshold or information from which a QP threshold is derived is indicated, e.g., in an Encoding Optimization Information (EOI) SEI message or in an AUR SEI message, and a block in a picture is determined to be comprised in a region when a QP associated to the block is lower than the QP threshold.

[0280] In some examples, an AUR SEI message may comprise an indication of whether the at least one rule applies to indicated regions or to areas outside indicated regions (e.g., leftover areas).

[0281] In some examples, an AUR SEI message comprises an indication that Al usage restrictions specified in the AUR SEI message apply to one or more types of data output by an NNPF, such as spatially extrapolated data (e.g., pixels that were generated by a spatial extrapolation NNPF), or temporally extrapolated data (e.g., pictures that were generated by a temporal extrapolation NNPF), or temporally interpolated data (e.g., pictures that were generated by a temporal interpolation NNPF). In other words, in at least one example embodiment, the at least one signal comprises fifth information indicating that the at least one rule is applicable to one or more types of data that is output by an NNPF and is associated with the one or more pictures, where the one or more types of data include at least one of the following: spatially extrapolated data, temporally extrapolated data, or temporally interpolated data.

[0282] In some examples, the information indicative of how a video may be used may comprise or may be associated to an indication of one or more spatial regions within a picture; the information indicative of how a video may be used may refer or be applicable to the indicated one or more spatial regions.

[0283] In some examples, the information indicative of how a video may be used is comprised in an AUR SEI message, and the one or more spatial regions are indicated by means of an Annotated Regions SEI message. In some examples, the one or more spatial regions comprise the object bounding boxes indicated in the associated Annotated Regions SEI message. In some examples, the one or more spatial regions comprise the leftover region that is not included in any of the object bounding boxes indicated in the associated Annotated Regions SEI message. In some examples, the one or more spatial regions comprise those object bounding boxes that are indicated in the associated Annotated Regions SEI message and identified in the AUR SEI message, for example by listing the object index values, the object label index values or the object labels of the object bounding boxes to which the usage restrictions apply. The associated Annotated Regions SEI message may be defined to be the Annotated Regions SEI message that persists for the picture for which the AUR SEI message applies.

[0284] In some examples, an AUR SEI message comprises a region indication type syntax element, the potential values of which may be indicative of a complete picture and / or specific types of indicating spatial region(s). In an example, a first value of the regionindication type syntax element indicates complete pictures, a second value of the region indication type syntax element indicates object bounding boxes indicated by the associated annotated regions SEI message, and a third value of the region indication type syntax element indicates the leftover region formed by the complete picture excluding all the bounding boxes indicated by the associated annotated regions SEI message.

[0285] In some examples, a decoder decodes an AUR SEI message comprising a region indication type syntax element, and in response to a region indication type that the decoder does not recognize or support, the decoder determines to apply the indicated Al usage restriction to a complete picture.

[0286] In some examples, an AUR SEI message comprises a region indication type (aur_region_type[ i ]) per each Al usage restriction indicated in the AUR SEI message. In an example, the syntax of the following Table 8 may be used.<

[0287] In an example, aur_region_type[ i ] may have the following semantics: aur_region_type[ i ] equal to 0 specifies that the i-th indicated Al usage restriction applies to complete pictures. aur_region_type[ i ] equal to 1 specifies that the i-th indicated Al usage restriction applies to the object bounding boxes indicated by the associated annotated regions SEI message. aur_region_type[ i ] equal to 2 specifies that the i-th Al usage restriction applies to the leftover region formed by the complete picture excluding all the bounding boxes indicated by the associated annotated regions SEI message. When both an annotated regions SEI message arSei and an Al usage restrictions SEI message aurSei persist for aparticular picture picA, arSei is the associated annotated regions SEI message for aurSei and picA.

[0288] In some examples, an encoder encodes and / or a decoder decodes multiple AUR SEI messages that persist for a same picture, differentiated by their region indication types. An AUR SEI message syntax comprises a region indication type (aur_region_type) syntax element. The semantics of aur_cancel_flag and aur_persistence_flag are specified to apply for the indicated value of aur_region_type. In an example, the syntax of the following Table 9 may be used.<

[0289] In an example, aur_region_type may have the following semantics: aur_region_type equal to 0 specifies that indicated Al usage restrictions apply to complete pictures. aur_region_type equal to 1 specifies that the indicated Al usage restrictions apply to the object bounding boxes indicated by the associated annotated regions SEI message. aur_region_type equal to 2 specifies that the indicated Al usage restrictions apply to the leftover region formed by the complete picture excluding all the bounding boxes indicated by the associated annotated regions SEI message. When both an annotated regions SEI message arSei and an Al usage restrictions SEI message aurSei persist for a particular picture picA, arSei is the associated annotated regions SEI message for aurSei and picA.

[0290] In some examples, an AUR SEI message indicates a restriction aur_restriction[i] equal to 0 (do not use for any Al application), and is associated with an indicated region. Thus, the indicated region in the picture to which the AUR SEI message applies should not be used for any Al application. In one example, an AUR SEI message indicates a restrictionaur_restriction[i] equal to 0 (do not use for any Al application), and is associated with an indicated region. Thus, any regions except the indicated region in the picture to which the AUR SEI message applies may not be used for any Al application.

[0291] In some examples, an AUR SEI message indicates a restriction aur_restriction[i] equal to 2 (e.g., do not use for generative (modification or creation) Al) and is associated with an indicated region. In some such examples, the indicated region in the picture to which the AUR SEI message applies may not be used for generative (modification or creation) Al.

[0292] In some examples, the AUR SEI message may indicate the region(s) to be those regions indicated in an annotated regions SEI message that satisfy one or more of the following: a bounding box of an object associated with a label text string ar_label[ arLabelldx ] that is equal to any of the label text strings indicated in the AUR SEI message, a bounding box of an object associated with a label index ar_label_idx[ i ] is equal to any of the label indices indicated in the AUR SEI message, or a bounding box of an object with an object index ar_object_idx[ i ] equal to any of the object indices indicated in the AUR SEI message.

[0293] In some examples, the information indicative of how a video may be used is comprised in an AUR SEI message, and the one or more spatial regions are indicated by means of a saliency map, an object mask, or alike.

[0294] In some examples, video input is provided to an encoder along with an optional saliency map. If an external saliency map is not provided, it is assumed that the encoder calculates such a map internally. An encoder uses the saliency map to modulate the quantization parameter (QP) values during encoding so that blocks including salient areas use more-fine grained quantization than blocks outside of salient areas. The encoder also sets appropriate SEI message(s) that communicate parameters specific to the saliency map so that a decoder may determine which blocks include salient areas and which blocks are outside salient areas. For example, an encoder optimization information SEI message may comprise information indicative of a first quantization parameter range for blocks including salient areas and / or a second quantization parameter range for blocks outside salient areas.

[0295] In some examples, a video decoder reconstructs decoded video from the bitstream along with a saliency map calculated by thresholding QP values. The decoder may decode appropriate SEI message(s), such as an encoder optimization information SEI message, to determine a first quantization parameter range for blocks that include salient areas and / or a second quantization parameter range for blocks that are outside salient areas. The decoder may then classify blocks according to their QP value to comprise salient areas or to be outside salient areas and consequently form a saliency map through such thresholding ofblock-wise QP values. Both the decoded video and saliency map can be used in further video processing blocks.

[0296] In some examples, the information indicative of how a video may be used is comprised in an AUR SEI message, and the one or more spatial regions are indicated by means of a saliency map that is calculated by thresholding QP values as described above.

[0297] FIG. 4 illustrates an example flowchart of a method to which one or more examples disclosed herein may be applied. The method may be computer-implemented. The method may be performed by a device, such as an encoder illustrated by and described with reference to FIGs. 1 and 3. In some examples, the encoder may be an example of an apparatus 40 illustrated by and described with reference to FIG. 2.

[0298] As shown in FIG. 4, the encoder at block 60 generates first information indicative of a video. For example, the encoder may include the means (e.g., a processor 42, a memory 44) for generating first information indicative of a video.

[0299] As shown in FIG. 4, the encoder at block 62 generates second information indicative of at least one rule for usage of the video. For example, the encoder may include the means (e.g., a processor 42, a memory 44) for generating second information indicative of at least one rule for usage of the video.

[0300] As shown in FIG. 4, the encoder at block 64 outputs at least one signal comprising a representation of the first information and the second information. For example, the encoder may include the means (e.g., a processor 42, a memory 44, a radio interface 49) for outputting at least one signal comprising a representation of the first information and the second information.

[0301] FIG. 5 illustrates an example flowchart of a method to which one or more examples disclosed herein may be applied. The method may be computer-implemented. The method may be performed by a device, such as an decoder illustrated by and described with reference to FIGs. 1 and 2. In some examples, the encoder may be an example of an apparatus 40 illustrated by and described with reference to FIG. 2.

[0302] As shown in FIG. 5, the decoder at block 70 receives at least one signal comprising a representation of first information indicative of a video and second information indicative of at least one rule for usage of the video. For example, the decoder may include the means (e.g., a processor 42, a memory 44, a radio interface 49) for receiving at least one signal comprising a representation of first information indicative of a video and second information indicative of at least one rule for usage of the video.

[0303] As shown in FIG. 5, the decoder at block 72 decodes the at least one signal to obtain the first information and the second information. For example, the decoder may include the means (e.g., a processor 42, a memory 44) for decoding the at least one signal to obtain the first information and the second information.

[0304] Even though the present disclosure has been described above with reference to an example according to the accompanying drawings, it is clear that the present disclosure is not restricted thereto but may be modified in several ways within the scope of the appended claims. Therefore, all words and expressions should be interpreted broadly, and they are intended to illustrate, not to restrict, the embodiment. It will be obvious to a person skilled in the art that, as technology advances, the inventive concept may be implemented in various ways. Further, it is clear to a person skilled in the art that the described embodiments may, but are not required to, be combined with other embodiments in various ways.

Claims

What is claimed is:

1. An apparatus, comprising:at least one processor; andat least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to:generate first information indicative of a video;generate second information indicative of at least one rule for usage of the video; andoutput at least one signal comprising a representation of the first information and the second information.

2. An apparatus according to claim 1, wherein the at least one signal comprises a supplemental enhancement information (SEI) message including the second information.

3. An apparatus according to claim 2, wherein the SEI message comprises an artificial intelligence usage restrictions (AUR) SEI message or a neural network post filter (NNPF) SEI message.

4. An apparatus according to claim 2, wherein the SEI message comprises a text description information SEI message.

5. An apparatus according to claim 2, wherein the SEI message comprises an artificial intelligence usage restrictions (AUR) SEI message, and wherein a value of an indicator included in the AUR SEI message indicates the at least one rule by indicating, to a receiver of the at least one signal, at least one of the following:to refrain from using the video as input to an artificial intelligence model for training of the artificial intelligence model; orto refrain from using the video to determine a ground-truth for training an artificial intelligence model.

6. An apparatus according to claim 5, wherein an AUR SEI message type associated with the AUR SEI message is associated with an SEI processing order (SPO) SEI message to indicate that the AUR SEI message is descriptive of at least one of the following:characteristics of a processing chain indicated in the SPO SEI message, pictures output by a processing chain indicated in the SPO SEI message, or a result of a processing chain indicated in the SPO SEI message.

7. An apparatus according to any of the claims 5 or 6, wherein the AUR SEI message is included in a processing order nesting (PON) SEI message and is associated with a processing order value.

8. An apparatus, comprising:at least one processor; andat least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to:receive at least one signal comprising a representation of first information indicative of a video and second information indicative of at least one rule for usage of the video; anddecode the at least one signal to obtain the first information and the second information.

9. An apparatus according to claim 8, wherein the at least one signal comprises a supplemental enhancement information (SEI) message including the second information.

10. An apparatus according to claim 9, wherein the SEI message comprises an artificial intelligence usage restrictions (AUR) SEI message or a neural network post filter (NNPF) SEI message.

11. An apparatus according to claims 9, wherein the SEI message comprises a text description information SEI message.

12. An apparatus according to claim 9, wherein the SEI message comprises an artificial intelligence usage restrictions (AUR) SEI message, and wherein a value of an indicator included in the AUR SEI message indicates the at least one rule by indicating, to the apparatus, at least one of the following:to refrain from using the video as input to an artificial intelligence model for training of the artificial intelligence model; orto refrain from using the video to determine a ground-truth for training an artificialintelligence model.

13. An apparatus according to claim 12, wherein an AUR SEI message type associated with the AUR SEI message is associated with an SEI processing order (SPO) SEI message to indicate that the AUR SEI message indicates at least one of the following:characteristics of a processing chain indicated in the SPO SEI message, pictures output by a processing chain indicated in the SPO SEI message, or a result of a processing chain indicated in the SPO SEI message.

14. An apparatus according to any of claims 12 or 13, wherein the AUR SEI message is included in a processing order nesting (PON) SEI message and is associated with a processing order value.

15. An apparatus according to any one of claims 12 through 14, wherein the instructions, when executed by the at least one processor, cause the apparatus at least to: decode the AUR SEI message; anddetermine that the at least one rule is applicable to a result of a processing chain based at least in part on the decoded AUR SEI message.

16. A method, comprising:generating first information indicative of a video;generating second information indicative of at least one rule for usage of the video; andoutputting at least one signal comprising a representation of the first information and the second information.

17. A method according to claim 16, wherein the at least one signal comprises a supplemental enhancement information (SEI) message including the second information.

18. A method according to claim 17, wherein the SEI message comprises an artificial intelligence usage restrictions (AUR) SEI message or a neural network post filter (NNPF) SEI message.

19. A method according to claim 17, wherein the SEI message comprises a text description information SEI message.

20. A method according to claim 17, wherein the SEI message comprises an artificial intelligence usage restrictions (AUR) SEI message, and wherein a value of an indicator included in the AUR SEI message indicates the at least one rule by indicating, to a receiver of the at least one signal, at least one of the following:to refrain from using the video as input to an artificial intelligence model for training of the artificial intelligence model; orto refrain from using the video to determine a ground-truth for training an artificial intelligence model.

21. A method according to claim 20, wherein an AUR SEI message type associated with the AUR SEI message is associated with an SEI processing order (SPO) SEI message to indicate that the AUR SEI message is descriptive of at least one of the following:characteristics of a processing chain indicated in the SPO SEI message, pictures output by a processing chain indicated in the SPO SEI message, or a result of a processing chain indicated in the SPO SEI message.

22. A method according to any of the claims 20 or 21, wherein the AUR SEI message is included in a processing order nesting (PON) SEI message and is associated with a processing order value.

23. A method, comprising:receiving at least one signal comprising a representation of first information indicative of a video and second information indicative of at least one rule for usage of the video; and decoding the at least one signal to obtain the first information and the second information.

24. A method according to claim 23, wherein the at least one signal comprises a supplemental enhancement information (SEI) message including the second information.

25. A method according to claim 24, wherein the SEI message comprises an artificial intelligence usage restrictions (AUR) SEI message or a neural network post filter (NNPF) SEI message.

26. A method according to claims 24, wherein the SEI message comprises a text description information SEI message.

27. A method according to claim 24, wherein the SEI message comprises an artificial intelligence usage restrictions (AUR) SEI message, and wherein a value of an indicator included in the AUR SEI message indicates the at least one rule by indicating at least one of the following:to refrain from using the video as input to an artificial intelligence model for training of the artificial intelligence model; orto refrain from using the video to determine a ground-truth for training an artificial intelligence model.

28. A method according to claim 27, wherein an AUR SEI message type associated with the AUR SEI message is associated with an SEI processing order (SPO) SEI message to indicate that the AUR SEI message indicates at least one of the following: characteristics of a processing chain indicated in the SPO SEI message, pictures output by a processing chain indicated in the SPO SEI message, or a result of a processing chain indicated in the SPO SEI message.

29. A method according to any of the claims 27 or 28, wherein the AUR SEI message is included in a processing order nesting (PON) SEI message and is associated with a processing order value.

30. A method according to any one of claims 27 through 29, wherein the method further comprises:decoding the AUR SEI message; anddetermining that the at least one rule is applicable to a result of a processing chain based at least in part on the decoded AUR SEI message.

Images