Encoding device, decoding device, encoding method, and decoding method

By encoding neural network loop filter models with dedicated data units in video coding, the flexibility and efficiency of encoding and decoding operations are enhanced, addressing the inflexibility in existing technologies.

WO2026239886A1PCT designated stage Publication Date: 2026-11-19PANASONIC INTELLECTUAL PROPERTY CORP OF AMERICA
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2026/016244
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2025-05-16
Filing Date
2026-04-23
Publication Date
2026-11-19

AI Technical Summary

Technical Problem

Existing video coding technologies, such as VVC, lack flexibility in switching neural network loop filtering processes on a slice-by-slice or picture-by-picture basis, leading to inefficiencies in encoding and decoding operations.

Method used

Implementing a configuration where neural network loop filter models are encoded with dedicated data units, such as APS or NAL units, allowing for flexible switching on a slice-by-slice, sub-picture-by-subpicture, or scene-by-scene basis, with identifiers in the picture or slice headers.

Benefits of technology

Enables more efficient and flexible transmission of neural network loop filter model switching, reducing the frequency of signaling events and improving encoding/decoding efficiency, image quality, and processing speed.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2026016244_19112026_PF_FP_ABST
    Figure JP2026016244_19112026_PF_FP_ABST
Patent Text Reader

Abstract

An encoding device (100) comprises a circuit (a1) and a memory (a2) that is connected to the circuit (a1). When operating, the circuit (a1) encodes a data unit for each of a plurality of neural network loop filter models that includes a parameter set related to the neural network loop filter model and can be configured using only the parameter set and encodes a first identifier for identifying the plurality of neural network loop filter models into a picture header or a slice header.
Need to check novelty before this filing date? Find Prior Art

Description

Encoding device, decoding device, encoding method, and decoding method

[0001] The present disclosure relates to an encoding device and the like.

[0002] Video coding technology has advanced from H.261 and MPEG-1 to H.264 / AVC (Advanced Video Coding), H.265 / HEVC (High Efficiency Video Coding), and H.266 / VVC (Versatile Video Coding). Along with this progress, there is a constant need to provide improvements and optimizations to video coding technology in order to process the ever-increasing amount of digital video data in various applications. The present disclosure relates to further advances, improvements and optimizations in video coding.

[0003] Note that Non-Patent Document 1 relates to an example of a conventional standard related to the video coding technology described above.

[0004] H.266 (ISO / IEC 23090-3) / VVC (Versatile Video Coding)

[0005] With respect to the above encoding schemes, proposals for new schemes are desired for purposes such as improving encoding efficiency, improving image quality, reducing processing load, reducing circuit scale, or appropriately selecting elements or operations such as filters, blocks, sizes, motion vectors, reference pictures, or reference blocks.

[0006] The present disclosure provides a configuration or method that can contribute to one or more of, for example, improving encoding efficiency, improving image quality, reducing processing load, reducing circuit scale, improving processing speed, appropriately selecting elements or operations, improving processing applied to decoded images, and providing new processing. Note that the present disclosure may include configurations or methods that can contribute to benefits other than those described above.

[0007] For example, an encoding device according to one aspect of the present disclosure comprises a circuit and a memory connected to the circuit, wherein the circuit, in operation, encodes a data unit for each of a plurality of neural network loop filter models, which includes a parameter set relating to the neural network loop filter model and can be composed solely of the parameter set, and encodes a first identifier for identifying the plurality of neural network loop filter models in a picture header or slice header.

[0008] Each embodiment, or any part thereof, of the present disclosure enables at least one of the following: improved encoding efficiency, improved image quality, reduced encoding / decoding processing load, reduced circuit size, or improved encoding / decoding processing speed. Alternatively, each embodiment, or any part thereof, of the present disclosure enables appropriate selection of components / operations such as filters, blocks, sizes, motion vectors, reference pictures, and reference blocks in encoding and decoding. The present disclosure also includes disclosures of configurations or methods that may provide benefits other than those mentioned above, such as configurations or methods that improve encoding efficiency while suppressing an increase in processing load.

[0009] Further advantages and effects of one aspect of this disclosure will be made apparent from the specification and drawings. Such advantages and / or effects may be obtained by several embodiments and features described in the specification and drawings, but not all of them are necessarily provided to obtain one or more advantages and / or effects.

[0010] These general or specific embodiments may be implemented as a system, integrated circuit, computer program, or recording medium such as a computer-readable CD-ROM, or as any combination of system, method, integrated circuit, computer program, and recording medium.

[0011] A configuration or method relating to one aspect of this disclosure may contribute to one or more of the following: improved encoding efficiency, improved image quality, reduced processing load, reduced circuit size, improved processing speed, and appropriate selection of elements or operations. A configuration or method relating to one aspect of this disclosure may also contribute to other benefits not mentioned above.

[0012] This is a schematic diagram showing an example of the configuration of a transmission system according to an embodiment. This is a block diagram showing an example of the implementation of an encoding device according to an embodiment. This is a block diagram showing an example of the configuration of an encoding device according to an embodiment. This is a flowchart showing an example of the overall encoding process by the encoding device according to an embodiment. This is a flowchart showing an example of the processing for each block by the encoding device according to an embodiment. This is a block diagram showing an example of the implementation of a decoding device according to an embodiment. This is a block diagram showing an example of the configuration of a decoding device according to an embodiment. This is a flowchart showing an example of the overall decoding process by the decoding device according to an embodiment. This is a flowchart showing an example of the processing for each block by the decoding device according to an embodiment. This is a schematic diagram showing another example of the configuration of a transmission system according to an embodiment. This is a schematic diagram showing an example of the configuration of a transmitting device according to an embodiment. This is a block diagram showing an example of the implementation of a transmitting device according to an embodiment. This is a schematic diagram showing an example of the configuration of a receiving device according to an embodiment. This is a block diagram showing an example of the implementation of a receiving device according to an embodiment. This is a schematic diagram showing an example of the configuration of a bitstream generation device according to an embodiment. This is a block diagram showing an example of the implementation of a bitstream generation device according to an embodiment. This is a schematic diagram showing an example of the configuration of a storage medium and a computer according to an embodiment. This is a schematic diagram showing an example of the configuration of a computer according to an embodiment. This is a diagram showing an example of the hierarchical structure of data in a stream. This is a diagram showing an example of the configuration of a slice. This is a diagram showing an example of the configuration of a tile. This is a diagram showing an example of the configuration of a time-scalable stream. This is a diagram showing an example of the configuration of a stream using multilayer encoding. This is a block diagram showing an example of the configuration of a loop filter section. This is a diagram showing an example of the filter shape used in an ALF (adaptive loop filter). This is a diagram showing another example of the filter shape used in an ALF. This is a diagram showing another example of the filter shape used in an ALF. This is a diagram showing an example of a CCALF. This is a diagram showing the filter shape of a CCALF. This is a block diagram showing an example of the detailed configuration of a loop filter section that functions as a DBF. This is a flowchart showing an example of the processing in the DBF processing section. This is a flowchart showing another example of the processing in the DBF processing section.This figure shows an example of a DBF having filter characteristics symmetric with respect to block boundaries. This figure illustrates an example of a block boundary on which DBF processing is performed. This figure shows an example of a block boundary strength Bs value. This block diagram shows an example of the configuration of a loop filter section. This flowchart shows an example of processing performed in the prediction section of an encoding device. This flowchart shows another example of processing performed in the prediction section of an encoding device. This flowchart shows another example of processing performed in the prediction section of an encoding device. This conceptual diagram shows another example of a reference picture. This conceptual diagram shows another example of generating a generated reference picture. This conceptual diagram shows another example of generating a generated reference picture. This conceptual diagram shows an example of a generated picture. This conceptual diagram shows another example of generating a generated picture. This conceptual diagram shows another example of generating a generated picture. This conceptual diagram shows a pipeline processing. This block diagram shows an example of a configuration used in NN processing. This block diagram shows another example of a configuration used in NN processing. This conceptual diagram shows a patch used in NN processing. This conceptual diagram shows an extended patch containing an extended region. This conceptual diagram shows an image containing two patches with overlapping regions. This conceptual diagram shows patches defined in each picture. This conceptual diagram shows an example of a patch generation method. This conceptual diagram shows another example of a generated picture. This is a conceptual diagram illustrating another example of generating a generated picture. This is a conceptual diagram illustrating another example of generating a generated picture. This is a conceptual diagram illustrating another example of generating a generated picture. This is a diagram illustrating an overview of the high-level syntax structure proposed in this disclosure. This is a flowchart illustrating an overview of processing by the encoding device of this disclosure. This is a flowchart illustrating an overview of processing by the decoding device of this disclosure. This is a diagram illustrating an example of the configuration of an NNLF APS in a first embodiment and a first example of a method for referencing an NNLF APS. This is a diagram illustrating an example of the configuration of an NNLF APS in a first embodiment and a second example of a method for referencing an NNLF APS. This is a diagram illustrating an example of the configuration of an NAL unit in a first embodiment and an example of a method for referencing an NAL unit. This is a conceptual diagram when the NNLF parameter set is changed for each scene. This is a conceptual diagram when the NNLF parameter set is changed for each picture.This is a conceptual diagram showing how the NNLF parameter set is changed for each subpicture and slice. This diagram illustrates an example of the configuration of an NNLF APS in the second embodiment and a specific example of how to reference an NNLF APS. This diagram illustrates an example of the configuration of an NAL unit in the second embodiment and an example of how to reference an NAL unit. This diagram illustrates an example of the configuration of an NNLF APS in the third embodiment and a specific example of how to reference an NNLF APS. This diagram illustrates an example of the configuration of an NAL unit in the third embodiment and an example of how to reference an NAL unit. This diagram illustrates an example of the configuration of an NNLF APS in the fourth embodiment and a specific example of how to reference an NNLF APS. This diagram illustrates an example of the configuration of an NAL unit in the fourth embodiment and an example of how to reference an NAL unit. This flowchart shows the basic processing in the encoding operation performed by the encoding device. This flowchart shows the basic processing in the decoding operation performed by the decoding device. This diagram shows an example of the overall configuration of a content supply system that realizes a content distribution service. This diagram shows an example of the configuration of a content distribution system. This diagram shows an example of a web page display screen. This figure shows an example of a web page display screen. This figure shows an example of a smartphone. This block diagram shows an example of a smartphone configuration.

[0013] [Introduction] This disclosure relates to the signaling of parameter sets for NNLF (Neural Network Loop Filter) in a bitstream. More specifically, this disclosure relates to performing NNLF parameter set signaling on a slice-by-slice, sub-picture-by-subpicture, picture-by-picture, or scene-by-scene basis. This disclosure also provides an apparatus and method that can embody NNLF parameter set signaling.

[0014] The following sections will explain some terms and describe techniques that can be used to signal the parameter set for the NNLF of this disclosure.

[0015] For example, in standards for conventional video coding technologies such as VVC (Versatile Video Coding), there is only one SPS (Special Picture Score) for the entire video content. Therefore, when signaling information that applies only to a specific slice (i.e., a part of the image) within an image, the slice header is designed to be used. In addition, in VVC and similar technologies, information that applies to the entire image is signaled using the picture header, not the slice header. In other words, in standards for conventional video coding technologies such as VVC, it is possible to switch between signaling on a slice-by-slice or picture-by-picture basis depending on the information being applied.

[0016] However, the NNVC-v12 currently under consideration uses a method of signaling the NNLF parameter set with an SPS (Sequence Parameter Set), which presents a problem in that it is not possible to switch the neural network loop filtering process on a slice-by-slice or picture-by-picture basis.

[0017] Therefore, the encoding device of Example 1 comprises a circuit and a memory connected to the circuit, wherein the circuit, in operation, encodes a data unit for each of a plurality of neural network loop filter models, which includes a parameter set relating to the neural network loop filter model and can be composed solely of the parameter set, and encodes a first identifier for identifying the plurality of neural network loop filter models into a picture header or slice header.

[0018] This may allow the decoder to transmit to the decoder that the neural network loop filter model is switching for each slice, sub-picture, picture, or scene, using a dedicated data unit that can be composed solely of the parameter set for the neural network loop filter model. Furthermore, since a dedicated data unit is used for each neural network loop filter model, it may be possible to transmit the switching of the neural network loop filter model to the decoder with greater flexibility.

[0019] Furthermore, the encoding device in Example 2 may be the encoding device in Example 1, wherein the data unit is an APS (Adaptation Parameter Set).

[0020] This allows for the use of a dedicated APS for each neural network loop filter model, potentially enabling more flexible transmission to the decoding device regarding the switching of neural network loop filter models for each slice, sub-picture, picture, or scene.

[0021] Furthermore, the encoding device in Example 3 may be the encoding device in Example 1, wherein the data unit is a NAL (Network Abstraction Layer) unit.

[0022] This allows for the use of a dedicated NAL unit for each neural network loop filter model, potentially enabling more flexible transmission to the decoding device regarding the switching of neural network loop filter models for each slice, sub-picture, picture, or scene.

[0023] Furthermore, the encoding device in Example 4 may be any of the encoding devices in Examples 1 to 3, wherein when the circuit encodes the first identifier into the slice header, the data unit is signaled before or after the picture data or slice data.

[0024] This allows for signaling information that is likely to be shared between multiple slices or pictures before or after the picture or slice data, potentially reducing the number of signaling events related to that information. Consequently, it may be possible to more efficiently transmit information about the switching of the neural network loop filter model to the decoder.

[0025] Furthermore, the encoding device of Example 5 may be any encoding device of Examples 1 to 4, wherein the parameter set relating to the neural network loop filter model includes a plurality of subparameter sets which are parameter sets for defining the neural network loop filter model, each of the plurality of subparameter sets is assigned a second identifier, and the circuit may, in operation, further encode the second identifier corresponding to at least some of the plurality of subparameter sets in the picture header when encoding the first identifier in the picture header, and encode the second identifier corresponding to at least some of the plurality of subparameter sets in the slice header when encoding the first identifier in the slice header.

[0026] This may allow the decoder to communicate that the sub-parameter set for the neural network loop filter model is switching for each slice, each sub-picture, each picture, or each scene.

[0027] Furthermore, the encoding device of Example 6 may be any encoding device of Examples 1 to 4, wherein the parameter set relating to the neural network loop filter model includes a subparameter set which is a parameter set for defining the neural network loop filter model, and the subparameter set is a parameter set which is uniquely determined for the neural network loop filter model.

[0028] This can simplify the configuration of data units. Therefore, when the parameter set changes from scene to scene, if a group of pictures in the same scene share the same parameter set, the amount of information to encode in the picture header can be reduced, and signaling may become more efficient.

[0029] Furthermore, when the parameter set is changed for each picture or sub-picture, information such as QP adaptation, scale, and offset may change for each picture or sub-picture. However, since this information may be shared across multiple slices or sub-pictures within a picture, signaling this information with the data unit mentioned above can reduce the frequency of information transmission and may even enable out-of-band transmission.

[0030] Furthermore, the encoding device of Example 7 may be any encoding device of Examples 1 to 4, wherein the parameter set relating to the neural network loop filter model includes a subparameter set which is a parameter set for defining the neural network loop filter model, and the circuit may, in operation, further encode a rewrite subparameter set for rewriting a part of the subparameter set into the picture header when encoding the first identifier into the picture header, and encode the rewrite subparameter set into the slice header when encoding the first identifier into the slice header.

[0031] This may allow the decoder to communicate that some of the sub-parameter sets contained in the data unit have been rewritten, on a slice-by-slice, sub-picture-by-subpicture, picture-by-picture, or scene-by-scene basis.

[0032] Furthermore, the encoding device of Example 8 may be any encoding device of Examples 1 to 4, wherein the circuit further encodes a subparameter set, which is a set of parameters for defining the neural network loop filter model, into the picture header when encoding the first identifier into the picture header, and encodes the subparameter set into the slice header when encoding the first identifier into the slice header.

[0033] This may allow parameter sets that may vary per slice or picture to be transmitted to the decoder in the slice header or picture header, and parameter sets that may be shared between different pictures or slices to be transmitted to the decoder in the data unit. Therefore, it is no longer necessary to encode parameter sets that are repeatedly used between slices or pictures in each slice header or picture header, and flexibility is maintained such that parameter sets can be changed for each scene without retransmitting the entire SPS.

[0034] Furthermore, the encoding device of Example 9 may be any encoding device of Examples 1 to 8, wherein the parameter set relating to the neural network loop filter model includes a subparameter set which is a parameter set for defining the neural network loop filter model, the subparameter set includes a parameter set for defining the output format for the luminance component and the chrominance component included in the output data obtained by applying the neural network loop filter model to the current block, and the circuit may further encode a flag in the picture header to identify whether or not to set the subparameter set in the neural network loop filter model when encoding the first identifier in the picture header, and encode the flag in the slice header in the neural network loop filter model when encoding the first identifier in the slice header.

[0035] This makes it possible to transmit the output format for the luminance component and the color difference component of the output data to the decoder via the slice header or picture header, even when these formats are changed for each slice or picture.

[0036] Furthermore, the decoding device of Example 10 comprises a circuit and a memory connected to the circuit, wherein the circuit, in operation, decodes a data unit for each of a plurality of neural network loop filter models, which includes a parameter set relating to the neural network loop filter model and can be composed solely of the parameter set, and decodes a first identifier for identifying the plurality of neural network loop filter models from a picture header or slice header.

[0037] This may allow switching between neural network loop filter models on a slice-by-slice, sub-picture-by-subpicture, picture-by-picture, or scene-by-scene basis, using a dedicated data unit composed solely of parameter sets related to the neural network loop filter model. Furthermore, since a dedicated data unit is used for each neural network loop filter model, it may be possible to switch between neural network loop filter models more flexibly.

[0038] Furthermore, the decoding device in Example 11 may be the decoding device in Example 10, wherein the data unit is an APS (Adaptation Parameter Set).

[0039] This allows for the use of a dedicated APS for each neural network loop filter model, potentially enabling more flexible switching between neural network loop filter models on a slice-by-slice, sub-picture-by-subpicture, picture-by-picture, or scene-by-scene basis.

[0040] Furthermore, the decoding device in Example 12 may be the decoding device in Example 10, wherein the data unit is a NAL (Network Abstraction Layer) unit.

[0041] This allows for the use of a dedicated NAL unit for each neural network loop filter model, potentially enabling more flexible switching between neural network loop filter models on a slice-by-slice, sub-picture-by-subpicture, picture-by-picture, or scene-by-scene basis.

[0042] Furthermore, the decoding device in Example 13 may be any decoding device from Examples 10 to 12, wherein when the circuit decodes the first identifier from the slice header, the data unit is signaled before or after the picture data or slice data.

[0043] This means that information likely to be shared across multiple slices or pictures is signaled before or after the picture or slice data, potentially reducing the number of signaling events related to that information. Consequently, the switching of the neural network loop filter model may be transmitted more efficiently.

[0044] Furthermore, the decoding device of Example 14 may be any decoding device of Examples 10 to 13, wherein the parameter set relating to the neural network loop filter model includes a plurality of subparameter sets which are parameter sets for defining the neural network loop filter model, and a second identifier is assigned to the plurality of subparameter sets, and the circuit may, in operation, further decode the second identifiers corresponding to at least some of the plurality of subparameter sets from the picture header when decoding the first identifier from the picture header, and decode the second identifiers corresponding to at least some of the plurality of subparameter sets from the slice header when decoding the first identifier from the slice header.

[0045] This may make it possible to perform processing by switching the sub-parameter set set in the neural network loop filter model for each slice, each sub-picture, each picture, or each scene.

[0046] Further, the decoding apparatus of Example 15 is the decoding apparatus according to any one of Examples 10 to 13, wherein the parameter set related to the neural network loop filter model includes a sub-parameter set that is a parameter set for defining the neural network loop filter model, and the sub-parameter set may be a decoding apparatus which is a parameter set uniquely determined for the neural network loop filter model.

[0047] Thereby, the configuration of the data unit can be simplified. Therefore, when the parameter set is changed for each scene, if a group of pictures in the same scene share the same parameter set, the amount of information decoded from the picture header can be reduced, and signaling may be further made more efficient in some cases.

[0048] Further, when the parameter set is changed for each picture or each sub-picture, information such as QP adaptation, scale, and offset may change for each picture or each sub-picture. However, since such information may be shared by a plurality of slices or sub-pictures in a picture, signaling such information in the above data unit may reduce the transmission frequency of the information, and may also enable transmission out of band in some cases.

[0049] Furthermore, the decoding device of Example 16 may be any decoding device of Examples 10 to 13, wherein the parameter set relating to the neural network loop filter model includes a subparameter set which is a parameter set for defining the neural network loop filter model, and the circuit may, in operation, further decode a rewrite subparameter set for rewriting a part of the subparameter set from the picture header when decoding the first identifier from the picture header, and decode the rewrite subparameter set from the slice header when decoding the first identifier from the slice header.

[0050] This may allow processing to be performed by rewriting some of the sub-parameter sets contained in the data unit, on a slice-by-slice, sub-picture-by-subpicture, picture-by-picture, or scene-by-scene basis.

[0051] Furthermore, the decoding device of Example 17 may be any decoding device of Examples 10 to 13, wherein the circuit further decodes a subparameter set, which is a set of parameters for defining the neural network loop filter model, from the picture header when decoding the first identifier from the picture header, and decodes the subparameter set from the slice header when decoding the first identifier from the slice header.

[0052] This means that parameter sets that may change per slice or per picture are transmitted in the slice header or picture header, while parameter sets that may be shared between different pictures or slices are transmitted in the data unit. Therefore, it is no longer necessary to decode parameter sets that are repeatedly used between slices or pictures per slice header or picture header, and flexibility is maintained such that parameter sets can be changed per scene without re-receiving the entire SPS.

[0053] Furthermore, the decoding device of Example 18 may be any decoding device of Examples 10 to 17, wherein the parameter set relating to the neural network loop filter model includes a subparameter set which is a parameter set for defining the neural network loop filter model, the subparameter set includes a parameter set for defining the output format for the luminance component and the chrominance component included in the output data obtained by applying the neural network loop filter model to the current block, and the circuit may further, in operation, decode a flag from the picture header to identify whether or not to set the subparameter set for the neural network loop filter model when decoding the first identifier from the picture header, and decode the flag from the slice header for the neural network loop filter model when decoding the first identifier from the slice header.

[0054] This may allow processing to be performed by changing the output format for the luminance component and the chrominance component of the output data on a slice-by-slice or picture-by-picture basis.

[0055] Furthermore, the encoding method of Example 19 is an encoding method in which, for each of the multiple neural network loop filter models, a data unit containing a parameter set relating to the neural network loop filter model, which can be composed solely of the parameter set, is encoded, and a first identifier for identifying the multiple neural network loop filter models is encoded in the picture header or slice header.

[0056] This may allow the decoder to transmit to the decoder that the neural network loop filter model is switching for each slice, sub-picture, picture, or scene, using a dedicated data unit that can be composed solely of the parameter set for the neural network loop filter model. Furthermore, since a dedicated data unit is used for each neural network loop filter model, it may be possible to transmit the switching of the neural network loop filter model to the decoder with greater flexibility.

[0057] Furthermore, the decoding method of Example 20 is a decoding method in which, for each of the multiple neural network loop filter models, a data unit containing a parameter set relating to the neural network loop filter model, and which can be composed solely of the parameter set, is decoded, and a first identifier for identifying the multiple neural network loop filter models is decoded from the picture header or slice header.

[0058] This may allow switching between neural network loop filter models on a slice-by-slice, sub-picture-by-subpicture, picture-by-picture, or scene-by-scene basis, using a dedicated data unit composed solely of parameter sets related to the neural network loop filter model. Furthermore, since a dedicated data unit is used for each neural network loop filter model, it may be possible to switch between neural network loop filter models more flexibly.

[0059] Furthermore, these comprehensive or specific embodiments may be implemented as systems, devices, methods, integrated circuits, computer programs, or non-temporary recording media such as computer-readable CD-ROMs, or as any combination of systems, devices, methods, integrated circuits, computer programs, and recording media.

[0060] [Definition of Terms] Each term may be defined as follows, for example:

[0061] (1) Image: A unit of data composed of a collection of pixels, consisting of pictures and smaller blocks, and includes both still images and videos.

[0062] (2) Chroma Chroma represents a sample sequence or single sample that represents one of two color difference signals. The two color difference signals may be represented by the symbols Cb and Cr. Chroma is an adjective, for example, represented by the symbols Cb and Cr, or U and V. The term chrominance can also be used instead of chroma.

[0063] (3) Luminance (luma) Luminance refers to a sample sequence or a single sample representing a monochrome signal associated with the primary colors. Luminance is an adjective represented by the symbols Y or L. The term luminance can also be used instead of lumina.

[0064] (4) Component A component represents a single array or sample. A component may be, for example, one of the three arrays (i.e., luminance and two color differences) that make up a color format picture, or a single sample, or one array or a single sample that makes up a monochrome format picture.

[0065] (5) A picture is an image processing unit composed of a set of picture samples, and is sometimes called a frame or field. A picture can be a set of luminance samples, a set of two color difference samples, or a set of both luminance samples and a set of two color difference samples. A set of samples can also be described as a matrix of samples.

[0066] (5-1) I-Picture An I-picture is a picture encoded using only intra-prediction. I-pictures can be decoded independently. I-pictures are also called intra-pictures, intra-frames, keypictures, or keyframes.

[0067] (5-2) P-Picture A P-Picture is a picture encoded using single prediction. In other words, only one picture is referenced to encode the P-Picture. Single prediction is also called unidirectional prediction.

[0068] (5-3) B-Picture A B-picture is a picture encoded using bidirectional prediction. Bidirectional prediction is also expressed as dual prediction and means a prediction that references multiple pictures. Bidirectional prediction may refer to multiple pictures in different directions from the picture being processed (for example, a forward picture and a backward picture in different temporal directions), or it may refer to pictures in the same direction from the picture being processed.

[0069] Note that P-pictures and B-pictures are also called inter-pictures or inter-frames.

[0070] (6) A block is a processing unit of a set containing a specific number of pixels, and its name is not restricted, as shown in the following examples. It is also not restricted in shape, and includes not only rectangles made up of M x N pixels and squares made up of M x M pixels, but also other shapes.

[0071] (Examples of blocks) Slice / Tile / Brick CTB (Coding Tree Block) / CTU (Coding Tree Unit) / Superblock / Segment / Basic division unit / CU / Processing block unit / Prediction block unit (PU) / Translation block unit (TU) / Unit / Subblock / VPDU / Hardware processing division unit

[0072] A CTB is a processing unit into which components are divided. A CTB contains samples. A CTB is also called a CTU, superblock, or basic division unit. A CTB may be, for example, an N x N square block.

[0073] A Coding Block (CB) is a processing unit obtained by dividing a Coding Block (CTB). For example, it may be an M x N rectangular block. A CB is also called a Coding Unit (CU).

[0074] (7) A pixel / sample is the smallest unit of a picture, and includes not only pixels at integer positions but also pixels at decimal positions that are generated based on pixels at integer positions. A pixel / sample is a fundamental element that makes up a picture.

[0075] (8) Pixel value / sample value: An intrinsic value of a pixel, which includes not only luminance value, chrominance value, and RGB gradation, but also depth value, or binary values ​​of 0 or 1.

[0076] (9) Flags A flag is a variable or a single-bit syntax element. For example, a flag can only take one of two values: 0 or 1.

[0077] (10) A symbol or code used to transmit signal information, which includes not only discretized digital signals but also analog signals that take continuous values.

[0078] (11) Stream / Bitstream: Refers to a sequence of digital data or a flow of digital data. A stream / bitstream may consist of a single stream, or it may be divided into multiple layers and composed of multiple streams. It also includes cases where data is transmitted via serial communication over a single transmission path, as well as cases where data is transmitted via packet communication over multiple transmission paths.

[0079] (12) In the case of difference / difference scalar quantities, it is sufficient that the difference operation is included in addition to the simple difference (x - y), and this includes the absolute value of the difference (|x - y|), the squared difference (x^2 - y^2), the square root of the difference (√(x - y)), the weighted difference (ax - by: a, b is a constant), and the offset difference (x - y + a: a is the offset). Note that the difference operation also includes cases where bit shifts are performed (x >> 1 - y >> 1).

[0080] (13) In the case of summation scalar quantities, it is sufficient that the sum operation is included in addition to the simple sum (x + y), and this includes the absolute value of the sum (|x + y|), the sum of squares (x^2 + y^2), the square root of the sum (√(x + y)), weighted sum (ax + by: a, b is a constant), and offset sum (x + y + a: a is the offset). Note that the sum operation also includes cases where bit shifts are performed (x >> 1 + y >> 1).

[0081] (14) Based on: This includes cases where factors other than the subject being based on are taken into consideration. It also includes cases where the result is obtained not only by obtaining a direct result, but also by obtaining an intermediate result.

[0082] (15) Using: This includes cases where elements other than the target of use are taken into consideration. It also includes cases where the result is obtained not only by obtaining the result directly, but also by obtaining the result via an intermediate result.

[0083] (16) Prohibit, forbid. This can be rephrased as not being allowed. Also, not prohibiting or allowing something does not necessarily mean it is an obligation.

[0084] (17) To restrict (limit, restriction / restrict / restricted) This can be rephrased as not being allowed. Also, not being prohibited or being permitted does not necessarily mean being obligated. Furthermore, it is sufficient if it is prohibited in part quantitatively or qualitatively, and it also includes cases where it is prohibited entirely.

[0085] (18) MV (motion vector) A two-dimensional vector used in interpretation, which is a value that indicates the offset from the coordinates of the decoded image to the coordinates of the reference image.

[0086] (19) Data assigned to a specific column or row in a database, such as an index table or list. For example, a reference index is an index for a reference picture list, where one reference picture is assigned to the reference picture index.

[0087] [Explanation of Descriptions] In the drawings, the same reference number indicates the same or similar component. Also, the size and relative position of components in the drawings are not necessarily depicted to a constant scale.

[0088] The embodiments will be described in detail below with reference to the drawings. Note that the embodiments described below are all general or specific examples. The numerical values, shapes, materials, components, arrangement and connection forms of components, steps, relationships and sequences of steps shown in the following embodiments are examples only and are not intended to limit the scope of the claims.

[0089] The following describes embodiments of encoding and decoding devices. These embodiments are examples of encoding and decoding devices to which the processes and / or configurations described in each aspect of this disclosure can be applied. The processes and / or configurations can also be implemented in encoding and decoding devices different from those in the embodiments. For example, with respect to the processes and / or configurations applicable to the embodiments, one of the following may be implemented:

[0090] (1) Any of the multiple components of the encoding or decoding device of the embodiments described in each aspect of the present disclosure may be replaced or combined with other components described in any of the aspects of the present disclosure.

[0091] (2) In the encoding or decoding device of the embodiment, any changes such as addition, replacement, or deletion of functions or processes performed by some of the multiple components of the encoding or decoding device may be made. For example, any of the functions or processes may be replaced or combined with other functions or processes described in any of the embodiments of this disclosure.

[0092] (3) In the methods performed by the encoding or decoding apparatus of the embodiment, any modifications such as additions, replacements, and deletions may be made to some of the processes included in the method. For example, any of the processes in the method may be replaced with or combined with other processes described in any of the embodiments of this disclosure.

[0093] (4) Some of the multiple components constituting the encoding or decoding device of the embodiment may be combined with components described in any of the embodiments of this disclosure, or with components that have some of the functions described in any of the embodiments of this disclosure, or with components that perform some of the processing performed by the components described in any of the embodiments of this disclosure.

[0094] (5) Components that provide some of the functions of the encoding or decoding device of the embodiment, or components that perform some of the processing of the encoding or decoding device of the embodiment, may be combined with or replaced with components described in any of the embodiments of this disclosure, components that provide some of the functions described in any of the embodiments of this disclosure, or components that perform some of the processing described in any of the embodiments of this disclosure.

[0095] (6) In a method performed by an encoding or decoding device of an embodiment, any of the processes included in the method may be replaced or combined with a process described in any of the embodiments of the present disclosure, or any of the similar processes.

[0096] (7) Some of the processes included in the methods performed by the encoding or decoding device of the embodiment may be combined with the processes described in any of the embodiments of this disclosure.

[0097] (8) The methods of carrying out the processes and / or configurations described in each aspect of the present disclosure are not limited to the encoding or decoding devices of the embodiments. For example, the processes and / or configurations may be carried out in devices used for purposes other than the video encoding or video decoding disclosed in the embodiments.

[0098] [System Configuration] Figure 1 is a schematic diagram showing an example of the configuration of the transmission system according to this embodiment.

[0099] A transmission system Trs is a system that transmits a stream generated by encoding an image and decodes the transmitted stream. Such a transmission system Trs includes, for example, an encoding device 100, a network Nw, and a decoding device 200, as shown in Figure 1.

[0100] An image is input to the encoding device 100. The encoding device 100 generates a stream by encoding the input image and outputs the stream to the network Nw. The stream includes, for example, the encoded image and control information for decoding the encoded image. The image is compressed by this encoding process.

[0101] The original image input to the encoding device 100 before encoding is also called the original image, original signal, or original sample. The image may be a moving image or a still image. Furthermore, the image is a higher-level concept than sequences, pictures, and blocks, and is not limited by spatial and temporal domains unless otherwise specified. The image consists of a sequence of pixels or pixel values, and the signal or pixel values ​​representing the image are also called samples.

[0102] Furthermore, the stream may also be called a bitstream, encoded bitstream, compressed bitstream, or encoded signal. In addition, the encoding device 100 may be called an image encoding device or a video encoding device, and the encoding method by the encoding device 100 may be called an encoding method, an image encoding method, or a video encoding method.

[0103] The network Nw transmits the stream generated by the encoding device 100 to the decoding device 200. The network Nw may be the Internet, a wide area network (WAN), a local area network (LAN), or a combination thereof. The network Nw is not necessarily limited to a bidirectional communication network; it may also be a unidirectional communication network that transmits broadcast waves such as terrestrial digital broadcasting or satellite broadcasting.

[0104] Furthermore, the network may be replaced by a storage medium that records streams, such as a DVD (Digital Versatile Disc) or a BD (Blu-Ray Disc®).

[0105] The decoding device 200 generates a decoded image, which is, for example, an uncompressed image, by decoding the stream transmitted by the network Nw. For example, the decoding device decodes the stream according to a decoding method that corresponds to the encoding method by the encoding device 100.

[0106] The decoding device 200 may also be called an image decoding device or a video decoding device, and the decoding method performed by the decoding device 200 may be called a decoding method, an image decoding method, or a video decoding method.

[0107] [Encoding device] Next, the encoding device 100 according to the embodiment will be described.

[0108] [Implementation Example of Encoding Device] Figure 2 is a block diagram showing an implementation example of the encoding device 100. The encoding device 100 includes a processor a1 and a memory a2. For example, the multiple components of the encoding device 100 shown in Figure 3, which will be described later, are implemented by the processor a1 and memory a2 shown.

[0109] Processor a1 is a circuit that performs information processing and is a circuit that can access memory a2. For example, processor a1 is a dedicated or general-purpose electronic circuit for encoding images. Processor a1 may be a processor such as a CPU. Alternatively, processor a1 may be a collection of multiple electronic circuits. Furthermore, for example, processor a1 may play the role of multiple components of the encoding device 100 shown in Figure 3, which will be described later, excluding the component for storing information.

[0110] Memory a2 is a dedicated or general-purpose memory in which information for the processor a1 to encode an image is stored. Memory a2 may be an electronic circuit and may be connected to the processor a1. Memory a2 may also be included in the processor a1. Memory a2 may also be a collection of multiple electronic circuits. Memory a2 may also be a magnetic disk or an optical disk, or may be described as storage or a recording medium. Memory a2 may also be a non-volatile memory or a volatile memory.

[0111] For example, memory a2 may store the image to be encoded, or it may store a stream corresponding to the encoded image. Alternatively, memory a2 may store a program for processor a1 to encode the image.

[0112] Furthermore, for example, memory a2 may play the role of an information storage component among the multiple components of the encoding device 100 shown in Figure 3, which will be described later. Specifically, memory a2 may play the role of the block memory 118 and frame memory 122 shown in Figure 3, which will be described later. More specifically, memory a2 may store a reconstructed image (specifically, a reconstructed block or a reconstructed picture, etc.).

[0113] The generated parameters disclosed in the embodiments may be stored in memory a2 and referenced in the processing of processor a1. The generated parameters may be stored in memory a2 and may or may not be encoded. Whether or not to encode them is determined appropriately based on the relationship between the increase in the amount of code due to encoding the parameters and the reduction in the processing load of the decoding device 200 due to receiving the parameters.

[0114] Furthermore, in the encoding device 100, not all of the multiple components shown in Figure 3 described later are to be implemented, nor are all of the multiple processes described above to be performed. Some of the multiple components shown in Figure 3 described later may be included in other devices, and some of the multiple processes described later may be performed by other devices.

[0115] The following is a general description of the encoding device 100, followed by a description of the components included in the encoding device 100.

[0116] [Example of Encoding Device Configuration] Figure 3 is a block diagram showing an example of the configuration of an encoding device 100 according to an embodiment. The encoding device 100 encodes images in block units.

[0117] As shown in Figure 3, the encoding device 100 is a device that encodes an image in block units and comprises a division unit 102, a subtraction unit 104, a transformation unit 106, a quantization unit 108, an entropy encoding unit 110, an inverse quantization unit 112, an inverse transformation unit 114, an addition unit 116, a block memory 118, a loop filter unit 120, a frame memory 122, and a prediction unit. The prediction unit comprises an intra-prediction unit 124, an inter-prediction unit 126, a prediction control unit 128, and a prediction parameter generation unit 130.

[0118] These components are implemented, for example, by the processor a1 and memory a2 of the encoding device 100 shown in Figure 2 above.

[0119] [Overall Encoding Process Flow] Figure 4A is a flowchart showing an example of the overall encoding process by the encoding device 100. Figure 4B is a flowchart showing an example of the block-by-block processing by the encoding device 100.

[0120] First, the division unit 102 of the encoding device 100 divides each picture contained in the original image into multiple blocks. For example, the division unit 102 first divides the picture into blocks of a fixed size (for example, 128 x 128 pixels) (step Sa_1a). These fixed-size blocks are sometimes called coding tree units (CTUs).

[0121] Then, the division unit 102 selects a division pattern for the fixed-size block and further divides the fixed-size block into multiple blocks to constitute the selected division pattern (step Sa_2a).

[0122] The further divided blocks are of a variable size, for example, 64 x 64 pixels or less. The vertical and horizontal pixel counts of the variable size can be any combination of 4, 8, 16, 32, or 64. These variable-sized blocks are sometimes called coding units (CUs), prediction units (PUs), or transformation units (TUs).

[0123] In various implementation examples, CU, PU, ​​and TU do not need to be distinguished, and some or all blocks within a picture may be processing units for CU, PU, ​​or TU.

[0124] Then, the encoding device 100 performs the processing shown in steps Sa_3 to Sa_11 in Figure 4B for each of the multiple blocks (step Sa_3a). Here, an example of a CTU size of 128 x 128 pixels is given, but other sizes are also possible. For example, the size of the CTU may be 256 x 256 pixels, or a larger size such as 512 x 512 pixels. In this case, the vertical and horizontal size of the divided blocks, i.e., the number of pixels, may be greater than 64 pixels, such as 128 pixels or 256 pixels.

[0125] Furthermore, the division unit 102 outputs parameters indicating the division pattern to the conversion unit 106, the inverse conversion unit 114, the intra-prediction unit 124, the inter-prediction unit 126, and the entropy coding unit 110. The conversion unit 106 may convert the prediction residuals based on these parameters, and the intra-prediction unit 124 and the inter-prediction unit 126 may generate a prediction image based on these parameters. The entropy coding unit 110 may also perform entropy coding on these parameters.

[0126] Then, the encoding device 100 processes each of the multiple blocks (step Sa_3a). Specifically, the encoding device 100 processes steps Sa_3 to Sa_11 as shown in Figure 4B. Then, the encoding device 100 determines whether or not the encoding of the entire picture is complete (step Sa_4a), and if it determines that it is not complete (No. in step Sa_4a), it repeats the processing from step Sa_2a.

[0127] Next, the processing for each block will be explained (Sa_3 to Sa_11 in Figure 4B). First, the prediction unit generates a predicted image of the current block (step Sa_3). Specifically, the prediction unit generates a predicted image of the current block by referring to a reconstructed image generated by encoding and then decoding other blocks. The predicted image is also called a predicted signal, predicted block, or predicted sample.

[0128] The reconstructed image may be, for example, the image of the reference picture, or it may be the image of an encoded block (i.e., the other block mentioned above) within the current picture, which is the picture containing the current block. An encoded block within the current picture is, for example, an adjacent block to the current block.

[0129] The predicted image is, for example, an intra-prediction image (intra-prediction signal) generated based on intra-prediction, or an inter-prediction image (inter-prediction signal) generated based on inter-prediction.

[0130] Intra prediction is a method of predicting the current block by referring to blocks within the current picture, and is also called in-screen prediction. Specifically, the intra prediction unit 124 performs intra prediction by referring to the pixel values ​​(e.g., luminance values ​​or chrominance values, etc.) of blocks adjacent to the current block that are included in the current picture stored in the block memory 118. As a result, the intra prediction unit 124 generates an intra prediction image and outputs the intra prediction image to the prediction control unit 128.

[0131] Inter-prediction is a method of predicting the current block by referring to a reference picture different from the current picture, and is also called inter-screen prediction. Specifically, the inter-prediction unit 126 generates an inter-predicted image by referring to a reference picture stored in the frame memory 122 and performing inter-prediction of the current block, and outputs the inter-predicted image to the prediction control unit 128.

[0132] Note that prediction processing using the reconstructed image may not be performed on some blocks, such as the first block of the first image to be encoded. In that case, the original image will be output in the next subtraction process without performing subtraction on the original image. The interpretation unit 126 may also generate a predicted image by performing prediction processing without using a reference image.

[0133] Next, the subtraction unit 104 subtracts the predicted image (the predicted image input from the prediction control unit 128) from the original image in block units that are input from the division unit 102 and divided by the division unit 102. In other words, the subtraction unit 104 generates the difference between the current block and the predicted image as the predicted residual (step Sa_4).

[0134] The prediction residual is also called the prediction error. The original image is the input signal to the encoding device 100, and is, for example, a signal representing the image of each picture that makes up the video (e.g., a luminance (luma) signal and two chroma (chroma) signals). If the prediction process is skipped, the prediction residual becomes the value of the original image. For example, the prediction process is skipped for the first block in the processing order.

[0135] Next, the conversion unit 106 applies a conversion process to the predicted residual to generate conversion coefficients and outputs the conversion coefficients to the quantization unit 108 (step Sa_5).

[0136] The transformation process performed by the transformation unit 106 is, for example, an orthogonal transformation such as a discrete cosine transform (DCT) or discrete sine transform (DST) that transforms the predicted residual in the spatial domain into transformation coefficients in the frequency domain. However, other transformation processes such as weblet transforms, non-orthogonal transforms, or transformation processes expressed by matrix operations such as NSST (non-separable secondary transform) as defined in VVC may also be used.

[0137] The conversion unit 106 may perform a conversion process selected from among a plurality of candidate conversion processes. The conversion unit 106 may output conversion coefficients generated by sequentially applying a plurality of conversion processes to the predicted residual, or it may output the predicted residual as is without performing any conversion processes.

[0138] Depending on the processing performed by the conversion unit 106, information indicating whether or not a conversion process is applied to the predicted residuals, and / or information indicating the conversion type, may be encoded in the bitstream. This information is, for example, signaled at the CU level, but is not limited to the CU level and may be encoded at other levels (e.g., sequence level, picture level, slice level, brick level, or CTU level).

[0139] The quantization unit 108 quantizes the conversion coefficients output from the conversion unit 106 (step Sa_6). Specifically, the quantization unit 108 quantizes the conversion coefficients based on the quantization parameter (QP) corresponding to the conversion coefficients. The quantization unit 108 then outputs a plurality of quantized conversion coefficients (hereinafter referred to as quantization coefficients) of the current block to the entropy coding unit 110 and the inverse quantization unit 112. In other words, the quantization unit 108 generates quantization coefficients and outputs them to the entropy coding unit 110 and the inverse quantization unit 112.

[0140] The quantization parameter (QP) is a parameter that defines the quantization step (quantization width). For example, if the value of the quantization parameter increases, the quantization step also increases. In other words, if the value of the quantization parameter increases, the error in the quantization coefficient (quantization error) increases.

[0141] Furthermore, the quantization unit 108 may quantize the conversion coefficients based on a quantization matrix. In other words, quantization parameters (QP) and / or a quantization matrix may be used for quantization. The quantization parameters (QP) and the quantization matrix may be encoded, for example, at the sequence level, picture level, slice level, brick level, or CTU level.

[0142] The quantization unit 108 performs quantization processing in a predetermined scanning order. This predetermined scanning order is the order for quantization / inverse quantization of the conversion coefficients. For example, the predetermined scanning order is defined as ascending order of frequency (from low frequency to high frequency) or descending order (from high frequency to low frequency).

[0143] Next, the entropy coding unit 110 generates a stream (step Sa_7) by coding (specifically, entropy coding) the plurality of quantization coefficients and the prediction parameters related to the generation of the predicted image. For this entropy coding, for example, CABAC (Context-based Adaptive Binary Arithmetic Coding), which is used in VVC, may be used. Alternatively, a coding method that is a modified version of CABAC used in VVC may be used.

[0144] Furthermore, the processing of the entropy encoding unit 110 does not necessarily have to be performed after the quantization process; for example, it may be performed outside the loop of processing for each block.

[0145] Next, the inverse quantization unit 112 and the inverse transformation unit 114 restore the predicted residuals by performing inverse quantization and inverse transformation on a plurality of quantization coefficients (steps Sa_8 and Sa_9).

[0146] Next, the summing unit 116 reconstructs the current block by adding the predicted image to the recovered predicted residual (step Sa_10). This generates a reconstructed image. The reconstructed image is also called a reconstructed block, and the reconstructed image generated by the encoding device 100 is also called a local decoded block or local decoded image.

[0147] Next, the loop filter unit 120 performs filtering on the reconstructed image as needed (step Sa_11). Specifically, the loop filter unit 120 applies loop filtering to the reconstructed image output from the adder unit 116 and outputs the filtered reconstructed image to the frame memory 122.

[0148] Loop filters are filters used to reduce block noise that occurs at block boundaries, and include, for example, adaptive loop filters (ALF), deblocking filters (DF or DBF), and sample adaptive offset (SAO). Here, loop filters are described as in-loop filters used within the coding loop, but loop filters may also be out-loop filters used outside the coding loop.

[0149] In the example described above, the encoding device 100 selects one division pattern for a fixed-size block and encodes each block according to that division pattern. However, it may also encode each block according to multiple division patterns. In this case, the encoding device 100 may evaluate the cost of each of the multiple division patterns and, for example, select the stream obtained by encoding according to the division pattern with the smallest cost as the final output stream.

[0150] Furthermore, the processes in steps Sa_1a to Sa_4a and Sa_3 to Sa_11 may be performed sequentially by the encoding device 100, some of these processes may be performed in parallel, and the order may be changed.

[0151] The encoding process performed by such an encoding device 100 is a hybrid encoding using predictive encoding and transformative encoding. Furthermore, predictive encoding is performed by an encoding loop consisting of a subtraction unit 104, a transformer unit 106, a quantization unit 108, an inverse quantization unit 112, an inverse transformer unit 114, an addition unit 116, a loop filter unit 120, a block memory 118, a frame memory 122, an intra-prediction unit 124, an inter-prediction unit 126, and a prediction control unit 128. In other words, the prediction processing unit consisting of the intra-prediction unit 124 and the inter-prediction unit 126 constitutes a part of the encoding loop.

[0152] [Decoding Device] Next, a decoding device 200 capable of decoding the stream output from the encoding device 100 will be described.

[0153] [Implementation Example of Decryption Device] Figure 5 is a block diagram showing an implementation example of the decoding device 200. The decoding device 200 includes a processor b1 and memory b2. For example, the multiple components of the decoding device 200 shown in Figure 6, which will be described later, are implemented by the processor b1 and memory b2 shown in Figure 5.

[0154] Processor b1 is a circuit that performs information processing and is a circuit that can access memory b2. For example, processor b1 is a dedicated or general-purpose electronic circuit for decoding streams. Processor b1 may be a processor such as a CPU. Alternatively, processor b1 may be a collection of multiple electronic circuits. Furthermore, for example, processor b1 may play the role of multiple components of the decoding device 200 shown in Figure 6, etc., described later, excluding the component for storing information.

[0155] Memory b2 is a dedicated or general-purpose memory in which information for the processor b1 to decode the stream is stored. Memory b2 may be an electronic circuit and may be connected to the processor b1. Memory b2 may also be included in the processor b1. Memory b2 may also be a collection of multiple electronic circuits. Memory b2 may also be a magnetic disk or an optical disk, or may be described as storage or a recording medium. Memory b2 may also be a non-volatile memory or a volatile memory.

[0156] For example, memory b2 may store an image or a stream. Alternatively, memory b2 may store a program for processor b1 to decode the stream.

[0157] Furthermore, for example, memory b2 may play the role of an information storage component among the multiple components of the decoding device 200 shown in Figure 6 below. Specifically, memory b2 may play the role of a block memory 210 and a frame memory 214 shown in Figure 6 below. More specifically, memory b2 may store a reconstructed image (specifically, a reconstructed block or a reconstructed picture, etc.).

[0158] The generated parameters disclosed in the embodiments may be stored in memory b2 and referenced in the processing of processor b1. The parameters may be generated by a decoding process or not.

[0159] Furthermore, the decoding device 200 does not need to implement all of the components shown in Figure 6, etc., described later, nor does it need to perform all of the processes described above. Some of the components shown in Figure 6, etc., described later, may be included in other devices, and some of the processes described above may be performed by other devices.

[0160] The following describes the decoding device 200 in general, followed by a description of its components. Note that detailed explanations may be omitted for components of the decoding device 200 that perform the same processing as those in the encoding device 100.

[0161] [Example of Decoding Device Configuration] Figure 6 is a block diagram showing an example of the configuration of a decoding device 200 according to an embodiment. The decoding device 200 is a device that decodes a stream, which is an encoded image, in block units.

[0162] As shown in Figure 6, the decoding device 200 includes an entropy decoding unit 202, an inverse quantization unit 204, an inverse transformation unit 206, an addition unit 208, a block memory 210, a loop filter unit 212, a frame memory 214, a prediction unit, a prediction control unit 220, a prediction parameter generation unit 222, and a division determination unit 224. The prediction unit includes an intra-prediction unit 216 and an inter-prediction unit 218.

[0163] These components are implemented, for example, by the processor b1 and memory b2 of the decoding device 200 shown in Figure 5 above.

[0164] Furthermore, components included in the decoding device 200 may perform the same processing as components included in the encoding device 100, and their explanation may be omitted. For example, the inverse quantization unit 204, inverse transform unit 206, adder unit 208, block memory 210, frame memory 214, intra prediction unit 216, inter prediction unit 218, prediction control unit 220, and loop filter unit 212 perform the same processing as the inverse quantization unit 112, inverse transform unit 114, adder unit 116, block memory 118, frame memory 122, intra prediction unit 124, inter prediction unit 126, prediction control unit 128, and loop filter unit 120, respectively.

[0165] [Overall Decryption Process Flow] Figure 7A is a flowchart showing an example of the overall decryption process by the decryption device 200. Figure 7B is a flowchart showing an example of the block-by-block processing by the decryption device 200.

[0166] First, the entropy decoding unit 202 of the decoding device 200 acquires the stream output from the encoding device 100. Then, the entropy decoding unit 202 decodes (specifically, entropy decodes) the encoded quantization coefficients and prediction parameters of the current block contained in the bitstream (step Sp_1a).

[0167] The division determination unit 224 determines the division pattern for each of the multiple fixed-size blocks (128 x 128 pixels) contained in the picture based on the parameters input from the entropy decoding unit 202 (step Sp_2a). This division pattern is the division pattern selected by the encoding device 100. The decoding device 200 then performs block-specific processing for each of the multiple blocks. Specifically, it performs the processing in steps Sp_1 to Sp_5 for each of the multiple blocks (step Sp_3a).

[0168] The decoding device 200 then determines whether or not the decoding of the entire picture is complete (step Sp_4a). If it determines that it is not complete (No. in step Sp_4a), it repeats the process from step Sp_2a.

[0169] Next, we will explain the processing for each block (Sp_1 to Sp_5 in Figure 7B).

[0170] First, the inverse quantization unit 204 inversely quantizes the quantization coefficients of the current block, which are input from the entropy decoding unit 202. Specifically, for each of the quantization coefficients of the current block, the inverse quantization unit 204 inversely quantizes the quantization coefficient based on the quantization parameter corresponding to that quantization coefficient (step Sp_1).

[0171] For example, the inverse quantization unit 204 may acquire a parameter indicating whether or not to perform inverse quantization, and quantization parameters (such as difference quantization parameters and QP index). The inverse quantization unit 204 may then decide whether or not to perform inverse quantization based on the acquired parameters, and may perform the inverse quantization process if it is decided to perform inverse quantization.

[0172] The inverse quantization unit 204 then outputs the inversely quantized quantization coefficients (i.e., transformation coefficients) of the current block to the inverse transformation unit 206.

[0173] Next, the inverse transform unit 206 restores the predicted residual by inversely transforming the transformation coefficients, which are input from the inverse quantization unit 204 (step Sp_2). Here, inverse transform refers to the reverse transformation process of the transformation process described in the encoding device 100. In other words, the inverse transform unit 206 can also be said to be performing a transformation process.

[0174] Next, the prediction unit, consisting of the intra-prediction unit 216, the inter-prediction unit 218, and the prediction control unit 220, generates a predicted image of the current block (step Sp_3). Here, the intra-prediction unit 216 and the inter-prediction unit 218 may perform the same processing as described above on the encoding device 100. Note that the predicted image needs to be generated before the addition process because it is used in the subsequent addition process, but it does not necessarily have to be done after the prediction residual restoration process. For example, the prediction image generation process may be performed in parallel with the prediction residual restoration process.

[0175] Next, the summing unit 208 reconstructs the current block into a reconstructed image (also called a decoded image block) by adding the predicted image to the predicted residual (step Sp_4). In other words, the summing unit 208 generates a reconstructed image of the current block by adding the predicted image to the predicted residual. The summing unit 208 then outputs the reconstructed image of the current block to the block memory 210 and the loop filter unit 212.

[0176] The block memory 210 is a block referenced in intra prediction and is a storage unit for storing blocks within the current picture. Specifically, the block memory 210 stores the reconstructed image output from the adder 208.

[0177] The loop filter unit 212 performs filtering on the reconstructed image (step Sp_5). The filtered reconstructed image is output to the frame memory 214 and the display device, etc.

[0178] In Figures 6 and 7B, the loop filter unit 212 processes the reconstructed image within the loop; that is, it outputs the filtered reconstructed image to the frame memory 214. Some or all of the processing in the loop filter unit 212 may be performed outside the loop. That is, filtering may be performed before outputting to a display device or the like.

[0179] Furthermore, the processes in steps Sp_1a to Sp_4a and Sp_1 to Sp_5 may be performed sequentially by the decoding device 200, some of these processes may be performed in parallel, or the order of these processes may be changed.

[0180] [Another Example of System Configuration] Figure 8 is a schematic diagram showing another example of the configuration of the transmission system according to this embodiment. The transmission system Trs is a system that transmits generated streams and receives transmitted streams. Such a transmission system Trs includes, for example, a transmitting device 2000, a network Nw, and a receiving device 3000, as shown in Figure 8. Note that the transmission system Trs does not need to have all of these components and may consist of only some of the devices.

[0181] The network Nw may be the Internet, a wide-area network (WAN), a local area network (LAN), or a combination thereof. The network Nw is not necessarily limited to a bidirectional communication network; it may also be a unidirectional communication network transmitting broadcast waves such as terrestrial digital broadcasting or satellite broadcasting. Furthermore, the network Nw may be replaced by a storage medium that records streams, such as a DVD (Digital Versatile Disc) or a BD (Blu-Ray Disc®).

[0182] The transmitting device 2000 transmits the stream over the network Nw. The transmitting device 2000 may also take the encoded stream as input and output the acquired stream. Alternatively, the transmitting device 2000 may have the encoding processing configuration disclosed as the encoding device 100, take the original image before encoding as input, encode the acquired image to generate a stream, and transmit it.

[0183] When the transmitting device 2000 generates a stream, the original input image before encoding is also called the original image, original signal, or original sample. The image may be a moving image or a still image. The stream includes, for example, an encoded image and control information for decoding that encoded image. This encoding compresses the image.

[0184] The transmitting device 2000 can reduce the load on stream transmission by reducing the amount of data in the stream, which includes the encoded image.

[0185] The receiving device 3000 receives an encoded stream from the network Nw. The receiving device 3000 may store the received stream in memory, transfer the received stream to another device, or perform any processing on the received stream. For example, the receiving device 3000 may have a decoding process configuration disclosed as the decoding device 200 and decode the received stream.

[0186] The receiving device 3000 can reduce the delay of the stream by reducing the amount of data in the received stream.

[0187] [Transmitting Device] Figure 9 is a schematic diagram showing an example configuration of the transmitting device 2000 according to this embodiment. The transmitting device 2000 is a device that transmits a stream.

[0188] [Implementation Example of Transmitting Device] Next, a transmitting device 2000 according to the embodiment will be described. Figure 10 is a block diagram showing an implementation example of the transmitting device 2000. The transmitting device 2000 includes a processor a1 and a transmitting unit a3 that outputs a stream. The transmitting device 2000 may also further include a memory a2 as shown in Figure 2, or a receiving unit that acquires a video or stream.

[0189] Processor a1 is a circuit that performs information processing. Processor a1 may also be a circuit that can access memory a2. For example, processor a1 is a dedicated or general-purpose electronic circuit that transmits a stream. Processor a1 may also be a processor such as a CPU. Alternatively, processor a1 may be a collection of multiple electronic circuits.

[0190] Memory a2 is a dedicated or general-purpose memory in which information for the processor a1 to transmit a stream is stored. Memory a2 may be an electronic circuit and may be connected to the processor a1. Memory a2 may also be included in the processor a1. Memory a2 may also be a collection of multiple electronic circuits. Memory a2 may also be a magnetic disk or an optical disk, or may be described as storage or a recording medium. Memory a2 may also be a non-volatile memory or a volatile memory.

[0191] For example, memory a2 may store the input image or the generated stream. Memory a2 may also store a program for processor a1 to send the stream.

[0192] The stream transmitted by the transmitting device 2000 may be the stream (also known as a bitstream) described in this disclosure.

[0193] Furthermore, in the transmitting device 2000, the processor a1 may perform some or all of the roles of the transmitting unit a3. Alternatively, the transmitting device 2000 may not have a transmitting unit a3, and the processor a1 may output the stream.

[0194] [Receiving device] Figure 11 is a schematic diagram showing an example of the configuration of a receiving device 3000 according to this embodiment. The receiving device 3000 is a device that receives a stream.

[0195] [Implementation Example of Receiving Device] Next, a receiving device 3000 according to the embodiment will be described. Figure 12 is a block diagram showing an implementation example of the receiving device 3000. The receiving device 3000 includes a processor b1 and a receiving unit b4 that receives a stream. The receiving device 3000 may also further include a memory b2 as shown in Figure 5, or a transmitting unit that transmits a video or stream.

[0196] Processor b1 is a circuit that performs information processing. Processor b1 may also be a circuit that can access memory b2. For example, processor b1 is a dedicated or general-purpose electronic circuit that receives a stream. Processor b1 may also be a processor such as a CPU. Alternatively, processor b1 may be a collection of multiple electronic circuits.

[0197] Memory b2 is a dedicated or general-purpose memory that stores information for processor b1 to receive streams. Memory b2 may be an electronic circuit and may be connected to processor b1. Memory b2 may also be included in processor b1. Memory b2 may also be a collection of multiple electronic circuits. Memory b2 may also be a magnetic disk or an optical disk, or may be described as storage or a recording medium. Memory b2 may also be a non-volatile memory or a volatile memory.

[0198] For example, memory b2 may store the input stream or the decoded image. Alternatively, memory b2 may store a program for processor b1 to receive the stream.

[0199] The stream received by the receiving device 3000 may be the stream (also called a bitstream) described in this disclosure.

[0200] Furthermore, in the receiving device 3000, the processor b1 may perform some or all of the functions of the receiving unit b4. Alternatively, the receiving device 3000 may not have a receiving unit b4, and the processor b1 may receive the stream.

[0201] [Bitstream Generation Device] Figure 13 is a schematic diagram showing an example of the configuration of a bitstream generation device according to this embodiment. The bitstream generation device 1000 is a device that generates a bitstream and transmits the generated stream.

[0202] An image is input to the bitstream generator 1000. The bitstream generator 1000 generates a stream by encoding the input image and outputs the stream. The stream includes, for example, the encoded image and control information for decoding the encoded image. The image is compressed by this encoding process.

[0203] The original image input to the bitstream generator 1000 before encoding is also called the original image, original signal, or original sample. The image may be a moving image or a still image.

[0204] Furthermore, an image is a broader concept than sequences, pictures, and blocks, and is not limited to spatial or temporal domains unless otherwise specified. An image consists of a sequence of pixels or pixel values, and the signal or pixel values ​​representing that image are also called samples. A stream may also be called a bitstream, encoded bitstream, compressed bitstream, or encoded signal.

[0205] Furthermore, the bitstream generation device may also be called the encoding device, image encoding device, or video encoding device, and the method for generating a bitstream by the bitstream generation device may also be called the encoding method, image encoding method, or video encoding method.

[0206] [Implementation Example of Bitstream Generation Device] Next, a bitstream generation device 1000 according to an embodiment will be described. Figure 14 is a block diagram showing an implementation example of the bitstream generation device 1000. The bitstream generation device 1000 includes a processor a1 and a memory a2. The bitstream generation device 1000 may also include an input unit for inputting moving images, and an output unit for outputting the generated bitstream.

[0207] Processor a1 is a circuit that performs information processing and is a circuit that can access memory a2. For example, processor a1 is a dedicated or general-purpose electronic circuit that generates a bitstream. Processor a1 may be a processor such as a CPU. Alternatively, processor a1 may be a collection of multiple electronic circuits.

[0208] Memory a2 is a dedicated or general-purpose memory in which information for processor a1 to generate a bitstream is stored. Memory a2 may be an electronic circuit and may be connected to processor a1. Memory a2 may also be included in processor a1. Memory a2 may also be a collection of multiple electronic circuits. Memory a2 may also be a magnetic disk or an optical disk, or may be described as storage or a recording medium. Memory a2 may also be a non-volatile memory or a volatile memory.

[0209] For example, memory a2 may store the input image or the generated stream. Memory a2 may also store a program for processor a1 to generate the bitstream.

[0210] The bitstream generator 1000 may have the same configuration as the encoding device 100 in this disclosure. Specifically, the bitstream generator 1000 may implement the multiple components shown in Figure 3. However, not all of the multiple components shown in Figure 3 are to be implemented, nor are all of the multiple processes to be performed. Some of the multiple components shown in Figure 3 may be included in other devices, and some of the multiple processes described above may be performed by other devices.

[0211] Furthermore, for example, processor a1 may perform the role of multiple components of the encoding device 100 shown in Figure 3, excluding the component for storing information. Also, for example, memory a2 may perform the role of the component for storing information among the multiple components of the encoding device 100 shown in Figure 3.

[0212] Specifically, memory a2 may function as the block memory 118 and frame memory 122 shown in Figure 3. More specifically, memory a2 may store reconstructed images (specifically, reconstructed blocks or reconstructed pictures, etc.).

[0213] Furthermore, during operation, processor a1 uses memory a2 to generate a bitstream that includes parameters for causing the decoding device 200 to execute processing, and / or parameters for switching the processing of the decoding device 200 or other specific processing. For example, processor a1 generates these parameters, includes the generated parameters in the bitstream, and generates the bitstream.

[0214] Here, the "parameters" to be executed by the decoding device 200 may be parameters for executing any decoding process exemplified in this disclosure as a process to be executed. For example, the "parameters" to be executed by the decoding device 200 may be any parameter in the syntax described in this disclosure. Furthermore, the "process of the decoding device 200" or "other specific process" that can be switched may be any decoding process exemplified in this disclosure as a process performed by obtaining the syntax.

[0215] Furthermore, the generated parameters disclosed in the embodiments may be stored in memory a2 and referenced in the processing of processor a1. The generated parameters may be stored in memory a2 and may or may not be included in the bitstream. Whether or not to include them in the bitstream is appropriately determined based on the relationship between the increase in the amount of code due to encoding the parameters and the reduction in the processing load of the decoding device 200 due to receiving the parameters.

[0216] [Storage Medium] Figure 15A is a schematic diagram showing an example of the configuration of a storage medium and a computer according to this embodiment. The storage medium 4000 is a medium for storing bitstreams. The storage medium 4000 is connected to a processor b1 included in the computer 5000 and outputs a bitstream, which is an executable instruction for the computer 5000, to the computer 5000. For example, when the computer 5000 reads the bitstream from the storage medium 4000, the storage medium 4000 outputs the bitstream to the computer 5000.

[0217] The encoded stream stored in the storage medium 4000 is an instruction that the computer 5000 can execute. The processor b1 in the computer 5000 generates a reconstructed image by executing a bitstream decoding process based on the acquired bitstream. In this way, the computer 5000, having acquired a stream from the storage medium 4000, can decode the bitstream based on the parameters and other information contained in the encoded stream.

[0218] Furthermore, the storage medium 4000 may be the memory b2 described herein. For example, the memory b2 may store a program for the processor b1 to generate a bitstream. The storage medium 4000 may be a non-transitory computer-readable medium. The storage medium 4000 is also referred to as a recording medium.

[0219] The storage medium 4000 may include an input / output unit for inputting and outputting bitstreams. The storage medium 4000 may also include an input unit for receiving bitstreams and an output unit for outputting bitstreams.

[0220] Furthermore, the computer 5000 may also be the decoding device 200. In other words, the computer 5000 may be read as the decoding device 200.

[0221] Figure 15B is a schematic diagram showing an example of the configuration of a computer according to this embodiment. The schematic diagram shown in Figure 15B differs from the example in Figure 15A in that the storage medium 4000 is included in the computer 5000. The storage medium 4000 and the processor b1 may be the same as those described in Figure 15A.

[0222] Here, the bitstream includes parameters that indicate instructions for the computer 5000 to execute. The "parameters" may include parameters that cause the computer 5000 to perform processing, and / or parameters that switch the processing or other specific processing of the computer 5000.

[0223] The “parameter” may be any parameter that performs any decryption process exemplified in this disclosure as an example of a process to be performed. For example, it may be any parameter in the syntax described in this disclosure. The process to be performed and the process to be switched may be any decryption process exemplified in this disclosure as a process performed by obtaining the syntax.

[0224] [Examples of Parameters] In the above, for the bitstream generator 1000, "parameters that cause the decoding device 200 to execute processing, and / or parameters that switch the processing of the decoding device 200 or other specific processing" are described. Also, for the storage medium 4000, "parameters that cause the computer 5000 to execute processing, and / or parameters that switch the processing of the computer 5000 or other specific processing" are described. These parameters may be, for example, the following parameters.

[0225] The "parameters" may include, for example, parameters indicating the method of dividing the block. The computer 5000 may divide the block based on the parameters indicating the method of dividing the block and perform any of the processes illustrated in this disclosure on the divided blocks.

[0226] The "parameters" may include, for example, parameters indicating the prediction mode to be applied. The computer 5000 may determine the prediction mode to be applied based on the parameters indicating the prediction mode to be applied, and may perform any prediction process exemplified in this disclosure corresponding to the determined prediction mode.

[0227] The "parameters" may include, for example, a parameter indicating whether or not process y can be performed within range x. The computer 5000 may determine whether or not process y can be performed on a target block based on the parameter indicating whether or not process y can be performed within range x.

[0228] A flag (parameter) indicating whether or not process y can be performed within range x is shown, for example, as xxx_yyy_enabled_flag / xxx_yyy_disabled_flag. Here, yyy may represent process y, which is indicated by the flag as either or not. Process y may also be any process exemplified in this disclosure.

[0229] xxx may represent the level to which parameters are assigned. For example, xxx may correspond to sps, pps, ph, sh, cu (Coding Unit), or block, and this level may indicate the range x. More specifically, if it is sps, the range x is a sequence; if it is pps, the range x is a picture; if it is ph, the range x is a picture; and if it is sh, the range x is a slice. Note that the levels to which parameters are assigned are not limited to these.

[0230] If the parameter indicates that process y is not feasible, then process y will not be performed in the blocks included in range x. If the parameter indicates that process y is feasible, then process y is feasible in the blocks included in range x. In other words, if the parameter indicates that process y is feasible, then process y may or may not be performed in the blocks included in range x.

[0231] The flag indicating whether or not process y can be performed may be defined for multiple headers. For example, process y may be permitted for a sequence, but not for a particular picture within that sequence.

[0232] In the above case, the sps_yyy_enabled_flag (or sps_yyy_disabled_flag) for the sequence may have a value indicating that process y can be performed. Furthermore, the ph_yyy_enabled_flag (or ph_yyy_disabled_flag) for a certain picture included in the sequence may have a value indicating that process y cannot be performed.

[0233] As described above, the levels correspond to sps, pps, ph, sh, cu (Coding Unit), or blocks, etc. An example definition for each level is shown below. Note that the definitions for each level can be changed.

[0234] The description of `sps` (Sequence Parameter Set) corresponds to a syntax structure containing syntax elements that apply to zero or more CLVS (Coded Layer Video Sequences). Here, the number of zero or more CLVSs is determined by the content of the syntax elements contained in the PPS referenced by the syntax elements contained in each picture header. In other words, `sps` corresponds to the sequence level.

[0235] The description of pps (Picture Parameter Set) corresponds to a syntax structure containing syntax elements that apply to zero or more encoded pictures. Here, zero or more encoded pictures are determined by the syntax elements contained in each picture header. In other words, pps corresponds to the picture level.

[0236] ph (PH: Picture Header) corresponds to a syntax structure containing syntax elements that apply to all slices of an encoded picture. In other words, ph corresponds to the picture level.

[0237] The entry for sh (SH: Slice Header) corresponds to the portion of the encoded slice that contains all tiles or data elements related to CTU rows within a tile. In other words, sh corresponds to the slice level.

[0238] The notation `cu` (Coding Unit) corresponds to the coding block of the sample and the syntax structure used to encode that sample. In other words, `cu` corresponds to the coding unit level or the block level.

[0239] Here, the above coding blocks correspond to (i) the coding block for the luminance sample and the two corresponding coding blocks for the chrominance sample of a picture having three sample sequences in single-tree mode, (ii) the coding block for the luminance sample of a picture having three sample sequences in dual-tree mode, (iii) the two coding blocks for the chrominance sample of a picture having three sample sequences in dual-tree mode, or (iv) the coding block for the sample of a monochrome picture.

[0240] Although a flag indicating whether or not process y can be performed has been described above, a flag indicating whether or not to perform process y (xxx_yyy_flag) may be used in a similar manner. Furthermore, the flag indicating whether or not process y can be performed and the flag indicating whether or not to perform process y may be used in combination.

[0241] [Data Structure] Figure 16 shows an example of the hierarchical structure of data in a stream. A stream includes, for example, a video sequence. This video sequence includes, for example, a VPS (Video Parameter Set), an SPS (Sequence Parameter Set), a PPS (Picture Parameter Set), an SEI (Supplemental Enhancement Information), and multiple pictures, as shown in Figure 16(a).

[0242] VPS includes encoding parameters common to multiple layers in a video composed of multiple layers, and encoding parameters related to the multiple layers included in the video, or to individual layers.

[0243] The SPS includes parameters used for the sequence, i.e., encoding parameters that the decoding device 200 refers to in order to decode the sequence. For example, the encoding parameters may indicate the width or height of the picture. Multiple SPSs may exist.

[0244] The PPS includes parameters used for the picture, i.e., encoding parameters that the decoding device 200 references to decode each picture in the sequence. For example, the encoding parameters may include a reference value for the quantization width used to decode the picture and a flag indicating the application of weighted prediction. There may be multiple PPSs. Also, the SPS and PPS are sometimes simply referred to as parameter sets.

[0245] The picture may include a picture header and one or more slices, as shown in Figure 16(b). The picture header includes encoding parameters that the decoding device 200 refers to in order to decode the one or more slices.

[0246] A slice includes a slice header and one or more Coding Tree Units (CTUs), as shown in Figure 16(c). The slice header includes coding parameters that the decoding device 200 references to decode the one or more CTUs.

[0247] A picture may not contain slices, but instead contain tile groups. In this case, a tile group may contain one or more tiles, and each tile may contain one or more CTUs. Furthermore, a tile may contain one or more slices, and a slice may contain one or more tiles.

[0248] A CTU is also called a superblock or basic partitioning unit. Such a CTU includes a CTU header and one or more CUs (Coding Units), as shown in Figure 16(d). The CTU header contains coding parameters that the decoding device 200 refers to in order to decode one or more CUs.

[0249] A CU may be divided into multiple smaller CUs. Furthermore, as shown in Figure 16(e), a CU includes a CU header, prediction information, and residual coefficient information. The prediction information is information for predicting the CU, and the residual coefficient information is information indicating the predicted residual, which will be described later.

[0250] Note that a CU is basically the same as a PU (Prediction Unit) and a TU (Transform Unit), but in SBT, for example, as described later, it may include multiple TUs smaller than the CU. Also, a CU may be processed for each VPDU (Virtual Pipeline Decoding Unit) that constitutes that CU. A VPDU is a fixed unit that can be processed in one stage when performing pipeline processing in hardware, for example.

[0251] Note that a stream does not necessarily have to have some of the hierarchical levels shown in Figure 16. Also, the order of these hierarchical levels may be changed, and any hierarchical level may be replaced by another hierarchical level.

[0252] Furthermore, the picture that is currently being processed by a device such as the encoding device 100 or the decoding device 200 is called the current picture. If the processing is encoding, the current picture is synonymous with the picture to be encoded; if the processing is decoding, the current picture is synonymous with the picture to be decoded.

[0253] Furthermore, a block, such as CU or CU, that is currently being processed by a device such as an encoding device 100 or a decoding device 200 is called the current block. If the processing is encoding, the current block is synonymous with the block to be encoded; if the processing is decoding, the current block is synonymous with the block to be decoded.

[0254] [Picture Composition: Slices / Tiles] To decode pictures in parallel, pictures may be composed of slices or tiles.

[0255] A slice is the basic encoding unit that makes up a picture. A picture is composed of, for example, one or more slices. A slice consists of one or more consecutive CTUs.

[0256] Figure 17 shows an example of a slice configuration. For example, a picture contains 11 x 8 CTUs and is divided into four slices (slices 1-4). Slice 1 consists of, for example, 16 CTUs, slice 2 consists of, for example, 21 CTUs, slice 3 consists of, for example, 29 CTUs, and slice 4 consists of, for example, 22 CTUs. Here, each CTU in the picture belongs to one of the slices.

[0257] The shape of a slice is a horizontal division of the picture. The boundaries of a slice do not have to be at the edges of the screen, but can be anywhere among the CTU boundaries within the screen. The processing order (encoding order or decoding order) of the CTUs within a slice is, for example, the raster scan order. A slice also includes a slice header and encoded data. The slice header may describe the characteristics of the slice, such as the CTU address of the beginning of the slice and the slice type.

[0258] A tile is a rectangular area that makes up a picture. Each tile may be assigned a number called a TileId in the order of the raster scan.

[0259] Figure 18 shows an example of a tile configuration. For example, a picture contains 11 x 8 CTUs and is divided into four rectangular tile regions (tiles 1-4). When tiles are used, the processing order of the CTUs is changed compared to when tiles are not used.

[0260] If tiles are not used, multiple CTUs within a picture are processed, for example, in raster scan order. If tiles are used, at least one CTU in each of the multiple tiles is processed, for example, in raster scan order. For example, as shown in Figure 18, the processing order of the multiple CTUs contained in tile 1 is from the left end of the first column of tile 1 to the right end of the first column of tile 1, and then from the left end of the second column of tile 1 to the right end of the second column of tile 1.

[0261] Note that one tile may contain one or more slices, and one slice may contain one or more tiles.

[0262] A picture may be composed of tilesets. A tileset may contain one or more tile groups, or one or more tiles. A picture may consist of only one of a tileset, a tile group, or a tile. For example, the order in which multiple tiles are scanned in raster order for each tileset is defined as the basic coding order of the tiles. Within each tileset, a collection of one or more tiles whose basic coding order is consecutive is defined as a tile group. Such a picture may be composed of the division unit 102 (see Figure 3) described later.

[0263] [Temporal Scalable Encoding] Figure 19 shows an example of a temporally scalable stream configuration.

[0264] The encoding device 100 may generate a temporally scalable stream by encoding multiple pictures in multiple layers, as shown in Figure 19. For example, the encoding device 100 achieves scalability by encoding each picture in its own layer, with enhancement layers existing above the base layer. This type of encoding of each picture is called temporally scalable encoding.

[0265] This allows the decoding device 200 to switch the frame rate of the image displayed by decoding the stream. In other words, the decoding device 200 decides which layers to decode based on internal factors such as its own performance and external factors such as the state of the communication bandwidth.

[0266] As a result, the decoding device 200 can freely switch between decoding the same content in low-frame-rate and high-frame-rate formats. For example, a user of the stream can watch part of the video using their smartphone while on the go, and then watch the rest of the video using an internet TV or other device after returning home.

[0267] Each of the aforementioned smartphones and devices incorporates a decoding device 200, which may or may not have the same performance. In this case, if the device decodes up to the upper layers of the stream, the user can view high-frame-rate video after returning home. This eliminates the need for the encoding device 100 to generate multiple streams with the same content but different frame rates, thereby reducing the processing load.

[0268] In temporally scalable coding, layers are sometimes referred to as temporal layers or temporal sublayers.

[0269] [Multilayer Encoding] Figure 20 shows an example of a stream configuration using multilayer encoding. Each of AU0 to AU3 corresponds to an AU (Access Unit). Each of Layer0 to Layer3 corresponds to a layer in multilayer encoding. Each PU (Picture Unit) corresponds to a picture. Each of POC_1 to POC_3 corresponds to a POC (Picture Of Count) and indicates the display order.

[0270] As shown in Figure 20, the encoding device 100 may generate a stream in which spatial resolution, image quality, or multiplexed content can be scalably changed by encoding multiple pictures into multiple layers. For example, the encoding device 100 achieves scalability by encoding pictures layer by layer, with enhancement layers existing above the base layer. This type of encoding of each picture is called multi-layer encoding.

[0271] This allows the decoding device 200 to switch the spatial resolution, image quality, or multiplexed content of the image displayed by decoding the stream. In other words, the decoding device 200 decides which layers to decode based on internal factors such as its own performance and external factors such as the state of the communication bandwidth.

[0272] As a result, the decoding device 200 can freely switch between decoding low-resolution and high-resolution content, low-quality and high-quality content, and basic and customized content for the same content. For example, a user of the stream can watch part of the video on their smartphone while on the go, and then watch the rest of the video on an internet TV or other device after returning home.

[0273] Furthermore, each of the aforementioned smartphones and devices incorporates a decoding device 200 with identical or different performance characteristics. In this case, if the device decodes up to the upper layers of the stream, the user can view high-definition video after returning home. This eliminates the need for the encoding device 100 to generate multiple streams with the same content but different image quality, thereby reducing the processing load.

[0274] [Metadata] Each layer of the stream may include metadata based on statistical information of the image. The decoding device 200 may generate high-resolution video by super-resolution the pictures of each layer based on the metadata. Super-resolution may be either an improvement in the signal-to-noise ratio (SN) at the same resolution, or an increase in resolution. The metadata may include information for identifying linear or nonlinear filter coefficients used in the super-resolution process, or information for identifying parameter values ​​in the filtering process, machine learning, or least-squares operation used in the super-resolution process.

[0275] Alternatively, the picture may be divided into tiles or the like, depending on the meaning of each object within it. In this case, the decoding device 200 may decode only a portion of the picture by selecting the tiles to be decoded.

[0276] Furthermore, object attributes (such as person, car, or ball) and their position within the picture (such as their coordinate position within the same picture) may be stored as metadata. In this case, the decoding device 200 can identify the position of a desired object based on the metadata and determine the tile containing that object. For example, the metadata is stored using a data storage structure different from pixel data, such as SEI in HEVC. This metadata may indicate, for example, the position, size, or color of the main object.

[0277] Furthermore, metadata may be stored in units consisting of multiple pictures, such as streams, sequences, or random access units. This allows the decoding device 200 to obtain information such as the time when a specific person appears in the video, and by using that time and the information in the picture units, it can identify the picture in which the object exists and the position of the object within that picture.

[0278] [Loop Filtering Section in Encoding Process] The loop filter section 120 in the encoding device 100 applies a filter to the reconstructed image output from the adder 116 and outputs the filtered reconstructed image to the frame memory 122. Here, the filter performed by the loop filter section 120 is a filter used within the encoding loop (also called a loop filter or in-loop filter).

[0279] Filtering is a technique that reduces encoding noise generated by quantization processing within the encoding loop. Filtering can directly reduce visually noticeable block distortion or ringing distortion, improving subjective and / or objective performance. Furthermore, the loop filter unit 120 prevents image quality degradation from propagating between frames by performing filtering within the loop.

[0280] Filters include, for example, adaptive loop filters (ALF), deblocking filters (DF or DBF), sample adaptive offset (SAO), luminance mapping with chroma scaling (LMCS), or any combination thereof.

[0281] Figure 21 is a block diagram showing an example of the configuration of the loop filter unit 120. The loop filter unit 120 includes, for example, an LMCS processing unit 120d, a DBF processing unit 120a, a SAO processing unit 120b, and an ALF processing unit 120c, as shown in Figure 21.

[0282] The LMCS processing unit 120d performs luminance mapping and color difference scaling on the reconstructed image. The DBF processing unit 120a performs DBF processing on the reconstructed image after LMCS processing. The SAO processing unit 120b performs SAO processing on the reconstructed image after DBF processing. In addition, the ALF processing unit 120c applies ALF processing to the reconstructed image after SAO processing. Details of ALF and DBF will be described later.

[0283] SAO processing is a process that improves image quality by reducing ringing (a phenomenon in which pixel values ​​are distorted in a wave-like manner around edges) and correcting pixel value misalignment. Examples of SAO processing include edge offset processing and band offset processing.

[0284] LMCS processing is a process that reassigns codewords assigned to rarely occurring pixel values ​​to codewords assigned to frequently occurring pixel values ​​when there is a large bias in the pixel value distribution of the luminance signal of the original image. Examples of LMCS processing include mapping the luminance signal using a piecewise linear model based on the pixel value distribution of the original image, and scaling the residual color difference signal according to the luminance pixel values.

[0285] Furthermore, the loop filter unit 120 does not necessarily have to include all of the processing units disclosed in Figure 21, and may include only some of them. Also, the loop filter unit 120 may perform the above-mentioned processing in an order different from the processing order disclosed in Figure 21.

[0286] [Loop Filter Section > Adaptive Loop Filter] In ALF, a least-squares error filter is applied to remove encoding distortion. For example, for each 2x2 pixel subblock within the current block, one filter selected from several filters is applied based on the direction and activity of the local gradient.

[0287] Specifically, first, subblocks (for example, 2x2 pixel subblocks) are classified into multiple classes (for example, 15 or 25 classes). The classification of subblocks is done, for example, based on the direction of the gradient and the activity level. In a specific example, a classification value C (for example, C = 5D + A) is calculated using the gradient direction value D (for example, 0 to 2 or 0 to 4) and the gradient activity value A (for example, 0 to 4). Then, based on the classification value C, the subblocks are classified into multiple classes.

[0288] The gradient direction value D is derived, for example, by comparing gradients in multiple directions (e.g., horizontal, vertical, and two diagonal directions). The gradient activation value A is derived, for example, by adding the gradients in multiple directions and quantizing the sum.

[0289] Based on the results of this classification, a filter for the subblock is determined from among multiple filters.

[0290] For example, circularly symmetrical shapes are used for filters in ALF. Figures 22A to 22C show several examples of filter shapes used in ALF. Figure 22A shows a 5x5 diamond-shaped filter, Figure 22B shows a 7x7 diamond-shaped filter, and Figure 22C shows a 9x9 diamond-shaped filter.

[0291] Information indicating the shape of the filter is typically signaled at the picture level. However, the signaling of information indicating the shape of the filter is not limited to the picture level; it may be at other levels (e.g., sequence level, slice level, brick level, CTU level, or CU level).

[0292] The on / off status of the ALF may be determined, for example, at the picture level or the CU level. For example, the decision to apply the ALF may be made at the CU level for luminance, and at the picture level for color difference. Information indicating whether the ALF is on or off is usually signaled at the picture level or the CU level.

[0293] Note that the signaling of ALF on / off information is not limited to picture level or CU level, but may be at other levels (e.g., sequence level, slice level, brick level, or CTU level). If the ALF on / off information read from the stream indicates that ALF is on, one filter is selected from several filters based on the direction and activity of the local gradient, and the selected filter is applied to the reconstructed image.

[0294] Furthermore, as described above, one filter is selected from among several filters and ALF processing is applied to the subblock. For each of these filters (for example, up to 15 or 25 filters), the set of coefficients used in that filter is usually signaled at the picture level. However, the signaling of the coefficient set is not limited to the picture level and may be at other levels (for example, sequence level, slice level, brick level, CTU level, CU level, or subblock level).

[0295] [Loop Filter Section > CCALF] CC-ALF (Cross Component Adaptive Loop Filter) is a type of restoration filter that uses correlations between color components. The filtering results of eight luminance signals corresponding to the positions of the color difference signal being processed are added to the color difference signal. To enable parallel processing with ALF, the luminance signal before the application of ALF is used for filtering.

[0296] Figure 22D shows a configuration diagram of an example of a CCALF in this disclosure. Figure 22E shows an example of a filter shape of a CCALF in this disclosure.

[0297] One example of CC-ALF operates by applying a linear diamond-shaped filter (Figures 22D and 22E) to the luminance channel of each chromatic difference component. For example, the filter coefficients are transmitted via APS, scaled by a factor of 2^10, and rounded for fixed-point representation. The application of the filter is controlled by a variable block size and indicated by a context-encoded flag received for each block of samples. The block size and CC-ALF enable flag are received at the slice level for each chromatic difference component.

[0298] The syntax and semantics of CC-ALF are provided in the encoding standard. Block sizes of 16x16, 32x32, 64x64, and 128x128 may also be supported for color difference samples.

[0299] [Loop Filter Section > DBF] In DBF processing, the loop filter section 120 reduces distortion occurring at block boundaries by applying a filter to the block boundaries of the reconstructed image.

[0300] Figure 23A is a block diagram showing an example of the detailed configuration of the DBF processing unit 120a. The DBF processing unit 120a includes, for example, a boundary determination unit 1201, a filter determination unit 1203, a filter processing unit 1205, a processing determination unit 1208, a filter characteristic determination unit 1207, and switches 1202, 1204, and 1206.

[0301] Figure 23B is a flowchart showing an example of the processing performed by the DBF processing unit 120a.

[0302] First, the DBF processing unit 120a determines the boundary to be processed in the boundary determination unit 1201 (step S_0b).

[0303] Next, the DBF processing unit 120a determines whether or not to perform DBF (step S_1b). If it determines to perform DBF, it proceeds to the next determination. If the DBF processing unit 120a determines not to perform DBF, it terminates the process without performing DBF (step S_2b).

[0304] Specifically, for example, the DBF processing unit 120a may determine in the boundary determination unit 1201 whether or not the target pixel for DBF processing is located near a block boundary, and if the target pixel is not located near a block boundary, it may decide not to perform DBF. Alternatively, the DBF processing unit 120a may calculate a Bs value and decide not to perform DBF according to the Bs value. For example, if the Bs value is equal to 0, it indicates that DBF will not be performed, and if it is 1 or greater, it indicates that there is a possibility of performing DBF.

[0305] Next, the DBF processing unit 120a determines whether or not to perform DBF on the target pixel in the filter determination unit 1203 (step S_3b). In other words, the DBF processing unit 120a determines the type of filter to be applied. Specifically, for example, the filter determination unit 1203 determines whether or not to perform DBF processing on the target pixel based on the pixel values ​​of at least one surrounding pixel located around the target pixel. The image before filtering is an image consisting of the target pixel and at least one surrounding pixel located around that target pixel.

[0306] The filter determination unit 1203 may compare a value based on the pixel value of at least one peripheral pixel, and / or the difference between multiple pixels, etc., with a threshold value, and decide whether to execute the DBF to be determined if the value is greater than the threshold value, and whether to execute the DBF to be determined if the value is less than or equal to the threshold value. The threshold value is, for example, β or tC determined based on the quantization parameter QP. In other words, in the DBF process, for example, the DBF to be performed is selected based on quantization.

[0307] In one example of the process for determining whether or not to perform the DBF to be judged, it is first determined whether or not to perform DBF(a). If it is determined that DBF(a) should be performed, DBF(a) is performed (step S_4b). If it is determined that DBF(a) should not be performed, the next determination is made. In the next determination, it is determined whether or not to perform DBF(b). If it is determined that DBF(b) should be performed, DBF(b) is performed (step S_5b). If it is determined that DBF(b) should not be performed, DBF is not performed (step S_2b).

[0308] Furthermore, if it is decided not to implement DBF(b), there may be multiple DBF candidates, such as deciding whether or not to implement DBF(c). Here, DBF(a), which has been decided to implement, may be further subdivided, and it may be decided whether or not to implement DBF(a_1) or DBF(a_2).

[0309] Here, DBF(a), DBF(b), DBF(a_1), and DBF(a_2) may be short filter, long filter, weak filter, and strong filter, respectively.

[0310] Figure 23C is a flowchart showing another example of the processing performed by the DBF processing unit 120a.

[0311] First, the DBF processing unit 120a determines the boundary to be processed in the boundary determination unit 1201 (step S_0c). Next, the DBF processing unit 120a determines whether or not to perform DBF (step S_1c), and if it determines to perform DBF, it selects the DBF to perform (step S_2c). Then, the DBF processing unit 120a performs the selected DBF (step S_3c). If the DBF processing unit 120a determines in S_1c not to perform DBF, it terminates the process without performing DBF (step S_4c).

[0312] Note that the processing of S_1c may be the same as or different from the processing of S_1b. Also, the processing of S_2c may be the same as or different from the processing of S_3b.

[0313] Figure 24 shows an example of a DBF (Digital Block Filter) with symmetrical filter characteristics with respect to block boundaries. DBF is a process that reduces distortion at block boundaries by correcting the sample values ​​on both sides of the block boundary. The number of pixels to be filtered and the number of pixels used for filtering vary depending on the type of filter. The higher the filter strength, the greater the number of pixels to be filtered and the greater the number of pixels used for filtering.

[0314] When DBF processing is performed on the block boundary between block P and block Q adjacent to block P, the pixels to be processed are pn (n=0, ..., n) in block P and qm (m=0, ..., m) in block Q. Pixels with a value of n or m of 0 are adjacent to the boundary, while pixels with larger values ​​are further from the boundary.

[0315] For example, in a strong filter, as shown in Figure 24, processing is performed on pixels p0 to p2 in block P and pixels q0 to q2 in block Q. The respective pixel values ​​of pixels q0 to q2 are changed to pixel values ​​q'0 to q'2 by performing the calculation shown in the following equation.

[0316] q'0=(p1+2×p0+2×q0+2×q1+q2+4) / 8 q'1=(p0+q0+q1+q2+2) / 4 q'2=(p0+q0+q1+3×q2+2×q3+4) / 8

[0317] In the above equations, p0 to p2 and q0 to q2 are the pixel values ​​of pixels p0 to p2 and pixels q0 to q2, respectively. Also, q3 is the pixel value of pixel q3, which is adjacent to pixel q2 on the opposite side of the block boundary. Furthermore, the coefficient multiplied by the pixel value of each pixel used in the deblocking filter process on the right-hand side of each of the above equations is the filter coefficient.

[0318] Furthermore, in DBF processing, clipping may be performed to ensure that the pixel value after calculation does not change beyond a threshold. In this clipping process, the pixel value after calculation using the above formula is clipped to "pre-calculation pixel value ± a × threshold (where a is an integer)" using a threshold determined from the quantization parameters. In other words, the amount of change in the pixel value before and after calculation is clipped so that it is within the range of ± a × threshold (where a is an integer). This prevents excessive smoothing. Specifically, for example, the pixel values ​​q'0 to q'2 are expressed by the following formula.

[0319] q'0=Clip3 (q0-3×tC, q0+3×tC, (p1+2×p0+2×q0+2×q1+q2+4)>>3) q'1=Clip3 (q1-2×tC, q1+2×tC, (p0+q0+q1+q2+2)>>2) q'2=Clip3 (q2-1×tC, q2+1×tC, (p0+q0+q1+3×q2+2×q3+4)>>3)

[0320] Furthermore, Clip3 may be defined as follows:

[0321]

[0322] Figure 25 is a diagram illustrating an example of a block boundary where DBF processing is performed. Figure 26 is a diagram illustrating an example of a block boundary strength Bs value.

[0323] The block boundaries on which DBF processing is performed are, for example, the CU, PU, ​​or TU boundaries of an 8x8 pixel block as shown in Figure 25. DBF processing is performed, for example, in units of 4 rows or 4 columns. First, for blocks P and Q shown in Figure 25, the Boundary Strength (Bs) value is determined as shown in Figure 26. The Bs value is determined independently for each color component.

[0324] In Figure 26, conditions listed higher up have higher priority, and evaluation is performed in order from the highest priority condition. If a condition is met, evaluation of lower priority conditions is not performed. The magnitude and range of block distortion differ depending on the boundary, and excessive smoothing can cause problems such as blurring. Therefore, in this example, the Bs value is determined according to the prediction modes of blocks P and Q, the presence or absence of motion vectors and transformation coefficients.

[0325] Furthermore, based on the determined Bs value, it may be decided whether or not to perform DBF processing of different strengths even for block boundaries belonging to the same image. DBF processing for chrominance signals is performed when the Bs value is 2. DBF processing for luminance signals is performed when the Bs value is 1 or greater and predetermined conditions are met. Note that the criteria for determining the Bs value are not limited to the example shown in Figure 26, and may be determined based on other parameters.

[0326] The above method for determining the Bs value and the method for performing DBF processing based on the Bs value are merely examples, and the method for determining the Bs value and the method for performing DBF processing based on the Bs value are not limited to the above example.

[0327] In DBF processing, it is determined whether or not to apply a filter to each of the luminance sample and chrominance sample, and the type of filter to be applied. Furthermore, in DBF processing, it is determined whether or not to apply a filter to each of the vertical and horizontal boundaries contained in each of the luminance sample and chrominance sample, and the type of filter to be applied.

[0328] Some or all of the processing may be performed on either the luminance sample or the chrominance sample, or on both. The luminance sample and the chrominance sample may undergo the same processing, or they may undergo different processing.

[0329] [Loop Filtering Unit in Decoding Process] The loop filtering unit 212 in the decoding device 200 applies a loop filter to the reconstructed image generated by the summing unit 208, and outputs the filtered reconstructed image to the frame memory 214 and the display device, etc.

[0330] Figure 27 is a block diagram showing an example of the configuration of the loop filter unit 212. The loop filter unit 212 has a configuration similar to that of the loop filter unit 120 of the encoding device 100. The loop filter unit 212 includes, for example, an LMCS processing unit 212d, a DBF processing unit 212a, a SAO processing unit 212b, and an ALF processing unit 212c, as shown in Figure 27.

[0331] The LMCS processing unit 212d performs luminance mapping and color difference scaling on the reconstructed image. The DBF processing unit 212a performs DBF processing on the reconstructed image after LMCS processing. The SAO processing unit 212b performs SAO processing on the reconstructed image after DBF processing. In addition, the ALF processing unit 212c applies ALF processing to the reconstructed image after SAO processing.

[0332] Note that the loop filter unit 212 does not necessarily have to include all of the processing units disclosed in Figure 27, and may include only some of them. Also, the loop filter unit 212 may perform the above-mentioned processing in an order different from the processing order disclosed in Figure 27.

[0333] [Prediction Unit (Intra Prediction Unit, Inter Prediction Unit, Prediction Control Unit)] Figure 28 is a flowchart showing an example of processing performed in the prediction unit of the encoding device 100. For example, the prediction unit consists of all or some of the components of the intra prediction unit 124, the inter prediction unit 126, and the prediction control unit 128. The prediction processing unit includes, for example, the intra prediction unit 124 and the inter prediction unit 126.

[0334] The prediction unit generates a predicted image of the current block (step Sb_1). The predicted image may be, for example, an intra-prediction image (intra-prediction signal) or an inter-prediction image (inter-prediction signal). Specifically, the prediction unit generates a predicted image of the current block using a reconstructed image already obtained by generating predicted images for other blocks, generating prediction residuals, generating quantization coefficients, restoring prediction residuals, and adding the predicted images.

[0335] The reconstructed image may be, for example, the image of the reference picture, or it may be the image of an encoded block (i.e., the other block mentioned above) within the current picture, which is the picture containing the current block. An encoded block within the current picture is, for example, an adjacent block to the current block.

[0336] Figure 29 is a flowchart showing another example of the processing performed in the prediction unit of the encoding device 100.

[0337] The prediction unit generates a predicted image using a first method (step Sc_1a), a second method (step Sc_1b), and a third method (step Sc_1c). The first, second, and third methods are different methods for generating predicted images, and may be, for example, an interpretation method, an intraprediction method, and other prediction methods. These prediction methods may use the reconstructed images described above.

[0338] Next, the prediction unit evaluates the predicted images generated in steps Sc_1a, Sc_1b, and Sc_1c (step Sc_2). For example, the prediction unit calculates a cost C for each of the predicted images generated in steps Sc_1a, Sc_1b, and Sc_1c, and evaluates the predicted images by comparing the costs C of those predicted images.

[0339] The cost C is calculated using the R-D optimization model formula, for example, C = D + λ × R. In this formula, D is the coding distortion of the predicted image, which can be expressed as, for example, the sum of the absolute differences between the pixel values ​​of the current block and the pixel values ​​of the predicted image. R is the bitrate of the stream, and λ is, for example, the Lagrange multiplier.

[0340] Next, the prediction unit selects one of the predicted images generated in steps Sc_1a, Sc_1b, and Sc_1c (step Sc_3). In other words, the prediction unit selects a method or mode for obtaining the final predicted image. For example, the prediction unit selects the predicted image with the smallest cost C based on the cost C calculated for those predicted images. Alternatively, the evaluation in step Sc_2 and the selection of the predicted image in step Sc_3 may be based on parameters used in the coding process.

[0341] The encoding device 100 may signal information to identify the selected prediction image, scheme, or mode into a stream. This information may be, for example, a flag. Based on this information, the decoding device 200 can generate a prediction image according to the scheme or mode selected by the encoding device 100.

[0342] In the example shown in Figure 29, the prediction unit generates predicted images using each method and then selects one of the predicted images. However, the prediction unit may also select a method or mode based on the parameters used in the encoding process described above before generating those predicted images, and then generate the predicted images according to that method or mode.

[0343] For example, the first method and the second method are intra-prediction and inter-prediction, respectively, and the prediction unit may select the final predicted image for the current block from the predicted images generated according to these prediction methods.

[0344] Figure 30 is a flowchart showing another example of the processing performed in the prediction unit of the encoding device 100.

[0345] First, the prediction unit generates a predicted image by intra-prediction (step Sd_1a) and then generates a predicted image by inter-prediction (step Sd_1b). The predicted image generated by intra-prediction is also called an intra-prediction image, and the predicted image generated by inter-prediction is also called an inter-prediction image.

[0346] Next, the prediction unit evaluates both the intra-predicted image and the inter-predicted image (step Sd_2). The cost C mentioned above may be used for this evaluation. The prediction unit may then select the prediction image with the smallest cost C from the intra-predicted image and the inter-predicted image as the final prediction image for the current block (step Sd_3). In other words, a prediction method or mode for generating the prediction image for the current block is selected.

[0347] [Prediction Control Unit] The prediction control unit 128 selects either an intra-prediction image (an image or signal output from the intra-prediction unit 124) or an inter-prediction image (an image or signal output from the inter-prediction unit 126), and outputs the selected prediction image to the subtraction unit 104 and the addition unit 116.

[0348] [Prediction Parameter Generation Unit] The prediction parameter generation unit 130 may output information regarding intra-prediction, inter-prediction, and the selection of a predicted image in the prediction control unit 128 as prediction parameters to the entropy coding unit 110. The entropy coding unit 110 may generate a stream based on the prediction parameters input from the prediction parameter generation unit 130 and the quantization coefficients input from the quantization unit 108. The prediction parameters may be used by the decoding device 200.

[0349] The decoding device 200 may receive and decode the stream and perform the same processing as the prediction processing performed in the intra prediction unit 124, the inter prediction unit 126, and the prediction control unit 128.

[0350] The prediction parameters may include a selected prediction signal (e.g., MV, prediction type, or prediction mode used in the intra-prediction unit 124 or the inter-prediction unit 126), or any index, flag, or value that is based on or indicates the prediction processing performed in the intra-prediction unit 124, the inter-prediction unit 126, and the prediction control unit 128.

[0351] [Generated Reference Picture] Figure 31A is a conceptual diagram showing another example of a reference picture. As illustrated in Figure 31A, a new image (PicA') generated based on an image (PicA) may be used as a reference picture. A reference picture generated in this way is defined as a generated reference picture. A generated reference picture may be used as a reference picture in the prediction process.

[0352] Additionally, the generated reference picture may be added to the same reference picture list as the reference picture list to which other reference pictures are added. Alternatively, a separate reference picture list may be generated, and the generated reference picture may be added to that separate reference picture list.

[0353] Figure 31B is a conceptual diagram illustrating another example of generating a generated reference picture. In the example in Figure 31B, a new image (PicAB') is generated by inputting two images (PicA and PicB) into an NPU (Neural Processing Unit) or a GPU (Graphics Processing Unit). The generated new image (PicAB') is used as a generated reference picture for interpretation.

[0354] Interpretation using this generated reference picture may also be called neural network intercoding (NN inter). Note that the NPU or GPU is not limited to two images, but can receive two or more images as input.

[0355] Figure 31C is a conceptual diagram illustrating another example of generating a generated reference picture. As illustrated in Figure 31C, the size of the input image and the size of the generated image may differ. In this example, the size of the generated image (PicA') is smaller than the size of the input image (PicA). The size of the generated image is not limited to this example and may be larger than the size of the input image. Image processing or the size of the generated image per frame may be based on the performance of the NPU or GPU. This size information may also be included in the bitstream.

[0356] In the example above, a new reference picture (also called a generated image or processed image) is generated from one or more encoded or decoded reference pictures using a neural network. The method for generating the new reference picture is not limited to the example above. For example, a new reference picture may be generated using a process such as image conversion, and the generated new reference picture may be defined as the generated reference picture.

[0357] Adding these generated reference pictures to the reference picture list and making them accessible may improve encoding efficiency.

[0358] In the example above, images are generated on a picture-by-picture basis, but the generation unit is not limited to this example. Reference data may be generated in predetermined units such as blocks, slices, or tiles contained within a picture.

[0359] [Generated Pictures] Generated pictures are images obtained through a generation process, unlike regular reference images (in other words, images decoded by intra / inter prediction). Here, as an example, we show how to generate a new reference picture by inputting one or more pictures into a neural network, but the method of generating a new reference picture is not limited to this.

[0360] The generation process may support the application of NN loop filter (NN LF) processing, which includes NN (Neural Network) processing, or it may support transformation processing (scaling, rotation, mirroring, shifting, etc.). Furthermore, the generation process may be a combination of multiple processes (for example, a process that generates a predicted image using an NN such as NN LF, a transformation process, and other processes).

[0361] The new reference picture generated by the generation process is also referred to as a generated reference picture, a generated picture, or a processed picture. Furthermore, such a new reference picture or its image may also be referred to as a generated reference image, a generated image, or a processed image. Adding the generated picture to the reference picture list and making it accessible may improve encoding efficiency.

[0362] First, we will explain the generated reference picture that is generated using the NN. The encoding device 100 and the decoding device 200 input one or more images to the NN and generate a generated picture.

[0363] Figure 32A is a conceptual diagram illustrating an example of a generated picture. As illustrated in Figure 32A, a new image (PicA') generated by inputting an image (PicA) into the NN may be used as a reference picture. For example, a reference picture generated by NN LF is an example of this.

[0364] Figure 32B is a conceptual diagram showing another example of generating a generated picture. As illustrated in Figure 32B, a new image (PicAB') generated by inputting two images (PicA and PicB) into the NN may be used as the reference picture. Note that there are not limited to two images input into the NN; more than two images can be input.

[0365] Figure 32C is a conceptual diagram showing another example of generating a generated picture. As illustrated in Figure 32C, if NN processing such as NN LF is not fast enough, a new image (PicA') smaller in size than the input image (PicA) may be used as a reference picture. In other words, the size of the input image and the size of the generated image may be different. Also, information about the size of the generated image may be included in the bitstream.

[0366] While this example shows generation at the picture level, the generation unit is not limited to this. The generated reference picture may be generated in predetermined units such as blocks, slices, or tiles contained within the picture.

[0367] Furthermore, the NN may be implemented using an NPU (Neural Processing Unit), a GPU (Graphics Processing Unit), or a TPU (Tensor Processing Unit).

[0368] Furthermore, as mentioned above, the newly generated image (PicA' or PicAB') may be used as a generated picture for inter prediction. Inter prediction using this generated picture may be called neural network intercoding (NN inter).

[0369] Furthermore, multiple neural networks (NNs) may be combined in the generation of the generated picture. In other words, for example, PicA and PicB may be input to NN1 to generate PicAB', and PicAB' may be input to another NN2 to generate PicAB''.

[0370] In typical image encoding on a CPU, a conventional pipeline is used, as shown in Figure 33A. Specifically, the entire encoding process is divided into several stages (0 to n), and processing is performed in block units within each stage. Furthermore, multiple stages are processed in parallel with each other.

[0371] In contrast, for generated pictures produced using NN processing, as can be seen from the configuration examples in Figures 33B and 33C, the image is first processed by the CPU, and then the data is passed to the NPU / GPU / TPU, etc. Then, NN processing is executed on the NPU / GPU / TPU side using a different pipeline than the CPU. In addition, processing may be executed on the NPU / GPU / TPU in units different from those of the CPU.

[0372] Due to overhead caused by data transfer and other processes, it may take some time for the generated picture produced by the NN processing to become available as a reference image for interpretation.

[0373] Patches may be defined to generate the generated picture by NN processing. A patch refers to a predetermined pixel region containing a predetermined number of pixels, as shown in Figure 34A. The shape of the patch may be a square, a non-square rectangle, or any other shape, as shown in Figure 34A. For example, a patch may have a non-rectangular shape such as a triangle or a rhombus.

[0374] Figure 34B is a conceptual diagram showing a patch containing an extended region. A patch containing an extended region includes an extended region in the vertical and / or horizontal directions relative to the overall size of the patch.

[0375] In the example shown in Figure 34B, the extension area has equal size in the vertical and horizontal directions, resulting in a square patch. The sizes of the vertical and horizontal extension areas are not limited to the example in Figure 34B and may be different. Also, the extension areas do not have to be equal, for example, at the top and bottom, or left and right. Furthermore, information regarding the shape or size of the patch may be added to the header area of ​​the bitstream, or it may be added to the bitstream as metadata (also called meta information).

[0376] Figure 34C is a conceptual diagram showing an image containing patches 1A and 1B that have overlapping regions. Patches 1A and 1B overlap each other across multiple pixels (=set of samples), as shown by the dotted lines in Figure 34C. Information indicating the shape (width and height) or size (number of pixels included in the overlapping region) of this overlapping region may be stored in the bitstream.

[0377] Furthermore, in the example in Figure 34C, each patch contains a 256x256 block. Therefore, each patch can also be described as a patch with an extended area relative to the 256x256 block in Figure 34B. In the example in Figure 34C, the 256x256 block in patch 1A and the 256x256 block in patch 1B are defined so as not to overlap.

[0378] Furthermore, while Figure 34C shows an example of two patches defined within a single picture, in an NN interface, for example, two patches may be defined in each of two pictures. In that case, the positions of the patches defined in the first and second pictures (for example, patch 1A in the first picture and patch 2A in the second picture, which is not shown) may be the same or different.

[0379] Figure 34D is a conceptual diagram showing the patches defined in each picture. Specifically, it shows multiple patches defined in an example such as an NN interface, including the first and third patches in the first picture, the second and fourth patches in the second picture, and the fifth and sixth patches in the third picture. The juxtaposed points of the third, fourth, and sixth patches are indicated by an "x" mark.

[0380] Figure 34E is a conceptual diagram showing an example of a method for generating the sixth patch. The sixth patch is obtained by cropping the seventh patch, which is generated by inputting the third and fourth patches into an NN generator. Similarly, the fifth patch may be obtained by cropping an image (the eighth patch, not shown) generated by inputting the first and second patches into an NN generator.

[0381] Furthermore, the first and second pictures may be ordinary reference pictures. In contrast, the third picture, which includes the fifth and sixth patches, may be a generated picture. In one embodiment, the encoding order and display order may be first picture → second picture → third picture. In another embodiment, the display order may be first picture → third picture → second picture.

[0382] Next, the generated picture produced using the transformation process will be described. The encoding device 100 and the decoding device 200 generate a generated picture by performing a transformation process on a single image. The single image input to the transformation process may be a normal reference image (in other words, an image decoded by intra / inter prediction) or an image generated by an NN. Furthermore, the image generated by the transformation process may be used as input to the NN.

[0383] Figure 35A is a conceptual diagram illustrating another example of a generated picture. As illustrated in Figure 35A, a new image (PicA') generated by scaling (enlarging or reducing) an image (PicA) may be used as the reference picture.

[0384] In the case of upscaling (enlarging), a new image PicA' is generated by enlarging the image and then cropping the resulting enlarged image. In the case of downscaling (reducing), a new image PicA' is generated by reducing the image and then padding the resulting reduced image. Note that enlargement can also be expressed as expansion.

[0385] Figure 35B is a conceptual diagram illustrating another example of generating a generated picture. As illustrated in Figure 35B, a new image (PicA') generated by rotating an image (PicA) may be used as the reference picture. For example, a new image PicA' is generated by rotating the image by an angle θ in a specified direction and then cropping and padding the rotated image. The rotation direction may always be clockwise or always counterclockwise, and the rotation direction may also be specified as a parameter included in the bitstream along with the rotation angle.

[0386] Figure 35C is a conceptual diagram illustrating another example of generating a generated picture. As illustrated in Figure 35C, a new image (PicA') generated by mirroring an image (PicA) may be used as a reference picture. In Figure 35C, the left-hand sample of the input image (PicA) is mirrored horizontally.

[0387] Mirroring is not limited to the example above; the sample on the right may be mirrored to the left, or vertical mirroring may be performed (mirroring the upper sample to the lower side, or mirroring the lower sample to the upper side). In this example, when mirroring, half of the width x of the input image (PicA) (or half of the height y in the case of vertical mirroring) is always mirrored, and the width / height of the mirroring is uniformly defined, but the width / height of the mirroring may also be specified by parameters.

[0388] Figure 35D is a conceptual diagram illustrating another example of generating a generated picture. As illustrated in Figure 35D, a new image (PicA') generated by shifting an image (PicA) may be used as the reference picture. In Figure 35D, the input image (PicA) is shifted to the right. In other words, the left side of the input image is padded and the right side is cropped. The shift is not limited to this example; the right side of the sample may be shifted to the left, or a vertical shift (shifted upwards or downwards) may be performed.

[0389] In the above, shifting the image to the right corresponds to shifting the image sample, i.e., the image content, to the right, and to shifting the processing range of the image to the left. Similarly, shifting the image to the left, up, and down corresponds to shifting the image sample, i.e., the image content, to the left, up, and down, respectively, and to shifting the processing range of the image to the right, down, and up, respectively.

[0390] Furthermore, multiple transformations may be combined. For example, an image (PicA) may be scaled to produce an image (Scaled PicA), and then a shifted image (Shifted PicA) may be generated.

[0391] [NN Loop Filter] The NN Loop Filter (NN LF: Neural Network Loop Filter) corresponds to a filtering process performed on an input image (such as a PicA) using a neural network (NN). The image generated by the NN LF may be used as a reference image for interpretation, or it may be used as the display image as is.

[0392] [NN Inter Prediction] NN Inter Prediction (Neural Network Inter Prediction) corresponds to inter prediction performed using a neural network (NN). Specifically, in NN inter prediction, one or more images (PicA and / or PicB, etc.) are used as input to the NN, and an image is generated as a generated image (PicAB', etc.). The generated image is then used as a reference image for inter prediction. Since the image is generated using an NN, the aforementioned NN processing may also be used.

[0393] The generated reference image used in NN interpretation is not an encoded or decoded image, and therefore may not contain motion information, reference image index, encoding information, or any combination thereof. Furthermore, since the generated prediction image is generated by predicting the current picture, it may have the same POC (Proof of Concept) as the current picture (CurrentPic).

[0394] Note that while this example shows the use of newly generated images using a neural network as an example of NN interpretation, the NN interpretation process is not limited to this example. For example, images to which NN LF processing has been applied may also be used.

[0395] [Translation Inter Prediction] Translation Inter Prediction corresponds to an inter prediction performed using a transformation process. Specifically, in translation inter prediction, a transformation process is applied to one or more images (PicA and / or PicB, etc.) to generate an image (PicAB', etc.). This generated image is then used as the reference image for inter prediction. The transformation process described above may also be used as the transformation process.

[0396] The generated reference image referenced in the conversion interface prediction is not an encoded or decoded image, and therefore may not contain motion information, reference image index, encoding information, or any combination thereof. Furthermore, since the generated prediction image is generated by predicting the current picture, it may have the same POC (Proof of Concept) as the current picture (CurrentPic).

[0397] [Prior Art Regarding Signaling of NNLF Parameter Sets in Bitstreams] JVET (Joint Video Expert Team) has been developing techniques for neural network loop filters (NNLFs) for the future development of video coding technology. Blocks contained in a picture (usually 128x128 pixels or 256x256 pixels, but other sizes are acceptable) are filtered by a pre-trained neural network loop filter and then stored in a buffer for the decoded image. The filtered signal is used as a reference image to predict future encoded images. In other words, an NNLF is a filter in a loop. NNLFs can be used with, or in place of, other loop filters commonly used in video coding standards such as deblocking filters (DBFs), adaptive loop filters (ALFs), or sample adaptive offsets (SAOs).

[0398] For example, the search software NNVC (Neural Network Video Coding) is based on the reference software VVC (Versatile Video Coding), with the addition of neural network (NN)-based technologies such as NNLF. The NNLF version currently under development in NNVC-v12 includes parameters divided into "picture parameters" and "slice parameters." The current "picture parameters" include information about the NNLF model used, the filtered block size, and the expanded size of each block, as well as information on whether to allow the application of QP slice values ​​as input to the NNLF. In addition, in the current NNVC-v12 design stage standard, the "picture parameters" are signaled (notified) by the Sequence Parameter Set (SPS). On the other hand, the current "slice parameters" include information about the scaling coefficients for each color component for combining the NN-filtered signal and the NN-unfiltered signal, and information about the offset for adjusting the NN-filtered signal. Also, in the current NNVC-v12 design stage standard, the "slice parameters" are signaled in the slice header of each slice.

[0399] In addition, in standards for conventional video coding technologies such as VVC, the SPS (Sequence Parameter Set) is designed to signal sequence-level information shared by all pictures within the bitstream. The SPS includes information that is not normally changed, such as profile, tier, level, picture format, coding, prediction, and transformation block structure. Generally, there is only one SPS for the entire video content, so it is not expected that the SPS will change frequently while the video is being viewed.

[0400] As described above, in VVC and similar systems, there is only one SPS for the entire video content. Therefore, when signaling information that applies only to a specific slice (i.e., a part of the image) within an image, the slice header is designed to be used. In VVC and similar systems, the slice header is typically designed to signal different information for different slices within the same image. Furthermore, in VVC and similar systems, information that applies to the entire image is signaled by the picture header, not the slice header.

[0401] Incidentally, the VVC standard defines a syntax structure called an Adaptation Parameter Set (APS). An APS is defined as a syntax structure containing syntax elements that apply to zero or more slices. Here, zero or more slices are determined by zero or more syntax elements contained in each slice header. While APS is typically designed to transmit information shared across multiple slices in multiple pictures within a sequence, it is also anticipated that it may change relatively frequently from picture to picture. Therefore, the VVC standard has pre-designed multiple types of APS to transmit different types of parameters. Each APS type can be used independently of other available types and is transmitted in its own Network Abstraction Layer (NAL) unit.

[0402] [Technical Challenges] In the currently under consideration NNVC-v12, some of the NNLF parameter sets, which are parameter sets related to the NNLF model, are applied to the entire bitstream or to multiple pictures within the bitstream, and therefore need to be signaled using a high-level syntax (HLS) structure. However, with the current method of signaling some of the NNLF parameter sets using SPS (Sequence Parameter Set), it is anticipated that filtering processes that change the NNLF parameter set for each group of frames cannot be properly performed. In fact, it is conceivable that the size of the filtering block or the number of QP adaptive parameters of the filter may need to be changed for each scene.

[0403] Furthermore, the current method of signaling some NNLF parameter sets in the slice header may not be efficient when the same NNLF parameter set is used throughout the entire picture, especially if the picture contains multiple slices or subpictures. In this case, information about the NNLF parameter set would be repeated at the beginning of each slice. In an even worse scenario, the NNLF parameter set might not be available in some slices of the picture, and those slices would depend on the slice containing the NNLF parameter set.

[0404] As described above, in the currently under consideration NNVC-v12, signaling on a picture-by-picture or slice-by-slice basis may be necessary for information regarding the NNLF parameter set, and a data structure that enables signaling on a picture-by-picture or slice-by-slice basis is required. The information regarding the NNLF parameter set may also be information regarding the NNLF model used. Furthermore, the information regarding the NNLF parameter set may also be information regarding the size of the filtered blocks and the expansion size of each block. Furthermore, the information regarding the NNLF parameter set may also be information regarding whether or not to allow the application of QP slice values ​​as input to the neural network. Furthermore, the information regarding the NNLF parameter set may also be information regarding the scaling coefficients of each color component for combining NN-filtered and non-NN-filtered signals. Furthermore, the information regarding the NNLF parameter set may also be information regarding the offset of the NN-filtered signal adjustment, etc.

[0405] Therefore, this disclosure relates to a proposed data unit that enables modification of NNLF parameter sets on a picture-by-picture or slice-by-slice basis. The data unit proposed in this disclosure is a data unit that can be composed solely of NNLF parameter sets and has a high-level syntactic structure. Furthermore, an NNLF parameter set corresponds to a "parameter set".

[0406] Figure 36 is a diagram illustrating the outline of the high-level syntax structure proposed in this disclosure.

[0407] The high-level syntax structure proposed in this disclosure uses a new type of APS that targets only NNLF data (NNLF parameter sets). For example, NNLF APS 0 to NNLF APS 2 shown in Figure 36 correspond to data units that can be composed solely of NNLF parameter sets. For information other than NNLF parameter sets, existing APS data structures (ALF APS 0 to ALF APS 2 in the example in Figure 36) are used. The high-level syntax structure proposed in this disclosure (NNLF APS 0 to NNLF APS 2 in the example in Figure 36) may be signaled before (prefix) or after (suffix) the corresponding picture data or slice data.

[0408] Furthermore, for high-level syntax structures, new NAL units, such as NAL units targeting only NNLF data, may be used. These NAL units may be signaled before (as a prefix) the corresponding picture data or slice data, or after (as a suffix) the slice data.

[0409] This disclosure describes a case in which either an APS, a data unit composed solely of NNLF data, or an NAL unit, a data unit composed solely of NNLF data, is used as a high-level syntax structure.

[0410] Furthermore, the slice header or picture header includes an identifier for APS or NAL units (specifically, the syntax element (aps_ids) shown in Figure 36) for signaling NNLF data applied to decoding the current slice or picture. For example, the syntax element signaled by PPS (specifically, pps_nnlf_info_ph_flag) is a syntax element for controlling whether the identifier for APS or NAL units is signaled in the picture header or the slice header. In this disclosure, filtering by NNLF processing is enabled on a picture-by-picture or slice-by-slice basis by signaling the identifier for APS or NAL units in the picture header or slice header based on such a syntax element signaled by PPS.

[0411] The picture header may also include an identifier for the APS or NAL unit (specifically, the syntax element (aps_ids) shown in Figure 36) to notify the NNLF data to be applied to the decoding of the current picture.

[0412] Furthermore, Figure 36 shows a case where three NNLF APS units numbered 0 to 2 are used as data units, but it is not limited to this. One or more NNLF APS units (or NAL units) may be used as data units. In particular, it is preferable that one to eight NNLF APS units (or NAL units) are used as data units.

[0413] Similarly, Figure 36 shows a case where three ALF APS units numbered 0 to 2 are used, but it is not limited to this. One or more ALF APS units may be used. It is preferable that one to eight ALF APS units are used as data units.

[0414] [Overview of Processing by the Encoding Device] Figure 37 is a flowchart showing an overview of the processing performed by the encoding device 100 of this disclosure. Figure 37 shows an example of processing for performing filtering by NNLF processing on a sub-picture or slice basis. A sub-picture is a processing unit obtained by dividing a picture, similar to a slice, but it is also a processing unit that includes multiple slices. In other words, a slice can be said to be a processing unit obtained by dividing a sub-picture. Furthermore, a sub-picture is a processing unit that can be decoded as an independent string.

[0415] First, the encoding device 100 determines the NNLF parameter set for the neural network loop filter model on a sub-picture or slice basis (S101). The NNLF parameter set determination process (S101) may include a process for searching for the RD (Rate-Distortion) cost.

[0416] Next, the encoding device 100 applies the neural network loop filter model corresponding to the determined NNLF parameter set on a sub-picture or slice basis (S102). For example, the encoding device 100 applies NNLF processing to a block within a sub-picture or a block within a slice using the neural network loop filter model.

[0417] Next, the encoding device 100 encodes the data unit containing the determined NNLF parameter set (S103). For example, the encoding device 100 writes the NNLF parameter set to a high-level syntax structure (specifically, a dedicated APS or NAL unit) as described in this disclosure, encodes the syntax structure, and includes the encoded syntax structure in the bitstream.

[0418] The encoding device 100 then encodes a first identifier corresponding to the determined NNLF parameter set into the slice header (S104). The first identifier is an identifier for identifying multiple neural network loop filter models. The encoding device 100 then terminates the process shown in Figure 37 by outputting a bitstream containing the high-level syntax structure of this disclosure.

[0419] Note that the order of processing performed by the encoding device 100 of this disclosure may differ from the order of processing illustrated in Figure 37. For example, the encoding device 100 may perform data unit encoding (S103) or first identifier encoding (S104) before applying NNLF processing (S102).

[0420] Furthermore, the encoding device 100 may determine a parameter set (NNLF parameter set) for the neural network loop filter model on a picture-by-picture basis. In such a case, the encoding device 100 encodes a first identifier corresponding to the NNLF parameter set necessary for applying NNLF processing in the picture header or another header, rather than in the slice header.

[0421] [Overview of Processing by Decryption Device] Figure 38 is a flowchart showing an overview of the processing performed by the decoding device 200 of this disclosure. Figure 38 shows an example of processing for performing filtering by NNLF processing on a sub-picture or slice basis.

[0422] First, the decoding device 200 decodes the data unit (high-level syntax structure) containing the NNLF parameter set necessary for filtering the current slice and the slice header (S201).

[0423] Next, the decoding device 200 reads a first identifier from the decoded slice header and determines a neural network loop filter model to apply to the current slice based on the first identifier and the data unit (S202). For example, the decoding device 200 determines the neural network loop filter model by reading the NNLF parameter set corresponding to the first identifier from the data unit.

[0424] Finally, the decoding device 200 applies the determined neural network loop filter model on a sub-picture or slice basis (S203). For example, the decoding device 200 applies NNLF processing to blocks within a sub-picture or blocks within a slice using the neural network loop filter model. The decoding device 200 then outputs the decoded and filtered slice, terminating the process shown in Figure 38.

[0425] In step S201, if the decoding device 200 decodes a picture header instead of a slice header, it may determine the parameters for the neural network loop filter model on a picture-by-picture basis. In such a case, the decoding device 200 applies the determined neural network loop filter model on a picture-by-picture basis.

[0426] [First Embodiment] In the first embodiment of this disclosure, information regarding the scale and offset of an NNLF parameter set can be signaled from multiple NNLF parameter sets within a referenced NNLF APS (Data Unit). Below, an example of the configuration of an NNLF APS is described, along with two specific examples of how to reference an NNLF APS.

[0427] Figure 39 is a diagram illustrating an example of the configuration of an NNLF APS in a first embodiment, and a first example of a method for referencing an NNLF APS. Figure 39 illustrates the case where the first identifier is signaled in the slice header.

[0428] As shown in Figure 39, an NNLF APS is referenced based on the syntax element sh_nnlf_aps_id shown in the slice header. In the example in Figure 39, "referenced" means using an NNLF APS that contains an identifier (for example, u(3) corresponding to the syntax element aps_adaptation_parameter_set_id in Figure 39) with the same value as a first identifier signaled in the slice header (for example, u(3) corresponding to the syntax element sh_nnlf_aps_id in Figure 39). For example, the encoding device 100 encodes a first identifier in the slice header that has the same value as the identifier used to identify the neural network loop filter model contained in the NNLF APS, so that the decoding device 200 can reference a particular NNLF APS. Furthermore, for example, the decoding device 200 determines a neural network loop filter model by referring to an NNLF APS containing an identifier that shows the same value as the first identifier signaled in the slice header, and applies the determined neural network loop filter model on a slice-by-slice or sub-picture-by-sub-picture basis.

[0429] Furthermore, as shown in Figure 39, the NNLF APS consists of a parameter set related to the neural network loop filter model. The following sections will describe each parameter set that constitutes the NNLF APS.

[0430] aps_adaptation_parameter_set_id is a syntax element that represents the ID of the NNLF APS.

[0431] aps_chroma_present_flag is a syntax element that indicates whether or not relevant information about the color difference component exists. Specifically, aps_chroma_present_flag is a flag used to specify the scale value and offset value for the color difference component in NNLF processing.

[0432] nnlf_model_id is a syntax element that specifies the identifier of the neural network loop filter model for NNLF processing.

[0433] nnlf_infer_size_base and nnlf_infer_size_ext are syntax elements that specify the patch size for NNLF processing. In other words, these syntax elements specify the size of the pixel blocks given as input data to or received as output data to a neural network loop filter model. The reason this information is necessary is that neural network loop filter models typically operate on a block basis. Specifically, nnlf_infer_size_base is a syntax element that specifies the base size of the output data obtained from the neural network loop filter model. On the other hand, nnlf_infer_size_ext is a syntax element that specifies the extension of the base size along the top, bottom, left, and right boundaries of the block that is input data to the neural network loop filter model.

[0434] The parameter set for the neural network loop filter model described above is merely an example. Therefore, other parameter sets for neural network loop filter models may also be included in NNLF APS.

[0435] As explained above, the aps_chroma_present_flag flag is used to specify the scale and offset values ​​for the chromatic difference component in NNLF processing. For example, if this flag is ON, the scale and offset values ​​for the chromatic difference component in NNLF processing may be applied at 1.5 times the normal value, and the scale and offset values ​​for the luminance component in NNLF processing may be applied at 0.8 times the normal value. In other words, if this flag is ON, it is possible to set the scale and offset values ​​for the chromatic difference component in NNLF processing to different values ​​from those for the luminance component in NNLF processing. On the other hand, if this flag is OFF, it is possible to apply NNLF processing with the scale and offset values ​​for the chromatic difference component being the same as those for the luminance component in NNLF processing.

[0436] Furthermore, as shown in Figure 39, the NNLF APS may contain the variable nnlfIdx. nnlfIdx contains multiple sub-parameter sets, which are parameter sets for defining the neural network loop filter model. For example, nnlfIdx may contain multiple sub-parameter sets for defining scale and offset, etc., associated with a specific identification number. The identification number associated with the sub-parameter set to be applied for each slice is signaled in the slice header, which shows the syntax element sh_nnlf_param_set_id. This sets a specific sub-parameter set from the multiple sub-parameter sets into the neural network loop filter model, and the neural network loop filter model is applied for each slice. In other words, the second identifier (specifically, ue(v)) associated with the syntax element sh_nnlf_param_set_id in Figure 39 represents the same value as the identification number associated with nnlfIdx.

[0437] The NNLF parameter set scale is a scaling factor applied in the form of a weighting between the signal filtered by the NNLF process and the original signal (including signals already filtered by other loop filtering processes such as deblocking filters), and is used to derive the final signal. Since the optimal scaling factor is inherently content-dependent, it may be beneficial to switch it per slice or picture. However, if the signals filtered by the NNLF process are similar, it is also conceivable that the NNLF parameter set scale be shared as a common value. Therefore, the data unit needs to have the flexibility to not only share the NNLF parameter set scale across different slices and / or pictures, but also to change it per slice.

[0438] The NNLF parameter set offset adjusts the values ​​of pixels filtered by the NNLF process. For example, the filtering value for each NNLF process can be changed by subtracting or adding a small offset. Note that, similar to the scaling factor in the NNLF parameter set, the optimal offset value depends on the content. Therefore, the optimal offset value may vary from slice to slice. However, if the filtered pixel values ​​are similar for the NNLF process, the same offset value may be maintained across different slices.

[0439] In the first embodiment, each of the multiple sub-parameter sets is assigned a second identifier. In Figure 39, the second identifiers corresponding to at least some of the multiple sub-parameter sets are encoded in the slice header. For example, the encoding device 100 encodes a second identifier in the slice header that has the same value as the identifier corresponding to a particular sub-parameter set, so that the decoding device 200 sets that specific sub-parameter set in the neural network loop filter model. Alternatively, for example, the decoding device 200 sets the sub-parameter set corresponding to the second identifier signaled in the slice header in the neural network loop filter model and applies the neural network loop filter model with the set sub-parameter set on a slice-by-slice basis.

[0440] Furthermore, the scale and offset of the NNLF parameter set described above are merely specific examples of subparameter sets. Therefore, subparameter sets may include information other than that described above.

[0441] Next, we will explain the syntax elements included in the slice header in Figure 39.

[0442] sh_nnlf_aps_id is a syntax element that represents the ID of the NNLF APS.

[0443] sh_nnlf_param_set_id is a syntax element that indicates the ID of the subparameter set.

[0444] The syntax elements described above are merely examples. Therefore, slice headers may contain syntax elements other than those listed above.

[0445] Figure 40 is a diagram illustrating an example of the configuration of an NNLF APS in the first embodiment, and a second example of a method for referencing an NNLF APS. Figure 40 illustrates the case where the first identifier is signaled in the picture header.

[0446] As shown in Figure 40, an NNLF APS is referenced using the syntax element ph_nnlf_aps_id shown in the picture header. In the example of Figure 40, "referenced" means using an NNLF APS that contains an identifier with the same value as the first identifier signaled in the picture header (for example, u(3) corresponding to the syntax element aps_adaptation_parameter_set_id in Figure 40). For example, the encoding device 100 encodes a first identifier in the picture header that has the same value as the identifier used to identify the neural network loop filter model contained in the NNLF APS, so that the decoding device 200 can reference a particular NNLF APS. Furthermore, for example, the decoding device 200 determines a neural network loop filter model by referring to an NNLF APS containing an identifier that shows the same value as the first identifier signaled in the picture header, and applies the determined neural network loop filter model on a picture-by-picture basis.

[0447] As shown in Figure 40, the NNLF APS consists of a parameter set related to the neural network loop filter model. The parameter set shown in Figure 40 is the same NNLF parameter set as the one described in Figure 39, so a detailed explanation will be omitted.

[0448] Also, as shown in Figure 40, the NNLF APS may include a variable nnlfIdx, and nnlfIdx may include a plurality of sub-parameter sets. The nnlfIdx shown in Figure 40 is also a variable similar to the nnlfIdx described with reference to Figure 39, and a detailed description thereof will be omitted. Note that in the example of Figure 40, the identification number associated with the sub-parameter set to be applied for each picture is signaled in the picture header in which the syntax element of ph_nnlf_param_set_id is indicated. Accordingly, a specific sub-parameter set among the plurality of sub-parameter sets is set in the neural network loop filter model, and the neural network loop filter model is applied for each picture. That is, it means that the second identifier (specifically, ue(v)) associated with the syntax element ph_nnlf_param_set_id in Figure 40 indicates the same value as the identification number associated with nnlfIdx.

[0449] Also, as shown in Figure 40, second identifiers corresponding to at least some of the plurality of sub-parameter sets are encoded in the picture header. For example, the encoding apparatus 100 encodes, in the picture header, a second identifier indicating the same value as the identifier corresponding to the sub-parameter set, such that a specific sub-parameter set is set in the neural network loop filter model by the decoding apparatus 200. Further, for example, the decoding apparatus 200 sets a sub-parameter set corresponding to the second identifier signaled in the picture header in the neural network loop filter model, and applies the neural network loop filter model in which the sub-parameter set is set on a picture-by-picture basis.

[0450] Next, the syntax elements included in the picture header of Figure 40 will be described.

[0451] ph_nnlf_aps_id is a syntax element representing the ID of the NNLF APS.

[0452] ph_nnlf_param_set_id is a syntax element indicating the ID of the sub-parameter set.

[0453] Note that the syntax elements described above are merely specific examples. Therefore, the picture header may include syntax elements other than those described above.

[0454] Furthermore, as an alternative to the data units described with reference to FIGS. 39 and 40, the data unit (high-level syntax structure) shown in FIG. 41 may be used. FIG. 41 is a diagram for explaining a configuration example of a NAL unit and an example of a NAL unit reference method according to the first embodiment. Note that FIG. 41 illustrates a case where the first identifier is signaled in a slice header.

[0455] As shown in FIG. 41, the NAL unit is referenced based on the syntax element sh_nnlf_id indicated in the slice header. Specifically, a NAL unit including an identifier (for example, u(3) corresponding to the syntax element nnlf_id in FIG. 41) having the same value as the first identifier signaled in the slice header (for example, u(3) corresponding to the syntax element sh_nnlf_id in FIG. 41) is used. For example, the encoding apparatus 100 encodes, into the slice header, the first identifier indicating the same value as the identifier for identifying the neural network loop filter model included in the NAL unit, such that the specific NAL unit is referenced by the decoding apparatus 200. Also, for example, the decoding apparatus 200 determines the neural network loop filter model by referencing the NAL unit including the identifier having the same value as the first identifier signaled in the slice header, and applies the determined neural network loop filter model on a slice-by-slice basis. Note that sh_nnlf_id is a syntax element representing the ID of the NAL unit.

[0456] Note that the parameter set shown in Figure 41 is the same as the parameter set explained in Figure 39, so a detailed explanation will be omitted. However, as shown in Figure 41, the syntax elements aps_adaptation_parameter_set_id and aps_chroma_present_flag are omitted in the NAL unit. Instead, nnlf_id is added to the parameter set shown in Figure 41. nnlf_id is a syntax element that represents the ID of the NAL unit.

[0457] Furthermore, as shown in Figure 41, the NAL unit may contain the variable nnlfIdx, and nnlfIdx may contain multiple sub-parameter sets. The identification number of the sub-parameter set to be applied for each slice is signaled in the slice header, which shows the syntax element sh_nnlf_param_set_id. For example, the encoding device 100 encodes a second identifier in the slice header that shows the same value as the identifier corresponding to the sub-parameter set, so that the decoding device 200 sets a specific sub-parameter set in the neural network loop filter model. Also, for example, the decoding device 200 sets the sub-parameter set corresponding to the second identifier signaled in the slice header in the neural network loop filter model, and applies the neural network loop filter model with the set sub-parameter set on a slice-by-slice basis.

[0458] Note that nnlfIdx shown in Figure 41 is the same variable as nnlfIdx explained in Figure 39, so a detailed explanation will be omitted.

[0459] Furthermore, even when the NAL unit shown in Figure 41 is used, the first identifier may be signaled in the picture header.

[0460] [Effects of the First Embodiment] As described above, the NNLF parameter set for the NNLF model (specifically, information such as the model identifier and patch size (filtering block size) used in NNLF processing) may be changed for each scene, each picture, or each slice, so it is conceivable that signaling on a picture-by-picture or slice-by-slice basis may be necessary. Therefore, in this disclosure, by using an APS, which is a data unit composed solely of the NNLF parameter set, or an NAL unit, which is a data unit composed solely of the NNLF parameter set, it may be possible to change the NNLF parameter set for each scene, each picture, or each slice.

[0461] Figure 42 is a conceptual diagram showing how the NNLF parameter set changes for each scene. As shown in Figure 42, for example, when using resampling (RPR) of a reference image, it is possible to use the NNLF parameter set defined by NNLF APS 0 in scene 1, the NNLF parameter set defined by NNLF APS 1 in scene 2, and the NNLF parameter set defined by NNLF APS 2 in scene 3. As in the example in Figure 42, it is effective to adapt the NNLF patch size and the model used in NNLF processing to match the different resolutions in the video sequence.

[0462] In the example shown in Figure 42, information such as the model identifier and patch size (filtering block size) used in NNLF processing is likely to be shared across multiple pictures. Also, in the example shown in Figure 42, the NNLF parameter set may be signaled in the slice header, but it is more often signaled in the picture header.

[0463] Figure 43 is a conceptual diagram showing how the NNLF parameter set changes for each picture. As shown in Figure 43, it is possible to use the NNLF parameter set defined by NNLF APS 0 for one picture (pic on the left) and the NNLF parameter set defined by NNLF APS N for another picture (pic on the right).

[0464] In the example shown in Figure 43, the NNLF parameter set may be signaled in the slice header, but it is more often signaled in the picture header.

[0465] Figure 44 is a conceptual diagram showing how the NNLF parameter set changes for each subpicture and slice. As shown in Figure 44, it is also possible to use the NNLF parameter set defined in NNLF APS 0 for subpic0, the NNLF parameter set defined in NNLF APS 1 for subpic1, and the NNLF parameter set defined in NNLF APS 2 for subpic2.

[0466] In the example shown in Figure 44, if information such as QP adaptation, scale, and offset may vary for different subpictures and slices within the same picture, signaling the NNLF parameter set in the slice header increases the likelihood of appropriate filtering being performed for each subpicture and slice.

[0467] Furthermore, if subsequent pictures have similar features, information such as QP adaptation, scale, and offset may be shared between some slices of different pictures. Therefore, in the first aspect of this disclosure, information that is likely to be shared between multiple slices or pictures may be signaled at the beginning of the data unit, while information that may change from slice to slice may be signaled in the slice header. Each slice header signals the NNLF parameter set for the neural network loop filter model applicable to that slice. As a result, if the same NNLF parameter set is used in common across multiple slices, in the first aspect of this disclosure, the same NNLF parameter set only needs to be signaled once at the beginning of the data unit, rather than being signaled at each slice header. This may allow for more flexible and efficient signaling of NNLF parameter sets in the first aspect of this disclosure.

[0468] [Second Aspect] The configuration of the data unit described in the second aspect of this disclosure differs from the configuration of the data unit described in the first aspect of this disclosure in that the sub-parameter set included in each data unit is uniquely determined for the neural network loop filter model. In other words, the configuration of the data unit described in the second aspect differs from the configuration of the data unit described in the first aspect in that it includes only one sub-parameter set. Below, a specific example of the configuration of the data unit in the second aspect will be described, as well as a method for referencing the data unit.

[0469] Figure 45 is a diagram illustrating an example of the configuration of an NNLF APS in the second embodiment, and a specific example of how to reference an NNLF APS. Figure 45 illustrates the case where the first identifier is signaled in the slice header.

[0470] As shown in Figure 45, an NNLF APS is referenced based on the syntax element sh_nnlf_aps_id shown in the slice header. Specifically, an NNLF APS is referenced if it contains an identifier (for example, u(3)) with the same value as a first identifier signaled in the slice header (for example, u(3)) corresponding to the syntax element sh_nnlf_aps_id in Figure 45). For example, the encoding device 100 encodes a first identifier in the slice header with the same value as an identifier for identifying a neural network loop filter model contained in a particular NNLF APS, so that the decoding device 200 can reference that particular NNLF APS. Furthermore, for example, the decoding device 200 determines a neural network loop filter model by referring to an NNLF APS containing an identifier with the same value as the first identifier signaled in the slice header, and applies the determined neural network loop filter model on a slice-by-slice basis.

[0471] Furthermore, as shown in Figure 45, the configuration of NNLF APS differs from the configuration of NNLF APS shown in Figure 39 in that the sub-parameter set is uniquely determined for the neural network loop filter model. For example, when using the NNLF APS shown in Figure 39, in order to switch the neural network loop filter model for each slice, it is necessary to encode a first identifier for identifying the neural network loop filter model and a second identifier corresponding to the sub-parameter set for defining the neural network loop filter model in the slice header. In contrast, when using the NNLF APS shown in Figure 45, since the sub-parameter set is uniquely determined for the neural network loop filter model, it is only necessary to encode the first identifier in the slice header. Therefore, while the slice header in Figure 39 encodes both the first and second identifiers, the slice header in Figure 45 encodes only the first identifier.

[0472] Although Figure 45 illustrates the case where the first identifier is signaled in the slice header, the first identifier may also be signaled in the picture header. Even when the first identifier is signaled in the picture header, the configuration example of the NNLF APS and the method of referencing the NNLF APS described above are the same, so a detailed explanation is omitted. Furthermore, when using the NNLF APS in the second embodiment, if the neural network loop filter model is switched for each picture, only the first identifier needs to be encoded in the picture header.

[0473] Furthermore, the syntax elements described within the NNLF APS shown in Figure 45 are the same as the syntax elements described within the NNLF APS explained in Figure 39. Similarly, the syntax elements described within the slice header shown in Figure 45 are the same as the syntax elements described within the slice header explained in Figure 39.

[0474] Furthermore, as an alternative to the data unit described in Figure 45, the data unit shown in Figure 46 (a high-level syntax structure) may be used. Figure 46 is a diagram illustrating an example of the configuration of a NAL unit in the second embodiment and an example of a method for referencing the NAL unit. In Figure 46, the case in which the first identifier is signaled in the slice header is explained.

[0475] As shown in Figure 46, NAL units are referenced based on the syntax element sh_nnlf_id shown in the slice header. Specifically, an NAL unit is referenced that contains an identifier (for example, u(3)) with the same value as the first identifier signaled in the slice header (for example, u(3)) corresponding to the syntax element sh_nnlf_id in Figure 46). For example, the encoding device 100 encodes a first identifier in the slice header with the same value as the identifier for identifying the neural network loop filter model contained in a particular NAL unit, so that the decoding device 200 can reference that particular NAL unit. Alternatively, for example, the decoding device 200 determines a neural network loop filter model by referring to an NAL unit that contains an identifier with the same value as the first identifier signaled in the slice header, and applies the determined neural network loop filter model on a slice-by-slice basis.

[0476] Furthermore, as shown in Figure 46, the configuration of the NAL unit differs from that of the NAL unit shown in Figure 41 in that the sub-parameter set is uniquely determined for the neural network loop filter model. For example, when the NAL unit shown in Figure 41 is used, in order to switch the neural network loop filter model for each slice, it is necessary to encode a first identifier for identifying the neural network loop filter model and a second identifier corresponding to the sub-parameter set for defining the neural network loop filter model in the slice header. In contrast, when the NAL unit shown in Figure 46 is used, since the sub-parameter set is uniquely determined for the neural network loop filter model, it is only necessary to encode the first identifier in the slice header. Therefore, the slice header shown in Figure 41 encodes both the first and second identifiers, whereas the slice header shown in Figure 46 encodes only the first identifier.

[0477] Although Figure 46 illustrates the case where the first identifier is signaled in the slice header, the first identifier may also be signaled in the picture header. Even when the first identifier is signaled in the picture header, the configuration example of the NAL unit and the method of referencing the NAL unit are the same as described above, so a detailed explanation is omitted. Furthermore, when switching the neural network loop filter model for each picture in the case where the NAL unit in the second embodiment is used, only the first identifier needs to be encoded in the picture header.

[0478] Furthermore, the syntax elements described within the NAL unit shown in Figure 46 are the same as the syntax elements described within the NAL unit in Figure 41. Similarly, the syntax elements described within the slice header shown in Figure 46 are the same as the syntax elements described within the slice header in Figure 41.

[0479] [Effects of the Second Embodiment] As described above, the data unit in the second embodiment contains only one set of subparameters. In other words, the data unit in the second embodiment contains a subparameter set that is uniquely determined for the neural network loop filter model. This makes the configuration of the data unit in the second embodiment simpler than the configuration of the data unit in the first embodiment. For example, in cases where the NNLF parameter set is changed for each scene as shown in Figure 42, if a group of pictures in the same scene share the same NNLF parameter set, the amount of information encoded in the picture header can be reduced, and signaling may become even more efficient.

[0480] Furthermore, as shown in Figure 43, when the NNLF parameter set is changed for each picture, information such as QP adaptation, scale, and offset may change for each picture. Similarly, as shown in Figure 44, when the NNLF parameter set is changed for each sub-picture and slice, information such as QP adaptation, scale, and offset may change for different sub-pictures and slices within the same picture. However, since this information may be shared among multiple slices or sub-pictures within a picture, signaling this information in the data unit in the second embodiment reduces the frequency of information transmission and enables out-of-band transmission.

[0481] [Third Aspect] The configuration of the data unit described in the third aspect of this disclosure differs from the configuration of the data unit described in the second aspect of this disclosure in that a portion of the NNLF parameter set contained in each data unit is overwritten by information contained in the slice header or picture header. Below, a specific example of the configuration of the data unit in the third aspect will be described, as well as a method for referencing the data unit.

[0482] Figure 47 illustrates an example of the NNLF APS configuration in the third embodiment, and a specific example of how to reference the NNLF APS. Figure 47 illustrates the case where the first identifier is signaled in the slice header.

[0483] As shown in Figure 47, the NNLF APS is referenced based on the syntax element sh_nnlf_aps_id indicated in the slice header. Specifically, an NNLF APS including an identifier having the same value as the first identifier signaled in the slice header (for example, u(3) corresponding to the syntax element sh_nnlf_aps_id in Figure 47) (for example, u(3) corresponding to the syntax element aps_adaptation_parameter_set_id in Figure 47) is referenced. For example, the encoding device 100 encodes, into the slice header, a first identifier having the same value as the identifier for identifying the neural network loop filter model included in the NNLF APS, such that the specific NNLF APS is referenced by the decoding device 200. Also, for example, the decoding device 200 determines the neural network loop filter model with reference to the NNLF APS including the identifier having the same value as the first identifier signaled in the slice header, and applies the determined neural network loop filter model on a slice basis.

[0484] Furthermore, in the configuration of the NNLF APS shown in Figure 47, similarly to the configuration of the NNLF APS in the second aspect, the sub-parameter set is uniquely determined for the neural network loop filter model.

[0485] Here, the sub-parameter set included in the NNLF APS shown in Figure 47 may be overwritten (rewritten) based on information signaled in the slice header. For example, as shown in Figure 47, a rewriting sub-parameter set for rewriting part or all of the plurality of sub-parameter sets included in the NNLF APS is included in the slice header. The rewriting sub-parameter set may include a plurality of sub-parameter sets related to scales, offset parameters, and the like. In addition, the rewriting sub-parameter set may include information that specifies a sub-parameter set to be rewritten among the plurality of sub-parameter sets included in the NNLF APS.

[0486] For example, when the encoding device 100 encodes the first identifier into the slice header, it encodes the rewrite subparameter set into the slice header. Also, for example, when the decoding device 200 decodes the first identifier and the rewrite subparameter set from the slice header, it rewrites part or all of the subparameter set into the rewrite subparameter set and sets the rewritten subparameter set in the neural network loop filter model corresponding to the first identifier. Then, the decoding device 200 applies the neural network loop filter model with the rewritten subparameter set set in slice units.

[0487] Furthermore, the rewrite subparameter set may also be data representing the difference from the subparameter set included in the NNLF APS. For example, if the rewrite subparameter set is data representing the difference from the subparameter set included in the NNLF APS, the resulting subparameter set, obtained by adding this difference to the subparameter set included in the NNLF APS, will be set in the neural network loop filter model.

[0488] Furthermore, the slice header may include a flag (for example, the syntax element sh_nnlf_overwrite_param_flag in Figure 47) that specifies whether to overwrite some or all of the subparameter sets included in the NNLF APS with the rewrite subparameter set. This flag may also specify that syntax elements with the same values ​​as the syntax elements in the subparameter sets included in the NNLF APS should be used, or it may specify that syntax elements with different values ​​than the syntax elements in the subparameter sets included in the NNLF APS should be used.

[0489] In Figure 47, the case in which the first identifier and the rewritable sub-parameter set are signaled in the slice header is explained, but the first identifier and the rewritable sub-parameter set may also be signaled in the picture header. Even when the first identifier and the rewritable sub-parameter set are signaled in the picture header, the configuration example of the NNLF APS and the method of referencing the NNLF APS described above are the same, so a detailed explanation will be omitted. For example, when the encoding device 100 encodes the first identifier in the picture header, it encodes the rewritable sub-parameter set in the picture header. Also, for example, when the decoding device 200 decodes the first identifier and the rewritable sub-parameter set from the picture header, it rewrites part or all of the sub-parameter set into the rewritable sub-parameter set and sets the rewritten sub-parameter set in the neural network loop filter model corresponding to the first identifier. Then, the decoding device 200 applies the neural network loop filter model with the rewritten sub-parameter set set on a picture-by-picture basis.

[0490] Furthermore, the rewrite subparameter set encoded in the slice header only applies to the slice being processed that is signaled in the slice header. Similarly, the rewrite subparameter set encoded in the picture header only applies to the picture being processed that is signaled in the picture header. In other words, the subparameter set included in the NNLF APS is the default subparameter set.

[0491] Furthermore, the syntax elements described within the NNLF APS shown in Figure 47 are the same as the syntax elements described within the NNLF APS explained in Figure 39. Also, the syntax elements described within the slice header shown in Figure 47 (specifically, syntax elements other than sh_nnlf_overwrite_param_flag) are the same as the syntax elements described within the slice header explained in Figure 39.

[0492] Furthermore, as an alternative to the data unit described in Figure 47, the data unit shown in Figure 48 (a high-level syntax structure) may be used. Figure 48 is a diagram illustrating an example of the configuration of a NAL unit in the third embodiment and an example of a method for referencing the NAL unit. In Figure 48, the case in which the first identifier is signaled in the slice header is explained.

[0493] As shown in Figure 48, NAL units are referenced based on the syntax element sh_nnlf_id shown in the slice header. Specifically, an NAL unit is referenced that contains an identifier (for example, u(3)) with the same value as the first identifier signaled in the slice header (for example, u(3)) corresponding to the syntax element sh_nnlf_id in Figure 48). For example, the encoding device 100 encodes a first identifier in the slice header with the same value as the identifier for identifying the neural network loop filter model contained in a particular NAL unit, so that the decoding device 200 can reference that specific NAL unit. Alternatively, for example, the decoding device 200 determines a neural network loop filter model by referring to an NAL unit that contains an identifier with the same value as the first identifier signaled in the slice header, and applies the determined neural network loop filter model on a slice-by-slice basis.

[0494] Furthermore, the configuration of the NAL unit shown in Figure 48 is similar to the configuration of the NAL unit in the second embodiment, in which the sub-parameter set is uniquely determined for the neural network loop filter model.

[0495] Here, the subparameter sets included in the NAL unit shown in Figure 48 can be overwritten (rewritten) based on information signaled in the slice header. For example, as shown in Figure 48, the slice header includes a subparameter set for rewriting some or all of the multiple subparameter sets included in the NAL unit. The rewriting subparameter set may include multiple subparameter sets related to scale and offset parameters, etc. The rewriting subparameter set may also include information that specifies which of the multiple subparameter sets included in the NAL unit is to be rewritten.

[0496] For example, when the encoding device 100 encodes the first identifier into the slice header, it encodes the rewrite subparameter set into the slice header. Also, for example, when the decoding device 200 decodes the first identifier and the rewrite subparameter set from the slice header, it rewrites some or all of the subparameter set into the rewrite subparameter set and sets the rewritten subparameter set in the neural network loop filter model corresponding to the first identifier. Then, the decoding device 200 applies the neural network loop filter model with the rewritten subparameter set set in slice units.

[0497] Furthermore, the rewriteable sub-parameter set may be data representing the difference from the sub-parameter set included in the NAL unit. For example, if the rewriteable sub-parameter set is data representing the difference from the sub-parameter set included in the NAL unit, the resulting sub-parameter set, obtained by adding this difference to the sub-parameter set included in the NAL unit, will be set in the neural network loop filter model.

[0498] The slice header may include a flag (syntax element sh_nnlf_overwrite_param_flag in Figure 48) that specifies whether to overwrite some or all of the subparameter sets included in the NAL unit with a rewrite subparameter set. This flag may also specify that syntax elements with the same values ​​as those included in the NAL unit should be used, or it may specify that syntax elements with different values ​​than those in the subparameter set included in the NAL unit should be used.

[0499] Although Figure 48 illustrates the case where the first identifier and the rewritable sub-parameter set are signaled in the slice header, the first identifier and the rewritable sub-parameter set may also be signaled in the picture header. Even when the first identifier and the rewritable sub-parameter set are signaled in the picture header, the configuration example of the NAL unit and the method of referencing the NAL unit described above are the same, so a detailed explanation will be omitted. For example, when the encoding device 100 encodes the first identifier in the picture header, it encodes the rewritable sub-parameter set in the picture header. Also, for example, when the decoding device 200 decodes the first identifier and the rewritable sub-parameter set from the picture header, it rewrites some or all of the sub-parameter set into the rewritable sub-parameter set and sets the rewritten sub-parameter set in the neural network loop filter model corresponding to the first identifier. The decoding device 200 then applies the neural network loop filter model with the rewritten sub-parameter set set on a picture-by-picture basis.

[0500] Furthermore, the rewrite subparameter set encoded in the slice header only applies to the slice being processed that is signaled in the slice header. Similarly, the rewrite subparameter set encoded in the picture header only applies to the picture being processed that is signaled in the picture header. In other words, the subparameter set included in the NAL unit is the default subparameter set.

[0501] Furthermore, the syntax elements described within the NAL unit shown in Figure 48 are the same as the syntax elements described within the NAL unit in Figure 41. Also, the syntax elements described within the slice header shown in Figure 48 (specifically, syntax elements other than sh_nnlf_overwrite_param_flag) are the same as the syntax elements described within the slice header in Figure 41.

[0502] [Effects of the Third Embodiment] While the scale values ​​of subparameter sets typically change with each slice, the scale values ​​in different slices of the same picture, or in similar pictures within the same sequence, tend to be very close to each other. Therefore, it may be more efficient to signal default values ​​in the data unit. In a third embodiment of this disclosure, a data unit (NNLF APS or NAL unit) is used to signal default values ​​for the scale and offset of the subparameter set, and a rewrite subparameter set for overriding the scale and offset values ​​of the subparameter set is signaled in the slice header. This allows the subparameter set to be changed for each slice or for each subpicture as needed. Furthermore, if changing the subparameter set for each slice is not necessary, the same values ​​can be used for the entire picture or for a group of pictures in the same scene by signaling the rewrite subparameter set in the picture header. In other words, the subparameter set can be changed for each picture or for each scene.

[0503] Furthermore, if information regarding the difference from the default value is signaled as a rewrite sub-parameter, the amount of information encoded in the slice header or picture header is reduced, which can make signaling more efficient.

[0504] [Fourth Embodiment] The configuration of the data unit described in the fourth embodiment of this disclosure differs from the configuration of the data unit described in the first embodiment in that it does not include multiple subparameter sets for defining a neural network loop filter model. In the fourth embodiment, instead of including subparameter sets in the data unit, one set of subparameter sets is encoded in a slice header or picture header and signaled. Below, a specific example of the configuration of the data unit in the fourth embodiment will be described, as well as a method for referencing the data unit.

[0505] Figure 49 is a diagram illustrating an example of the configuration of an NNLF APS in the fourth embodiment, and a specific example of how to reference an NNLF APS. In Figure 49, the case in which the first identifier is signaled in the slice header is explained.

[0506] As shown in Figure 49, an NNLF APS is referenced based on the syntax element sh_nnlf_aps_id shown in the slice header. Specifically, an NNLF APS is referenced if it contains an identifier (for example, u(3)) with the same value as a first identifier signaled in the slice header (for example, u(3)) corresponding to the syntax element sh_nnlf_aps_id in Figure 49). For example, the encoding device 100 encodes a first identifier in the slice header with the same value as an identifier for identifying a neural network loop filter model contained in a particular NNLF APS, so that the decoding device 200 can reference that particular NNLF APS. Furthermore, for example, the decoding device 200 determines a neural network loop filter model by referring to an NNLF APS containing an identifier with the same value as the first identifier signaled in the slice header, and applies the determined neural network loop filter model on a slice-by-slice basis.

[0507] Furthermore, the NNLF APS shown in Figure 49 does not include a subparameter set (specifically, the parameter set corresponding to nnlfIdx in Figure 39), which is a parameter set for defining the neural network loop filter model. On the other hand, in the fourth embodiment, as shown in Figure 49, the slice header may include additional syntax elements for signaling the subparameter set. For example, as shown in Figure 49, the slice header may include a subparameter set for defining the scale and offset. For example, when the encoding device 100 encodes the first identifier into the slice header, it encodes the subparameter set into the slice header. Also, for example, when the decoding device 200 decodes the first identifier and the subparameter set from the slice header, it sets the subparameter set into the neural network loop filter model corresponding to the first identifier. The decoding device 200 then applies the neural network loop filter model with the set subparameter set on a slice-by-slice basis.

[0508] Furthermore, as shown in Figure 49, the slice header may include a flag (for example, sh_nnlf_param_flag in Figure 49) that specifies whether or not to set the subparameter set encoded in the slice header in the neural network loop filter model. For example, if the flag specifies that the subparameter set should be set in the neural network loop filter model, the subparameter set encoded in the slice header is set in the neural network loop filter model corresponding to the first identifier, and that neural network loop filter model is applied on a slice-by-slice basis. On the other hand, if the flag specifies that the subparameter set should not be set in the neural network loop filter model, the neural network loop filter model corresponding to the first identifier is kept at its default value, and that neural network loop filter model is applied on a slice-by-slice basis.

[0509] Although Figure 49 illustrates the case where the first identifier and subparameter set are signaled in the slice header, the first identifier and subparameter set may also be signaled in the picture header. Even when the first identifier and subparameter set are signaled in the picture header, the configuration example of the NNLF APS and the method of referencing the NNLF APS described above are the same, so a detailed explanation will be omitted. For example, when the encoding device 100 encodes the first identifier in the picture header, it encodes the subparameter set in the picture header. Also, for example, when the decoding device 200 decodes the first identifier and subparameter set from the picture header, it sets the subparameter set in the neural network loop filter model corresponding to the first identifier and applies the neural network loop filter model with the set subparameter set on a picture-by-picture basis.

[0510] Furthermore, the syntax elements described within the NNLF APS shown in Figure 49 are the same as those described within the NNLF APS in Figure 39. Also, the syntax elements described within the slice header shown in Figure 49 (specifically, syntax elements other than sh_nnlf_param_flag) are the same as those described within the slice header in Figure 39.

[0511] Furthermore, as an alternative to the data unit described in Figure 49, the data unit shown in Figure 50 (a high-level syntax structure) may be used. Figure 50 is a diagram illustrating an example of the configuration of an NAL unit in the fourth embodiment and an example of a method for referencing an NAL unit. Figure 50 illustrates the case where the first identifier is signaled in a slice header.

[0512] As shown in Figure 50, NAL units are referenced based on the syntax element nnlf_id indicated in the slice header. Specifically, an NAL unit is referenced that contains an identifier (for example, u(3)) with the same value as the first identifier signaled in the slice header (for example, u(3)) corresponding to the syntax element sh_nnlf_id in Figure 50). For example, the encoding device 100 encodes a first identifier in the slice header with the same value as the identifier for identifying the neural network loop filter model contained in a particular NAL unit, so that the decoding device 200 can reference that specific NAL unit. Alternatively, for example, the decoding device 200 determines a neural network loop filter model by referring to an NAL unit that contains an identifier with the same value as the first identifier signaled in the slice header, and applies the determined neural network loop filter model on a slice-by-slice basis.

[0513] Furthermore, the NAL unit shown in Figure 50 does not include a subparameter set (specifically, the parameter set corresponding to nnlfIdx in Figure 39), which is a parameter set for defining the neural network loop filter model. On the other hand, in the fourth embodiment, as shown in Figure 50, the slice header may include additional syntax elements for signaling the subparameter set. For example, as shown in Figure 50, the slice header may include a subparameter set for defining scale and offset. For example, when the encoding device 100 encodes the first identifier into the slice header, it encodes the subparameter set into the slice header. Also, for example, when the decoding device 200 decodes the first identifier and the subparameter set from the slice header, it sets the subparameter set into the neural network loop filter model corresponding to the first identifier. The decoding device 200 then applies the neural network loop filter model with the set subparameter set on a slice-by-slice basis.

[0514] Furthermore, as shown in Figure 50, the slice header may include a flag (for example, sh_nnlf_param_flag in Figure 50) that specifies whether or not to set the subparameter set encoded in the slice header in the neural network loop filter model. For example, if the flag specifies that the subparameter set should be set in the neural network loop filter model, the subparameter set encoded in the slice header is set in the neural network loop filter model corresponding to the first identifier, and the neural network loop filter model is applied on a slice-by-slice basis. On the other hand, if the flag specifies that the subparameter set should not be set in the neural network loop filter model, the neural network loop filter model corresponding to the first identifier is maintained at its default value, and the neural network loop filter model is applied on a slice-by-slice basis.

[0515] Although Figure 50 illustrates the case where the first identifier and subparameter set are signaled in the slice header, the first identifier and subparameter set may also be signaled in the picture header. Even when the first identifier and subparameter set are signaled in the picture header, the configuration example of the NAL unit and the method of referencing the NAL unit described above are the same, so a detailed explanation will be omitted. For example, when the encoding device 100 encodes the first identifier in the picture header, it encodes the subparameter set in the picture header. Also, for example, when the decoding device 200 decodes the first identifier and subparameter set from the picture header, it sets the subparameter set in the neural network loop filter model corresponding to the first identifier and applies the neural network loop filter model with the set subparameter set on a picture-by-picture basis.

[0516] Furthermore, the syntax elements described within the NAL unit shown in Figure 50 are the same as the syntax elements described within the NAL unit in Figure 41. Also, the syntax elements described within the slice header shown in Figure 50 (specifically, syntax elements other than sh_nnlf_param_flag) are the same as the syntax elements described within the slice header in Figure 41.

[0517] [Effects of the Fourth Aspect] In the fourth aspect of this disclosure, NNLF parameter sets (specifically, sub-parameter sets) that may vary from slice to slice are signaled in the slice header or picture header, rather than being signaled at all in the data unit. This allows the NNLF parameter sets to be changed for each slice, each sub-picture, or each picture.

[0518] Furthermore, NNLF parameter sets that may be shared between different pictures or slices (specifically, parameter sets that specify the patch size for NNLF processing) are signaled in the data unit. This eliminates the need to encode parameter sets that are repeatedly used between pictures or slices for each slice header or picture header, and maintains the flexibility to change NNLF parameter sets for each scene without retransmitting the entire SPS.

[0519] [Combination with other aspects] This can be done by combining at least some of the syntax elements relating to the data units and slice headers or picture headers described in each aspect of this disclosure. Furthermore, it can also be done by combining the configurations of the data units described in each aspect of this disclosure. Similarly, it can also be done by combining the configurations of the slice headers or picture headers described in each aspect of this disclosure.

[0520] It should be noted that not all of the syntax elements relating to the data units and slice headers or picture headers described in each aspect of this disclosure are necessarily required, and only some of the syntax elements may be included in the data units and slice headers or picture headers in each aspect.

[0521] [Representative Examples] Representative examples of the encoding and decoding operations described above are shown below. For example, the encoding operation is performed by the encoding device 100, and the decoding operation is performed by the decoding device 200.

[0522] Figure 51 is a flowchart showing the basic processing in the encoding operation performed by the encoding device 100. For example, the encoding device 100 includes a circuit and a memory connected to the circuit. The circuit and memory of the encoding device 100 may correspond to the processor a1 and memory a2 shown in Figure 2. In operation, the circuit of the encoding device 100 performs the processing shown in Figure 51.

[0523] Specifically, the circuit of the encoding device 100 encodes a data unit (S301). The data unit encoded in step S301 is a data unit that includes a parameter set for each of the multiple neural network loop filter models, and is a data unit that can be composed solely of that parameter set.

[0524] Next, the circuit of the encoding device 100 determines whether or not the neural network loop filter model has been applied on a picture-by-picture basis (S302).

[0525] Next, if the encoding device 100 determines that the neural network loop filter model has been applied on a picture-by-picture basis (Yes in step S302), it encodes a first identifier for identifying multiple neural network loop filter models into the picture header (S303). On the other hand, if the encoding device 100 determines that the neural network loop filter model has not been applied on a picture-by-picture basis (No in step S302), it encodes a first identifier for identifying multiple neural network loop filter models into the slice header (S304).

[0526] This may make it possible to transmit to the decoding device 200 that the neural network loop filter model is switching for each slice, sub-picture, picture, or scene, using a dedicated data unit that can be composed solely of the parameter set for the neural network loop filter model. Furthermore, since a dedicated data unit is used for each neural network loop filter model, it may be possible to transmit to the decoding device 200 more flexibly that the neural network loop filter model is switching.

[0527] Furthermore, for example, the data unit may be an APS (Adaptation Parameter Set).

[0528] As a result, since a dedicated APS is used for each neural network loop filter model, it may be possible to more flexibly transmit to the decoding device 200 that the neural network loop filter model is switched for each slice, each sub-picture, each picture, or each scene.

[0529] Furthermore, for example, the data unit may be a NAL (Network Abstraction Layer) unit.

[0530] As a result, since a dedicated NAL unit is used for each neural network loop filter model, it may be possible to more flexibly transmit to the decoding device 200 that the neural network loop filter model is switched for each slice, each subpicture, each picture, or each scene.

[0531] Furthermore, for example, if the circuit encodes a first identifier in the slice header, the data unit may be signaled before or after the picture data or slice data.

[0532] This makes it possible to signal information that is likely to be shared between multiple slices or pictures before or after the picture data or slice data, potentially reducing the number of signaling events related to such information. Consequently, it may be possible to more efficiently transmit information about the switching of the neural network loop filter model to the decoding device 200.

[0533] Furthermore, for example, a parameter set for a neural network loop filter model may include multiple sub-parameter sets, which are parameter sets for defining the neural network loop filter model. Each of these sub-parameter sets may also be assigned a second identifier. Additionally, in operation, the circuit may, when encoding the first identifier in the picture header, encode the second identifiers corresponding to at least some of the sub-parameter sets in the picture header, and when encoding the first identifier in the slice header, encode the second identifiers corresponding to at least some of the sub-parameter sets in the slice header.

[0534] This may allow the decoding device 200 to receive information that the sub-parameter set for the neural network loop filter model is switching for each slice, each sub-picture, each picture, or each scene.

[0535] Furthermore, for example, the parameter set for a neural network loop filter model may include a subparameter set, which is a parameter set for defining the neural network loop filter model. The subparameter set may also be a parameter set that is uniquely determined for the neural network loop filter model.

[0536] This can simplify the configuration of data units. Therefore, when the parameter set changes from scene to scene, if a group of pictures in the same scene share the same parameter set, the amount of information to encode in the picture header can be reduced, and signaling may become more efficient.

[0537] Furthermore, when the parameter set is changed for each picture or sub-picture, information such as QP adaptation, scale, and offset may change for each picture or sub-picture. However, since this information may be shared across multiple slices or sub-pictures within a picture, signaling this information with the data unit mentioned above can reduce the frequency of information transmission and may even enable out-of-band transmission.

[0538] Furthermore, for example, the parameter set for a neural network loop filter model may include a subparameter set that defines the neural network loop filter model. In addition, the circuit may, in its operation, further encode a rewrite subparameter set in the picture header to rewrite a portion of the subparameter set when encoding the first identifier in the picture header, or encode a rewrite subparameter set in the slice header when encoding the first identifier in the slice header.

[0539] This may allow the decoding device 200 to receive information that some of the sub-parameter sets included in the data unit have been rewritten, on a slice-by-slice, sub-picture-by-subpicture, picture-by-picture, or scene-by-scene basis.

[0540] Furthermore, for example, in operation, the circuit may also encode a subparameter set, which is a set of parameters for defining a neural network loop filter model, into the picture header if it encodes the first identifier into the picture header, or encode the subparameter set into the slice header if it encodes the first identifier into the slice header.

[0541] This may allow parameter sets that may vary from slice to slice or picture to be transmitted to the decoder 200 in the slice header or picture header, and parameter sets that may be shared between different pictures or slices to be transmitted to the decoder 200 in the data unit. Therefore, it is no longer necessary to encode parameter sets that are repeatedly used between slices or pictures in each slice header or picture header, and flexibility is maintained such that parameter sets can be changed for each scene without retransmitting the entire SPS.

[0542] Furthermore, for example, the parameter set for the neural network loop filter model may include a subparameter set, which is a parameter set for defining the neural network loop filter model. The subparameter set may also include a parameter set for defining the output format for the luminance component and the chrominance component included in the output data obtained by applying the neural network loop filter model to the current block. In addition, the circuit may, in its operation, encode a flag in the picture header to identify whether or not to set a subparameter set for the neural network loop filter model if the first identifier is encoded in the picture header, or encode a flag in the slice header for the neural network loop filter model if the first identifier is encoded in the slice header.

[0543] This makes it possible to transmit the output format for the luminance component and the output format for the color difference component included in the output data to the decoding device 200 via the slice header or picture header, even when these formats are changed for each slice or picture.

[0544] Furthermore, for example, the loop filter section 120 of the encoding device 100 may perform the operations described above as part of the circuitry of the encoding device 100.

[0545] Figure 52 is a flowchart showing the basic processing in the decoding operation performed by the decoding device 200. For example, the decoding device 200 includes a circuit and a memory connected to the circuit. The circuit and memory of the decoding device 200 may correspond to the processor b1 and memory b2 shown in Figure 5. In operation, the circuit of the decoding device 200 performs the processing shown in Figure 52.

[0546] Specifically, the circuit of the decoding device 200 decodes the data unit (S401). The data unit decoded in step S401 is a data unit that includes a parameter set for each of the multiple neural network loop filter models, and is a data unit that can be composed solely of that parameter set. In addition, in step S401, the circuit of the decoding device 200 may decode multiple data units.

[0547] Next, the circuit of the decoding device 200 decodes a first identifier for identifying multiple neural network loop filter models from the picture header or slice header (S402).

[0548] This may allow switching between neural network loop filter models on a slice-by-slice, sub-picture-by-subpicture, picture-by-picture, or scene-by-scene basis, using a dedicated data unit composed solely of parameter sets related to the neural network loop filter model. Furthermore, since a dedicated data unit is used for each neural network loop filter model, it may be possible to switch between neural network loop filter models more flexibly.

[0549] Furthermore, for example, the data unit may be an APS (Adaptation Parameter Set).

[0550] This allows for the use of a dedicated APS for each neural network loop filter model, potentially enabling more flexible switching between neural network loop filter models on a slice-by-slice, sub-picture-by-subpicture, picture-by-picture, or scene-by-scene basis.

[0551] Furthermore, for example, the data unit may be a NAL (Network Abstraction Layer) unit.

[0552] This allows for the use of a dedicated NAL unit for each neural network loop filter model, potentially enabling more flexible switching between neural network loop filter models on a slice-by-slice, sub-picture-by-subpicture, picture-by-picture, or scene-by-scene basis.

[0553] Furthermore, for example, if the circuit decodes the first identifier from the slice header, the data unit may be signaled before or after the picture data or slice data.

[0554] This means that information likely to be shared across multiple slices or pictures is signaled before or after the picture or slice data, potentially reducing the number of signaling events related to that information. Consequently, the switching of the neural network loop filter model may be transmitted more efficiently.

[0555] Furthermore, for example, a parameter set relating to a neural network loop filter model may include multiple sub-parameter sets, which are parameter sets for defining the neural network loop filter model. A second identifier may also be assigned to each of these sub-parameter sets. Additionally, in operation, the circuit may, when decoding the first identifier from the picture header, decode the second identifiers corresponding to at least some of the sub-parameter sets from the picture header, and when decoding the first identifier from the slice header, decode the second identifiers corresponding to at least some of the sub-parameter sets from the slice header.

[0556] This may allow you to switch the set of subparameters set in the neural network loop filter model for each slice, each subpicture, each picture, or each scene, and then perform the processing.

[0557] Furthermore, for example, the parameter set for a neural network loop filter model may include a subparameter set, which is a parameter set for defining the neural network loop filter model. The subparameter set may also be a parameter set that is uniquely determined for the neural network loop filter model.

[0558] This can simplify the configuration of data units. Therefore, when the parameter set changes from scene to scene, if a group of pictures in the same scene share the same parameter set, the amount of information to decode from the picture header can be reduced, and signaling may become more efficient.

[0559] Furthermore, when the parameter set is changed for each picture or sub-picture, information such as QP adaptation, scale, and offset may change for each picture or sub-picture. However, since this information may be shared across multiple slices or sub-pictures within a picture, signaling this information with the data unit mentioned above can reduce the frequency of information transmission and may even enable out-of-band transmission.

[0560] Furthermore, for example, the parameter set for a neural network loop filter model may include a subparameter set, which is a parameter set for defining the neural network loop filter model. In addition, the circuit may, in its operation, decode a rewrite subparameter set from the picture header to rewrite a portion of the subparameter set when decoding the first identifier from the picture header, and decode a rewrite subparameter set from the slice header when decoding the first identifier from the slice header.

[0561] This may allow processing to be performed by rewriting some of the sub-parameter sets contained in the data unit, on a slice-by-slice, sub-picture-by-subpicture, picture-by-picture, or scene-by-scene basis.

[0562] Furthermore, for example, in operation, the circuit may also decode a subparameter set, which is a set of parameters for defining a neural network loop filter model, from the picture header when decoding the first identifier from the picture header, and the subparameter set may be decoded from the slice header when decoding the first identifier from the slice header.

[0563] This means that parameter sets that may change per slice or per picture are transmitted in the slice header or picture header, while parameter sets that may be shared between different pictures or slices are transmitted in the data unit. Therefore, it is no longer necessary to decode parameter sets that are repeatedly used between slices or pictures per slice header or picture header, and flexibility is maintained such that parameter sets can be changed per scene without re-receiving the entire SPS.

[0564] Furthermore, for example, the parameter set for the neural network loop filter model may include a subparameter set, which is a parameter set for defining the neural network loop filter model. The subparameter set may also include a parameter set for defining the output format for the luminance component and the chrominance component included in the output data obtained by applying the neural network loop filter model to the current block. In addition, the circuit may, in its operation, decode a flag from the picture header to identify whether or not to set a subparameter set for the neural network loop filter model when decoding the first identifier from the picture header, and decode a flag from the slice header for the neural network loop filter model when decoding the first identifier from the slice header.

[0565] This may allow processing to be performed by changing the output format for the luminance component and the chrominance component of the output data on a slice-by-slice or picture-by-picture basis.

[0566] Furthermore, for example, the loop filter section 212 of the decoding device 200 may perform the operations described above as part of the circuitry of the decoding device 200.

[0567] [Supplement] The encoding device 100 and decoding device 200 in each of the above examples may be used as an image encoding device and an image decoding device, respectively, or as a video encoding device and a video decoding device. Alternatively, the components included in the encoding device 100 or the decoding device 200 may be used as a device. The other components may be included in other devices. For example, the encoding device 100 or the decoding device 200 may be used as a prediction device, a conversion device, or a quantization device.

[0568] Furthermore, the encoding device 100 may also include an input unit and an output unit. For example, one or more pictures are input to the input unit of the encoding device 100, and a bitstream is output from the output unit of the encoding device 100. The decoding device 200 may also include an input unit and an output unit. For example, a bitstream is input to the input unit of the decoding device 200, and one or more pictures are output from the output unit of the decoding device 200. The bitstream may include quantization coefficients to which variable-length coding is applied, and control information.

[0569] This disclosure may include configuration examples of an encoding device 100 or decoding device 200 that determines "if (A) then (B) will not be performed," and configuration examples of an encoding device 100 or decoding device 200 that determines "if (A) then (B) will be prohibited." These configuration examples may be modified to the following configuration examples. Here, "(A)" and "(B)" are, for example, arbitrary states or processes to be performed in an encoding or decoding process included in this disclosure, represented using variables.

[0570] An example configuration of the encoding device 100 may be as follows: The encoding device 100 writes information indicating whether or not (A) is true to the bitstream, and if the information indicates that (A) is true, it omits writing information indicating whether or not (B) is true to the bitstream. On the other hand, if the encoding device 100 indicates that the information is not (A), it writes information indicating whether or not (B) is true to the bitstream.

[0571] An example configuration of the decoding device 200 may be as follows: The decoding device 200 reads information from the bitstream indicating whether or not it is (A), and if the information indicates that it is (A), it determines that it is not (B) without reading information from the bitstream indicating whether or not it is (B). On the other hand, if the information indicates that it is not (A), the decoding device 200 reads information from the bitstream indicating whether or not it is (B), and determines whether or not it is (B) based on the information read.

[0572] An example configuration of the encoding device 100 may be as follows: The encoding device 100 writes information indicating whether or not it is (B) to the bitstream, and if the information indicates that it is (B), it omits writing information indicating whether or not it is (A) to the bitstream. On the other hand, if the encoding device 100 indicates that it is not (B), it writes information indicating whether or not it is (A) to the bitstream.

[0573] An example configuration of the decoding device 200 may be as follows: The decoding device 200 reads information from the bitstream indicating whether or not it is (B), and if the information indicates that it is (B), it determines that it is not (A) without reading information from the bitstream indicating whether or not it is (A). On the other hand, if the information indicates that it is not (B), the decoding device 200 reads information from the bitstream indicating whether or not it is (A), and determines whether or not it is (A) based on the information read.

[0574] According to the above configuration example, the amount of information written to the bitstream can be reduced, potentially improving encoding efficiency.

[0575] This disclosure may include configuration examples of an encoding device 100 or decoding device 200 that determines "if (A), then do (B)", and configuration examples of an encoding device 100 or decoding device 200 that determines "if (A), then allow (B)". These configuration examples may be modified to the following configuration examples. Here, "(A)" and "(B)" are, for example, arbitrary states or processes to be performed in an encoding or decoding process included in this disclosure, represented using variables.

[0576] An example configuration of the encoding device 100 may be as follows: The encoding device 100 writes information indicating whether or not (A) is true to the bitstream, and if the information indicates that (A) is true, it selects process (B). In other words, the encoding device 100 selects and executes process (B) from among several candidate processes (B).

[0577] An example configuration of the decoding device 200 may be as follows: The decoding device 200 reads information from the bitstream indicating whether or not it is (A), and if the information indicates that it is (A), it selects process (B). In other words, the decoding device 200 selects and executes process (B) from among several candidate processes (B).

[0578] An example configuration of the encoding device 100 may be as follows: The encoding device 100 writes information indicating whether or not (A) is true to the bitstream, and if the information indicates that (A) is true, it always executes process (B). In other words, if (A) is true, the encoding device 100 always also executes process (B).

[0579] An example configuration of the decoding device 200 may be as follows: The decoding device 200 reads information from the bitstream indicating whether or not (A) is true, and if the information indicates that (A) is true, it always executes process (B). In other words, if (A) is true, the decoding device 200 always also executes process (B).

[0580] An example configuration of the encoding device 100 may include the following: The encoding device 100 performs a determination process and switches processing based on the determination process. Specifically, in the determination process, the encoding device 100 determines whether or not (A) is true, writes information indicating "whether or not (A) is true" to the bitstream, and if (A) is true, executes processing (B).

[0581] An example configuration of the decoding device 200 may include the following: The decoding device 200 performs a determination process and switches processing based on the determination process. Specifically, the decoding device 200 reads information from the bitstream indicating whether or not it is (A), determines in the determination process whether or not the information indicates that it is (A), and if the information indicates that it is (A), it executes processing (B).

[0582] Alternatively, the encoding device 100 may not write information indicating whether or not (A) is true to the bitstream, and the encoding device 100 and the decoding device 200 may each independently determine whether or not (A) is true.

[0583] This disclosure may include several examples of switching processes or selecting information to be used for encoding or decoding based on conditional decisions. Any of these examples may be replaced by the following processes:

[0584] For example, the encoding device 100 may generate a flag or parameter indicating the result of a conditional judgment, and based on the flag or parameter, switch processing or select information to be used for encoding. The encoding device 100 may also write the flag or parameter to a bitstream.

[0585] Furthermore, if the above-mentioned flag or parameter is included in the bitstream, the decoding device 200 may obtain the flag or parameter from the bitstream and, based on the flag or parameter, switch processing or select information to be used for decoding.

[0586] In this case, the decoding device 200 can perform decoding without having to perform conditional judgments that require a large amount of processing time, thus potentially reducing the processing load. Furthermore, the encoding device 100 may also be able to reduce its processing load by using flags or parameters, for example, when the results of the conditional judgment are used multiple times.

[0587] In this disclosure, the generated information (including flags, parameters, or indices, etc.) may be included in the bitstream. By including the information generated in the encoding process in the bitstream, the decoding device 200, which performs the decoding process corresponding to the encoding process, can perform the processing based on the information included in the bitstream, thereby reducing the processing load. On the other hand, the amount of code may increase. Therefore, the generated information does not necessarily have to be included in the bitstream.

[0588] Even if the information generated during the encoding process is not included in the bitstream, the decoding device 200, which performs the decoding process corresponding to the encoding process, can generate the same information using the same processing procedure, provided that the processing procedure is defined. This may increase the processing load of the decoding device 200, but it may reduce the amount of code in the bitstream, potentially improving encoding efficiency and compression ratio.

[0589] Where the terms “parameter,” “flag,” “identifier,” or “index” appear in this disclosure, those terms may be replaced with other terms among these terms.

[0590] For example, where the use of a “flag” is described in this disclosure, a parameter that represents one of three or more values ​​may be used instead of a flag that represents one of two values.

[0591] Specifically, for example, this disclosure may describe the use of "flags" as information that indicates either "YES" or "NO," or in other words, information that indicates "whether or not (A) is true." Even if such a description is made, other information may be combined with "whether or not (A) is true," or "whether or not (A) is true" may be further subdivided.

[0592] Therefore, a single-bit flag may be replaced by a multi-bit parameter. In other words, the term "flag" may be replaced by another term, such as "parameter" or "index."

[0593] An "index" is a value that specifies a particular column or row in a database, such as a table or list. Specifically, an index is often a value that increases from 0 to 1, but is not limited to this. For example, an index may start from 1, or it may be a value that increases by 0.1, etc. An index is often assigned to each of the multiple candidates for which the encoding device 100 or decoding device 200 selects a single element in a particular process.

[0594] Generally, candidates are processed in order from smallest index value to best candidate, so assigning smaller index values ​​to candidates that are more likely to be used may reduce the processing load. Also, generally, in entropy coding, smaller index values ​​are assigned shorter code lengths, and larger index values ​​are assigned longer code lengths. Therefore, assigning smaller index values ​​to candidates that occur more frequently may reduce the amount of code required.

[0595] An "identifier" is a value that uniquely identifies specific data or entities. Identifiers are typically unique and are used to uniquely identify specific items within a database or system. For example, an identifier may be used in an image codec to identify a specific frame or block.

[0596] Both "index" and "identifier" can be used to specify particular data. Therefore, in this disclosure, "index" may be interpreted as "identifier," and conversely, "identifier" may be interpreted as "index."

[0597] Furthermore, parameters, flags, identifiers, and indices may refer to the values ​​of the parameter, flag, identifier, and index, respectively. In other words, parameters, flags, identifiers, and indices may be replaced by the values ​​of the parameter, flag, identifier, and index, respectively.

[0598] Furthermore, A < B and A ≤ B may be interpreted interchangeably, and similarly, A > B and A ≥ B may be interpreted interchangeably. For example, in a comparison process with a threshold in a determination process, it may be stated in this disclosure that if value A is greater than threshold B, it is determined to be YES, and if value A is less than or equal to threshold B, it is determined to be NO. This statement may be reinterpreted as if value A is greater than or equal to threshold B, it is determined to be YES, and if value A is less than threshold B, it is determined to be NO.

[0599] Furthermore, in this disclosure, decimal notation may be interpreted as binary notation, and vice versa. Also, division and multiplication in binary notation may be represented by bit shifts.

[0600] For example, x >> y indicates a right shift in binary, shifting x to the right by y. A right shift of n bits is equivalent to dividing a number by 2 to the power of n in decimal. x << y indicates a left shift in binary, shifting x to the right by y. A left shift of n bits is equivalent to multiplying a number by 2 to the power of n in decimal.

[0601] Furthermore, while binary and decimal numbers were used as examples in this explanation, other number systems such as hexadecimal may also be used.

[0602] Furthermore, the multiplication symbol (×) may be replaced by the letter (x) or another symbol (*). Therefore, these symbols, letters, and other symbols may be substituted for each other.

[0603] Furthermore, if the use of "motion information" is described in this disclosure, the motion information may include not only two-dimensional vectors but also matrices and the like.

[0604] Furthermore, even if the use of "vectors" such as motion vectors or block vectors is described, such description may be replaced with the use of "matrices" or "multiple vectors." In other words, in this disclosure, descriptions such as "vectors," "motion vectors," or "block vectors" may be interpreted as "matrices" or "multiple vectors."

[0605] Furthermore, the expression "encode" may be replaced with expressions such as "store," "include," "write," "describe," "signalize," "transmit," "notify," or "save," and these expressions may be interchangeable. For example, encoding information may mean including information in a bitstream. Also, encoding information into a bitstream may mean encoding information and generating a bitstream that contains the encoded information.

[0606] Furthermore, the expression "decode" may be replaced with expressions such as "read out," "decipher," "read," "read in," "derive," "obtain," "receive," "extract," or "restore," and these expressions may be interchangeable. For example, decode information may mean obtaining information from a bitstream. Also, decode information from a bitstream may mean decode the bitstream and obtain the information contained in the bitstream.

[0607] Furthermore, encoding and decoding correspond to each other. Therefore, encoding and decoding in this disclosure may be interpreted interchangeably.

[0608] Furthermore, at least some of the examples described above may be used as encoding methods, decoding methods, interpretation methods, bitstream generation methods, or other methods.

[0609] Furthermore, each component may be implemented by being composed of dedicated hardware or by executing a software program suitable for each component. Each component may also be implemented by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

[0610] Specifically, each of the encoding device 100 and the decoding device 200 may include a processing circuit and a storage device electrically connected to and accessible from the processing circuit. For example, the processing circuit corresponds to a processor a1 or b1, and the storage device corresponds to memory a2 or b2.

[0611] The processing circuit includes at least one of dedicated hardware and a program execution unit, and performs processing using a memory device. Furthermore, if the processing circuit includes a program execution unit, the memory device stores the software program executed by that program execution unit.

[0612] An example of the software program described above is a bitstream. The bitstream contains an encoded image and syntax for decoding the image. The bitstream causes the decoding device 200 to decode the image by having it perform processing based on the syntax.

[0613] Furthermore, the software that implements the encoding device 100 or decoding device 200 described above may be a program like the following. For example, this program may cause the computer 5000 to execute the encoding method described above. Also, for example, this program may cause the computer 5000 to execute the decoding method described above. Also, for example, this program may cause the computer 5000 to execute the bitstream generation method described above.

[0614] Furthermore, each component may be a circuit, as described above. These circuits may form a single circuit as a whole, or they may be separate circuits. Also, each component may be implemented using a general-purpose processor, or it may be implemented using a dedicated processor.

[0615] Furthermore, a process performed by one component may be performed by another component. Also, the order in which processes are executed may be changed, and multiple processes may be executed in parallel. Additionally, the encoding / decoding device may comprise an encoding device 100 and a decoding device 200.

[0616] Furthermore, the first and second ordinal numbers used in the explanation may be rearranged as appropriate. Ordinal numbers may also be newly assigned to or removed from the constituent elements. In addition, these ordinal numbers may be assigned to elements to identify them, and may not correspond to a meaningful order.

[0617] Furthermore, for example, the expression "at least one (or more) of the first element, second element, and third element" corresponds to the first element, second element, third element, or any combination thereof.

[0618] One or more embodiments disclosed herein may be implemented in combination with at least some of the other embodiments of this disclosure. Furthermore, some processes, some configurations of devices, some syntax, etc., described in the flowcharts of one or more embodiments disclosed herein may be implemented in combination with the other embodiments. For example, any two or more embodiments of the multiple embodiments of this disclosure may be combined, or any two or more features of the multiple features of this disclosure may be combined.

[0619] Although the embodiments of the encoding device 100 and the decoding device 200 have been described above based on several examples, the embodiments of the encoding device 100 and the decoding device 200 are not limited to these examples. Without departing from the spirit of this disclosure, embodiments constructed by applying various modifications to each example that a person skilled in the art could conceive, and embodiments constructed by combining components from different examples, may also be included within the scope of the embodiments of the encoding device 100 and the decoding device 200.

[0620] [Implementation and Application] In each of the above embodiments, each functional or operational block can typically be implemented by a processor and various storage media.

[0621] For example, the processor may be an MPU (Micro Processing Unit) or a system-on-a-chip. The storage medium may be volatile memory or a non-volatile storage medium. The volatile memory may be RAM (Random Access Memory), etc. The non-volatile storage medium may be flash memory or EEPROM (Electrical Erasable Programmable Read-Only Memory), etc.

[0622] Furthermore, the processing performed by each functional block may be implemented as a program execution unit, such as a processor, that reads and executes software (programs) recorded on ROM (Read Only Memory) or other non-volatile storage media. This software may be distributed. This software may be recorded on various recording media, such as semiconductor memory. It is also possible to implement each functional block using hardware (dedicated circuits).

[0623] The processing described in each embodiment may be implemented by centralized processing using a single device (system), or by distributed processing using multiple devices. Furthermore, the processor executing the above program may be one or multiple. In other words, centralized processing may be performed, or distributed processing may be performed.

[0624] The embodiments of this disclosure are not limited to those described above, and various modifications are possible, which are also included within the scope of the embodiments of this disclosure.

[0625] Furthermore, here we will describe application examples of the video encoding method (image encoding method) or video decoding method (image decoding method) shown in each of the above embodiments, and various systems for implementing these application examples. Such systems may be characterized by having an image encoding device using the image encoding method, an image decoding device using the image decoding method, or an image encoding and decoding device that includes both. Other configurations of such systems can be appropriately modified as needed.

[0626] [Example of Use] Figure 53 shows an example of the overall configuration of a suitable content supply system ex100 for realizing a content distribution service. The service area for communication services is divided into cells of a desired size, and within each cell, a conventional fixed base station or a base station employing next-generation wireless access technology (e.g., base stations ex106, ex107, ex108, ex109, ex110) is installed. For example, the next-generation wireless access technology may be LTE (registered trademark) (Long Term Evolution) or 5G (fifth-generation mobile communication system), etc.

[0627] In this content supply system ex100, various devices such as a computer ex111, a game console ex112, a camera ex113, a home appliance ex114, and a smartphone ex115 are connected to the internet ex101 via an internet service provider ex102 or a communication network ex104, and base stations ex106 to ex110. The content supply system ex100 may also be configured to connect any combination of the above devices.

[0628] In various implementations, multiple devices may be directly or indirectly interconnected via a telephone network, mobile network, or short-range wireless communication technology without going through base stations ex106 to ex110. For example, the mobile network may be LTE® (Long Term Evolution) or 5G (Fifth Generation Mobile Communication System). The short-range wireless communication technology may be Bluetooth®, Wi-Fi®, or NFC (Near Field Communication).

[0629] Furthermore, the streaming server ex103 may be connected to various devices such as a computer ex111, a game console ex112, a camera ex113, a home appliance ex114, and a smartphone ex115 via the internet ex101, etc. Note that the streaming server ex103 may be replaced with a Contents Delivery Network (CDN) server.

[0630] Furthermore, the streaming server ex103 may be connected via satellite communication to various mobile networks, such as hotspots within the aircraft ex117. The LEO (Low Earth Orbit) satellite network may be used for satellite communication. For example, satellite communication may be performed via satellite ex116.

[0631] In addition, instead of base stations ex106 to ex110, wireless access points, 5G small base stations, or hotspots may be used. Furthermore, streaming server ex103 may be connected directly to the communication network ex104 without going through the internet ex101 or internet service provider ex102, or it may be connected directly to the airplane ex117 without going through satellite ex116.

[0632] Camera ex113 refers to devices such as digital cameras capable of taking still images and videos. Smartphone ex115 refers to smartphones, mobile phones, PHS (Personal Handyphone System), or equivalent devices that support mobile communication systems such as 2G, 3G, 4G, and 5G. Home appliance ex114 refers to refrigerators, smart home appliances, or other energy management devices that can interact with each other within the home.

[0633] In the content supply system ex100, a terminal with a shooting function is connected to the streaming server ex103 via the base station ex106, etc., enabling live streaming and other functions.

[0634] In live streaming, terminals (such as a computer ex111, a game console ex112, a camera ex113, a home appliance ex114, a smartphone ex115, and a terminal on an airplane ex117) may perform the encoding process described in each of the above embodiments on still images or video content captured by a user using the terminal, and may multiplex the video data obtained by encoding with sound data encoded from the sound corresponding to the video, and may transmit the obtained data to a streaming server ex103. In other words, each terminal functions as an image encoding device according to one aspect of this disclosure.

[0635] Meanwhile, the streaming server ex103 streams the content data transmitted to the requesting client. The client is a computer ex111, a game console ex112, a camera ex113, a home appliance ex114, a smartphone ex115, or a terminal on an airplane ex117, etc., that is capable of decoding the encoded data. Each device that receives the distributed data decodes and plays back the received data. That is, each device may function as an image decoding device according to one aspect of this disclosure.

[0636] [Example of use using a CDN server] The content supply system ex100 may, as an alternative example, function as a content distribution system including a CDN server and other devices. For example, this content distribution system is configured in cooperation with an origin server, CDN server, DNS server, traffic management device, edge server, load balancer, network equipment, security device, and a receiving device owned by the user. Here, the receiving device may be a computer ex111, a game console ex112, or a smartphone ex115, etc.

[0637] To explain each device, first, the origin server, as the primary source of content, has the function of providing the latest or dynamic content that has not been cached. The CDN server periodically synchronizes with the origin server to maintain the latest data and caches or replicates the content to edge servers located closer to the user.

[0638] The DNS server and traffic management device analyze user requests and, considering geographical location information and network conditions, distribute access to the optimal CDN server or edge server. The edge server then transmits the content to the receiving device.

[0639] Furthermore, the load balancer evenly distributes network traffic to each server, and network equipment such as routers and switches ensures high-speed and stable data transfer. Security devices, including firewalls, WAFs, and DDoS protection systems, ensure the overall security of communications. The receiving device plays the content.

[0640] Next, let's look at some examples of the functions of a CDN server. For example, a CDN server first has caching and content replication functions, and efficiently distributes content obtained from the origin server to each edge server.

[0641] Furthermore, the CDN server can utilize the latest transmission protocols and incorporate functions such as data reconstruction during transfer, error correction (also called error correction), and adaptive transmission control. The latest transmission protocols may include HTTP / 2 (Hypertext Transfer Protocol version 2), HTTP / 3 (Hypertext Transfer Protocol version 3), or QUIC (Quick UDP Internet Connections).

[0642] This enables CDN servers to deliver low-latency, high-quality content to users. Furthermore, CDN servers can be designed to improve the overall reliability and security of communications by integrating load balancing, routing control, and security measures such as DDoS attack protection and access control.

[0643] The user's receiving device receives encoded video and audio data transmitted from the CDN server or edge server and performs dedicated decoding processing to achieve accurate and high-quality content playback. The receiving device is designed to handle data loss and delays caused by network transmission by performing error correction, synchronization control, and buffering within the device itself. This enables a seamless and comfortable viewing experience for the user.

[0644] Furthermore, in services that distribute data over various networks, data encoded using the technologies disclosed herein may be received and distributed by a cache server. In other words, data encoded at other terminals, such as an origin server, using the technologies disclosed herein may be received and distributed by a cache server.

[0645] For example, global-level caching, routing, and load balancing may be implemented by a cache server. Based on such an implementation, a service may be provided that delivers existing content quickly and efficiently without requiring re-encoding. In such a service, data generated using the technologies disclosed herein may be exchanged.

[0646] Furthermore, a platform may be provided that dynamically delivers images that have already been encoded in a predetermined format, uploaded or taken from an external source, in a manner appropriate to the user's environment or device, such as a cloud-based image optimization service. Data generated using the technologies disclosed herein may be delivered on such a platform.

[0647] Furthermore, in the example of phased data distribution, the origin server may pre-send common low-resolution data to the CDN server for caching. In this case, when a receiving terminal requests higher-resolution content, the CDN server may request the corresponding high-resolution data segment from the origin server based on that request. The high-resolution data supplied from the origin server may then be pushed to the receiving terminal via the CDN server.

[0648] Furthermore, this system may be operated as a client-driven pull-type communication method, where the user sequentially requests each media segment necessary for playback. Specifically, for example, the receiving terminal (client) selects the optimal segment (such as low resolution or high resolution) according to the network conditions or the state of the playback buffer and sends an HTTP GET request. The server then sends back the segment corresponding to that request as a response.

[0649] This means that content is delivered in response to client requests, rather than the server proactively pushing content. This allows clients to dynamically retrieve data according to their playback conditions, and can potentially achieve flexibility for optimal use of network bandwidth.

[0650] Furthermore, in services that distribute data over various networks, data encoded using the encoding technology disclosed herein may be received and distributed by a cache server. In other words, content encoded on an origin server, etc., may be distributed quickly and efficiently without re-encoding in processes such as global-scale cache-based distribution, routing, and load balancing.

[0651] Furthermore, in cloud-based image optimization services, video or image data uploaded or imported from external sources, which has already been encoded in a predetermined format, may be dynamically converted and delivered in a form appropriate to the user's environment and device.

[0652] Figure 54 shows an example of the configuration of a content distribution system. In the typical example shown in Figure 54, the content distribution system comprises an origin server, multiple CDN servers, and multiple receiving devices, each of which performs the role described above.

[0653] [Distributed Processing] The streaming server ex103 may also consist of multiple servers or computers that process, record, and distribute data in a distributed manner. For example, the streaming server ex103 may be implemented using a CDN (Contents Delivery Network), and content distribution may be achieved through a network connecting numerous edge servers distributed around the world.

[0654] In a CDN, the physically closest edge server is dynamically assigned depending on the client. Latency is reduced by caching and delivering content to the corresponding edge server. Furthermore, in the event of several types of errors or changes in communication conditions due to increased traffic, processing can be distributed across multiple edge servers, the delivery authority can be switched to another edge server, or delivery can continue by bypassing the failed portion of the network, resulting in fast and stable delivery.

[0655] Furthermore, beyond the distributed processing of the distribution itself, the encoding process of the captured data may be performed on each terminal, on the server side, or shared amongst them. As an example, encoding generally involves two processing loops. In the first loop, the complexity or code amount of the image on a frame or scene basis is detected. In the second loop, processing is performed to improve encoding efficiency while maintaining image quality.

[0656] For example, by having a terminal perform the first encoding process and the server receiving the content perform the second encoding process, it is possible to reduce the processing load on each terminal while improving the quality of the content and the efficiency of processing. In this case, if there is a request to receive and decode near real time, the data that has been encoded once by one terminal can be received and played back by another terminal, enabling more flexible real-time distribution.

[0657] In the above example, the second encoding process is performed on the server side, but the second encoding process may also be performed on the terminal. Alternatively, instead of the second encoding process, a process including optimization using AI may be performed. Alternatively, instead of the second encoding process, inference processing may be performed on the edge terminal.

[0658] Another example is the camera ex113, which extracts features from an image, compresses the feature data as metadata (also called meta-information), and sends it to the server. The server performs compression according to the meaning (or importance of content) of the image, for example, by determining the importance of objects from the features and switching the quantization precision. Feature data is particularly effective in improving the accuracy and efficiency of motion vector prediction during further compression on the server.

[0659] Alternatively, a simple encoding method such as VLC (Variable-Length Coding) may be used on the terminal, while a more computationally intensive encoding method such as CABAC (Context-Adaptive Binary Arithmetic Coding) may be used on the server.

[0660] Another example is a location such as a stadium, shopping mall, or factory, where multiple video data sets of nearly identical scenes may exist, captured by multiple terminals. In this case, the encoding process is distributed among the multiple terminals that captured the footage, along with other terminals and servers as needed that did not capture the footage, by assigning encoding tasks to them, for example, at the Group of Picture (GOP) level, picture level, or tile level (where a picture is divided). This reduces latency and enables more real-time performance.

[0661] Since multiple video data sets depict essentially the same scene, the server may manage and / or instruct the video data captured by each terminal to reference each other. Alternatively, the server may receive the encoded data from each terminal, change the reference relationships between the multiple data sets, or correct or replace the pictures themselves and re-encode them. This allows for the creation of a stream with improved quality and efficiency for each individual data set.

[0662] Furthermore, the server may transcode the video data to change its encoding scheme before distributing it. For example, the server may convert an MPEG-based encoding scheme to a VP-based scheme (e.g., VP9), or convert H.264 to H.265.

[0663] Thus, the encoding process can be performed by a terminal or one or more servers. Therefore, in the following, the terms "server" or "terminal" will be used to refer to the entity performing the processing, but some or all of the processing performed by the server may be performed by the terminal, and some or all of the processing performed by the terminal may be performed by the server. The same applies to the decoding process.

[0664] [3D, Multi-angle] There is a growing trend to integrate and utilize images or videos of different scenes captured by multiple nearly synchronized devices, or the same scene captured from different angles. Multiple nearly synchronized devices may include cameras ex113 and / or smartphones ex115, or wearable devices and / or drones. Videos captured by each device are integrated based on the relative positional relationship between the devices, or on areas where feature points in the video coincide.

[0665] The server may not only encode two-dimensional video but also encode still images automatically based on scene analysis of the video, or at a time specified by the user, and send them to the receiving terminal. Furthermore, if the server can obtain the relative positional relationship between the shooting terminals, it can generate a three-dimensional shape of the scene based not only on two-dimensional video but also on video footage of the same scene taken from different angles.

[0666] Alternatively, the server may separately encode the 3D data (also referred to as 3D data or 3D content data) generated by a point cloud or the like. The server may then select or reconstruct the video to be transmitted to the receiving terminal from video footage captured by multiple terminals, based on the results of recognizing or tracking a person or object using the 3D data.

[0667] The server may not only encode two-dimensional video, but also convert 3D content data acquired from external sources or generated by itself into two-dimensional data (also expressed as 2D data or 2D content data), encode it, and transmit it to other terminals. The process of converting 3D content data into two-dimensional data may be performed by the server itself, or the server may acquire data that has already been converted by each shooting terminal, for example, and then encode that two-dimensional data.

[0668] The method for converting 3D content data to 2D data is not limited to the method described below; any method can be used.

[0669] For example, 3D content data may be encoded using methods such as VPCC (Video-based Point Cloud Compression) or VMMC (Video-based Dynamic Mesh Compression). During the encoding process, the 3D content data may be converted into a two-dimensional image such as a range image or texture map, and this two-dimensional image may be efficiently encoded by the server.

[0670] Furthermore, in the case of multi-view or stereoscopic video, 3D scene reconstruction and stereoscopic video distribution may be achieved by individually acquiring viewpoint images from multiple cameras, utilizing correlation information between viewpoints, and processing and encoding the 2D image group. In addition, in light field video, a method may be applied in which a large number of minute viewpoint images are acquired, and data that allows for retrospective changes to focus and viewpoint is compressed and transmitted based on each viewpoint image.

[0671] Furthermore, information obtained from 3D sensors such as LiDAR may be converted into a 2D format such as a range image or depth map and transmitted using image compression technology. In addition, in next-generation holographic images, a method may be used in which the 3D light field information itself is treated as a group of 2D images acquired from multiple angles and these are integrated.

[0672] In real-time 3D transmission for Volumetric Video and AR / VR, low-latency and highly efficient transmission can be achieved by converting part or all of the 3D data into a 2D image format and applying image codec technology.

[0673] Alternatively, the three-dimensional shape of the real world may be reconstructed from multiple 2D images using AI-based 3D reconstruction technology. Examples of 3D reconstruction technologies that may be used include Neural Radiance Fields, Gaussian Platting, or Neural Volumes Rendering.

[0674] Furthermore, methods for generating high-precision 3D point cloud data using multiple images or laser sensors, such as SfM (Structure from Motion) or LiDAR technology, may be applied. The 3D data thus obtained may be individually encoded as point clouds or the like. Alternatively, the results obtained by recognizing or tracking a person or object using the 3D data may be used to reconstruct the video transmitted to the receiving terminal.

[0675] In addition to 3D reconstruction techniques on the decoding / rendering side, AI-based methods may also be applied on the encoding side to efficiently convert already represented 3D data into 2D data in a format that can be transmitted. Specifically, volumetric data or point cloud data may be projected into a 2D format such as a depth map, range image, or texture atlas using an algorithm that converts data from 3D to 2D using deep learning. The data converted to a lower dimension in this way may be encoded in a form that is compatible with 2D image encoding techniques.

[0676] Furthermore, in the encoding process, pre-processing of the input 3D data may be performed, such as noise reduction, geometric consistency correction, and feature extraction. In addition, post-processing may be applied to the output 2D data to improve the accuracy of 3D reconstruction after transmission, such as super-resolution reconstruction or angle of view correction.

[0677] Thus, a bidirectional pipeline may be constructed that includes pre-processing and post-processing incorporating AI-based methods, in addition to the encoding and decoding processes. This allows for the efficient and high-fidelity transmission and restoration of rich 3D information while utilizing a transmission path based on 2D encoding technology.

[0678] In this way, users can enjoy scenes by arbitrarily selecting each video corresponding to each shooting terminal, or they can enjoy content in which a video from a selected viewpoint is extracted from 3D data reconstructed using multiple images or videos. Furthermore, along with the video, sound is also collected from multiple different angles, and the server may multiplex the sound from a specific angle or space with the corresponding video and transmit the multiplexed video and sound.

[0679] Furthermore, in recent years, content that links the real world with the virtual world, such as Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR), has also become widespread.

[0680] In the case of VR images, the server may create separate viewpoint images for the right and left eyes and perform encoding that allows referencing between the viewpoint images using Multi-View Coding (MVC), or it may encode them as separate streams without referencing each other. When decoding the separate streams, it is preferable to play them back in sync so that a virtual three-dimensional space is reproduced according to the user's viewpoint.

[0681] In the case of AR images, the server superimposes virtual object information from the virtual space onto camera information from the real space, based on its three-dimensional position or the user's viewpoint movement. The decoding device may acquire or retain the virtual object information and three-dimensional data, generate a two-dimensional image according to the user's viewpoint movement, and create superimposed data by smoothly stitching them together. Alternatively, the decoding device may send the user's viewpoint movement to the server in addition to requesting virtual object information.

[0682] The server may create superimposed data from the 3D data stored on the server in accordance with the movement of the viewpoint received, encode the superimposed data, and distribute it to the decoding device. The superimposed data may have an α value indicating transparency in addition to RGB, and the server may set the α value of parts other than the object created from the 3D data to 0, so that those parts are transparent, and encode the data. Alternatively, the server may set predetermined RGB values ​​as the background, similar to chroma keying, and generate data in which parts other than the object are the background color.

[0683] Similarly, the decryption process of the distributed data can be performed on each client terminal, on the server side, or shared between them. For example, one terminal may send a reception request to the server, other terminals may receive the content corresponding to that request and perform the decryption process, and the decrypted signal may be transmitted to a device with a display. By distributing the processing and selecting appropriate content regardless of the performance of the communication-capable terminals themselves, it is possible to play back data with good image quality.

[0684] Another example is when a large image file is received on a TV or similar device, and only a portion of the image, such as tiles, is decoded and displayed on the viewer's personal device. This allows for sharing the overall picture while simultaneously enabling users to check their own area of ​​responsibility or areas they wish to examine in more detail.

[0685] In situations where multiple short-range, medium-range, or long-range wireless communication networks are available both indoors and outdoors, it may be possible to seamlessly receive content using a distribution system standard. MPEG-DASH (Dynamic Adaptive Streaming over HTTP) or HLS (HTTP Live Streaming) may be used as the distribution system standard.

[0686] Users may freely select and switch in real time between decoding devices or display devices, such as their own terminals or displays located indoors or outdoors. Furthermore, they can perform decoding while switching between the decoding terminal and the display terminal using their own location information. This makes it possible to map and display information on a part of the wall or ground of an adjacent building where a displayable device is embedded, while the user is traveling to their destination.

[0687] Furthermore, it is possible to switch the bitrate of the received data based on the ease of access to the encoded data over the network, such as when the encoded data is cached on a server that can be accessed quickly from the receiving terminal, or when it is copied to an edge server in the content delivery service.

[0688] [Web Page Optimization] Figure 55 shows an example of how a web page is displayed on a computer ex111, etc. Figure 56 shows an example of how a web page is displayed on a smartphone ex115, etc. As shown in Figures 55 and 56, a web page may contain multiple linked images, which are links to image content, and their appearance will differ depending on the viewing device.

[0689] If multiple linked images are visible on the screen, the display device (decoder) may display the still images or iPictures of each content as linked images until the user explicitly selects a linked image, or until the linked image approaches the center of the screen or the entire linked image is within the screen area. It may also display video such as a GIF animation using multiple still images or iPictures, or it may receive only the base layer and decode and display the video.

[0690] When a user selects a linked image, the display device decrypts it, prioritizing the base layer. However, if the HTML (Hyper Text Markup Language) / CSS (Cascading Style Sheets) that make up the web page contain information indicating responsive design, the display device may decrypt up to the enhancement layer.

[0691] Furthermore, to ensure real-time performance, if selection has not yet been made or if the communication bandwidth is very limited, the display device can decode and display only forward-referenced pictures (I-pictures, P-pictures, and B-pictures that only use forward references), thereby reducing the delay between the decoding time and the display time of the first picture (the delay from the start of content decoding to the start of display). In addition, the display device may deliberately ignore the reference relationships of the pictures and roughly decode all B-pictures and P-pictures using forward references, and then perform normal decoding as time passes and more pictures are received.

[0692] [Automated Driving] When transmitting or receiving still images or video data, such as two-dimensional or three-dimensional map information, for the purpose of automated driving or driving assistance of a vehicle, the receiving terminal may receive metadata such as weather or construction information in addition to image data belonging to one or more layers, and decode these in association with each other. The metadata may belong to a layer, or it may simply be multiplexed with the image data.

[0693] In this case, since the vehicle, drone, or airplane containing the receiving terminal is moving, the receiving terminal can transmit its location information, enabling seamless reception and decoding by switching between base stations ex106 to ex110 (e.g., a group of base stations constituting a 5G communication network). Furthermore, the receiving terminal can dynamically switch how much metadata is received or how much map information is updated, depending on the user's selection, the user's situation, and / or the state of the communication bandwidth.

[0694] The content supply system ex100 allows the client to receive, decode, and play back encoded information transmitted by the user in real time.

[0695] [Distribution of Personal Content] In addition, the content supply system ex100 allows for unicast or multicast distribution of not only high-definition, long-duration content from video distribution companies, but also low-definition, short-duration content from individuals. It is expected that the amount of such personal content will continue to increase. To make personal content even better, the server may perform editing processing before encoding processing. This can be achieved, for example, using the following configuration.

[0696] During shooting, or after shooting, the server performs recognition processing such as detecting shooting errors, searching for scenes, analyzing meaning, and detecting objects from the original image data or encoded data. Based on the recognition results, the server then manually or automatically edits the images, correcting out-of-focus or shaky images, deleting less important scenes such as those with lower brightness or out of focus compared to other pictures, emphasizing object edges, and altering color tones.

[0697] The server encodes the edited data based on the editing results. It is also known that viewership decreases if the shooting time is too long, so the server may automatically clip scenes of low importance, as well as scenes with little movement, based on the image processing results, so that the content falls within a specific time range according to the shooting time. Alternatively, the server may generate and encode a digest based on the results of semantic analysis of the scenes.

[0698] Personal content may sometimes contain elements that infringe on copyright, moral rights, or portrait rights, and the scope of sharing may exceed the intended scope, which can be inconvenient for the individual. Therefore, for example, the server may intentionally defocus the image by changing parts of the screen, such as people's faces at the edges or the interior of a house, before encoding. Furthermore, the server may recognize whether the image to be encoded contains the face of a person other than those previously registered, and if so, it may apply a mosaic effect to the face.

[0699] Alternatively, as a pre- or post-processing step before encoding, the user may specify a person or background area in the image that they wish to modify from a copyright or other perspective. The server may then perform processing such as replacing the specified area with another image or blurring the focus. If it is a person, the server can track the person in the video and replace the image of the person's face.

[0700] Viewing personal content with small data volumes requires real-time processing. Depending on the bandwidth, the decoding device will first prioritize receiving the base layer and then decode and play it back. During this time, the decoding device may also receive the enhancement layer, and if playback is looped or otherwise required to occur more than once, it may play back the high-quality video including the enhancement layer.

[0701] With a stream that uses this kind of scalable encoding, the video may appear rough when unselected or at the beginning of viewing, but the stream can gradually become smoother and the image quality can improve, providing a similar user experience. Even without scalable encoding, a similar experience can be provided if the rough stream played the first time and the second stream, encoded by referencing the first video, are configured as a single stream.

[0702] [Other Application Examples] In addition, these encoding or decoding processes are generally processed in the LSI ex500 that each terminal has. The LSI (large scale integration circuit) ex500 (see Figure 53) may be a single chip or a configuration consisting of multiple chips.

[0703] Furthermore, video encoding or decoding software may be embedded in some recording medium (such as a CD-ROM, flexible disk, or hard disk) that can be read by a computer such as ex111, and the encoding or decoding process may be performed using that software. In addition, if the smartphone ex115 has a camera, video data acquired by that camera may be transmitted. The video data in this case is data encoded by the LSIex500 of the smartphone ex115.

[0704] The LSIex500 may also be configured to be activated by downloading application software. In this case, the terminal first determines whether it supports the content encoding method or whether it has the capability to perform the specific service. If the terminal does not support the content encoding method or does not have the capability to perform the specific service, the terminal downloads the codec or application software, and then acquires and plays the content.

[0705] Furthermore, not only the content supply system ex100 via the Internet ex101, but also digital broadcasting systems can incorporate at least one of the video encoding device (image encoding device) or video decoding device (image decoding device) of each of the above embodiments.

[0706] In digital broadcasting systems, multiplexed data containing both video and audio is transmitted and received via satellite or other means, which is more suited to multicast than the unicast-friendly configuration of the content supply system ex100. However, the encoding and decoding processes can be applied in a similar manner.

[0707] [Hardware Configuration] Figure 57 shows further details of the smartphone ex115 shown in Figure 53. Figure 58 shows an example configuration of the smartphone ex115. The smartphone ex115 includes an antenna ex450 for transmitting and receiving radio waves with the base station ex110, a camera unit ex465 capable of taking video and still images, and a display unit ex458 that displays video captured by the camera unit ex465 and data decoded from video received by the antenna ex450.

[0708] The smartphone ex115 further comprises an operation unit ex466, which is a touch panel, etc.; an audio output unit ex457, which is a speaker, etc., for outputting voice or sound; an audio input unit ex456, which is a microphone, etc., for inputting voice; a memory unit ex467, which can store captured video or still images, recorded audio, received video or still images, encoded data such as emails, or decoded data; and a slot unit ex464, which is an interface unit with a SIM (Subscriber Identity Module) ex468 for identifying the user and authenticating access to various data, including the network. External memory may be used instead of the memory unit ex467.

[0709] The main control unit ex460, which comprehensively controls the display unit ex458 and the operation unit ex466, is connected to the power supply circuit unit ex461, the operation input control unit ex462, the video signal processing unit ex455, the camera interface unit ex463, the display control unit ex459, the modulation / demodulation unit ex452, the multiplexing / decompression unit ex453, the audio signal processing unit ex454, the slot unit ex464, and the memory unit ex467 via the synchronization bus ex470.

[0710] When the power key is turned on by the user, the power supply circuit unit ex461 starts up the smartphone ex115 to an operational state and supplies power to each part from the battery pack.

[0711] The smartphone ex115 performs tasks such as making calls and data communications based on the control of the main control unit ex460, which has a CPU, ROM, RAM, etc.

[0712] During a call, the audio signal picked up by the audio input unit ex456 is converted into a digital audio signal by the audio signal processing unit ex454, spread spectrum processing is performed by the modulation / demodulation unit ex452, digital-to-analog conversion and frequency conversion processing are performed by the transmission / reception unit ex451, and the resulting signal is transmitted via the antenna ex450.

[0713] Furthermore, the received data is amplified, frequency conversion processing and analog-to-digital conversion processing are performed, the modulation / demodulation unit ex452 performs despread spectrum processing, and the audio signal processing unit ex454 converts it into an analog audio signal, which is then output from the audio output unit ex457.

[0714] In data communication mode, text, still images, or video data are sent to the main control unit ex460 via the operation input control unit ex462 based on operations performed on the main unit's operation panel ex466, etc. The same transmission and reception process is performed.

[0715] When transmitting video, still images, or video and audio in data communication mode, the video signal processing unit ex455 compresses and encodes the video signal stored in the memory unit ex467 or the video signal input from the camera unit ex465 using the video encoding method shown in each of the above embodiments, and sends the encoded video data to the multiplexing / decoupling unit ex453. The audio signal processing unit ex454 encodes the audio signal picked up by the audio input unit ex456 while the camera unit ex465 is capturing video or still images, and sends the encoded audio data to the multiplexing / decoupling unit ex453.

[0716] The multiplexing / decompression unit ex453 multiplexes the encoded video data and encoded audio data in a predetermined manner, modulates and converts the data in the modulation / demodulation unit (modulation / demodulation circuit unit) ex452 and the transmit / receive unit ex451, and transmits it via the antenna ex450.

[0717] When receiving video attached to an email or chat, or video linked to a webpage, etc., in order to decode the multiplexed data received via antenna ex450, the multiplexing / decompression unit ex453 separates the multiplexed data, dividing it into a bitstream of video data and a bitstream of audio data. It then supplies the encoded video data to the video signal processing unit ex455 and the encoded audio data to the audio signal processing unit ex454 via the synchronization bus ex470.

[0718] The video signal processing unit ex455 decodes the video signal using a video decoding method corresponding to the video encoding method shown in each of the above embodiments, and the video or still image contained in the linked video file is displayed from the display unit ex458 via the display control unit ex459.

[0719] The audio signal processing unit ex454 decodes the audio signal, and the audio is output from the audio output unit ex457. As real-time streaming becomes increasingly widespread, audio playback may be socially inappropriate depending on the user's situation. Therefore, as an initial setting, it is preferable to play only the video data without playing the audio signal, and to play the audio in sync only when the user performs an action such as clicking on the video data.

[0720] While the smartphone ex115 was used as an example here, in addition to a transceiver-type terminal that has both an encoder and a decoder, there are three other possible implementation formats for terminals: a transmitting terminal that has only an encoder, and a receiving terminal that has only a decoder.

[0721] In the digital broadcasting system, it was explained that multiplexed data, in which audio data is multiplexed with video data, is received or transmitted. However, the multiplexed data may also contain other data besides audio data, such as text data related to the video. Furthermore, the video data itself may be received or transmitted instead of the multiplexed data.

[0722] Although it was explained that the main control unit ex460, including the CPU, controls the encoding or decoding process, various terminals often also have a GPU (Graphics Processing Unit). Therefore, a configuration is also possible in which a wide area is processed collectively by utilizing the performance of the GPU, using memory shared by the CPU and GPU, or memory whose addresses are managed so that it can be used in common.

[0723] This reduces encoding time, ensures real-time performance, and achieves low latency. In particular, it is more efficient to perform motion detection, deblocking filters, SAO (Sample Adaptive Offset), and transformation / quantization processes on the GPU in batches, such as on a picture-by-picture basis, rather than on the CPU.

[0724] This disclosure can be used, for example, in television receivers, digital video recorders, car navigation systems, mobile phones, digital cameras, digital video cameras, video conferencing systems, or electronic mirrors.

[0725] 100 Encoding device 102 Splitting unit 104 Subtraction unit 106 Transformation unit 108 Quantization unit 110 Entropy coding unit 112, 204 Inverse quantization unit 114, 206 Inverse transform unit 116, 208 Addition unit 118, 210 Block memory 120, 212 Loop filter unit 122, 214 Frame memory 124, 216 Intra prediction unit 126, 218 Inter prediction unit 128, 220 Prediction control unit 130, 222 Prediction parameter generation unit 200 Decoding device 202 Entropy decoding unit 224 Splitting determination unit 1000 Bitstream generation device 2000 Transmitting device 3000 Receiving device 4000 Storage medium 5000 Computer a1, b1 Processor a2, b2 Memory a3 Transmitter b4 Receiver

Claims

1. An encoding device comprising a circuit and a memory connected to the circuit, wherein the circuit, in operation, encodes a data unit for each of a plurality of neural network loop filter models that includes a parameter set relating to the neural network loop filter model, and which can be composed solely of the parameter set, and encodes a first identifier for identifying the plurality of neural network loop filter models into a picture header or slice header.

2. The encoding device according to claim 1, wherein the data unit is an APS (Adaptation Parameter Set).

3. The encoding device according to claim 1, wherein the data unit is a Network Abstraction Layer (NAL) unit.

4. The encoding device according to claim 1, wherein when the circuit encodes the first identifier into the slice header, the data unit is signaled before or after the picture data or slice data.

5. The encoding device according to any one of claims 1 to 4, wherein the parameter set relating to the neural network loop filter model includes a plurality of subparameter sets which are parameter sets for defining the neural network loop filter model, each of the plurality of subparameter sets is assigned a second identifier, and the circuit further encodes the second identifiers corresponding to at least some of the plurality of subparameter sets in the picture header when encoding the first identifier in the picture header, and encodes the second identifiers corresponding to at least some of the plurality of subparameter sets in the slice header when encoding the first identifier in the slice header.

6. The encoding device according to any one of claims 1 to 4, wherein the parameter set relating to the neural network loop filter model includes a subparameter set which is a parameter set for defining the neural network loop filter model, and the subparameter set is a parameter set which is uniquely determined for the neural network loop filter model.

7. The encoding device according to any one of claims 1 to 4, wherein the parameter set relating to the neural network loop filter model includes a subparameter set which is a parameter set for defining the neural network loop filter model, and the circuit further encodes, in operation, a rewrite subparameter set for rewriting a part of the subparameter set in the picture header when encoding the first identifier in the picture header, and encodes the rewrite subparameter set in the slice header when encoding the first identifier in the slice header.

8. The encoding device according to any one of claims 1 to 4, wherein the circuit further encodes, in operation, a subparameter set which is a parameter set for defining the neural network loop filter model in the picture header when encoding the first identifier in the picture header, and encodes the subparameter set in the slice header when encoding the first identifier in the slice header.

9. The encoding device according to any one of claims 1 to 4, wherein the parameter set relating to the neural network loop filter model includes a subparameter set which is a parameter set for defining the neural network loop filter model, the subparameter set includes a parameter set for defining the output format for the luminance component and the chrominance component included in the output data obtained by applying the neural network loop filter model to the current block, and the circuit further encodes in the picture header a flag in the picture header for identifying whether or not to set the subparameter set for the neural network loop filter model when encoding the first identifier in the picture header, and encodes the flag in the slice header for the neural network loop filter model when encoding the first identifier in the slice header.

10. A decoding device comprising a circuit and a memory connected to the circuit, wherein the circuit, in operation, decodes a data unit for each of a plurality of neural network loop filter models, which includes a parameter set relating to the neural network loop filter model and can be composed solely of the parameter set, and decodes a first identifier for identifying the plurality of neural network loop filter models from a picture header or slice header.

11. The decoding device according to claim 10, wherein the data unit is an APS (Adaptation Parameter Set).

12. The decoding device according to claim 10, wherein the data unit is a Network Abstraction Layer (NAL) unit.

13. The decoding device according to claim 10, wherein when the circuit decodes the first identifier from the slice header, the data unit is signaled before or after the picture data or slice data.

14. The decoding device according to any one of claims 10 to 13, wherein the parameter set relating to the neural network loop filter model includes a plurality of subparameter sets which are parameter sets for defining the neural network loop filter model, the plurality of subparameter sets are assigned a second identifier, and the circuit, in operation, further decodes the second identifiers corresponding to at least some of the plurality of subparameter sets from the picture header when decoding the first identifier from the picture header, and decodes the second identifiers corresponding to at least some of the plurality of subparameter sets from the slice header when decoding the first identifier from the slice header.

15. The decoding device according to any one of claims 10 to 13, wherein the parameter set relating to the neural network loop filter model includes a subparameter set which is a parameter set for defining the neural network loop filter model, and the subparameter set is a parameter set which is uniquely determined for the neural network loop filter model.

16. The decoding device according to any one of claims 10 to 13, wherein the parameter set relating to the neural network loop filter model includes a subparameter set which is a parameter set for defining the neural network loop filter model, and the circuit further, in operation, decodes from the picture header a rewrite subparameter set for rewriting a portion of the subparameter set when decoding the first identifier from the picture header, and decodes from the slice header the rewrite subparameter set when decoding the first identifier from the slice header.

17. The decoding device according to any one of claims 10 to 13, wherein the circuit, in operation, further decodes a subparameter set which is a set of parameters for defining the neural network loop filter model from the picture header when decoding the first identifier from the picture header, and decodes the subparameter set from the slice header when decoding the first identifier from the slice header.

18. The decoding device according to any one of claims 10 to 13, wherein the parameter set relating to the neural network loop filter model includes a subparameter set which is a parameter set for defining the neural network loop filter model, the subparameter set includes a parameter set for defining the output format for the luminance component and the chrominance component included in the output data obtained by applying the neural network loop filter model to the current block, and the circuit further, in operation, decodes from the picture header a flag for identifying whether or not to set the subparameter set for the neural network loop filter model when decoding the first identifier from the picture header, and decodes the flag from the slice header for the neural network loop filter model when decoding the first identifier from the slice header.

19. An encoding method comprising encoding a data unit for each of a plurality of neural network loop filter models, which includes a parameter set relating to the neural network loop filter model and can be composed solely of the parameter set, and encoding a first identifier for identifying the plurality of neural network loop filter models in a picture header or slice header.

20. A decoding method comprising decoding a data unit containing a parameter set for each of a plurality of neural network loop filter models, the data unit being composed solely of the parameter set, and decoding a first identifier for identifying the plurality of neural network loop filter models from a picture header or slice header.