Selection and use of an applicable padding method in a coding scheme

Abstract

The following operations are performed for an encoding process for input video. One of multiple padding methods is selected to perform padding of an outer region defined outside a region. The region is from a picture used in the encoding process, and the selecting is performed based on information providing a ranking of the multiple padding methods. The selected padding method is performed to form an extension region defined outside the region. The following operations are performed for a decoding process using a bitstream. A selected padding method is determined to use in relation to a region of a picture from the bitstream. The selected padding method is applied to the region to form an extension region that is defined outside the region.

Description

SELECTION AND USE OF AN APPLICABLE PADDING METHOD IN A CODING SCHEMETECHNICAL FIELD

[0001] Examples of embodiments herein relate generally to video coding and decoding and, more specifically, relate to filling locations for samples that are out-of-bounds of a region for video coding.BACKGROUND

[0002] Motion vectors (MVs) are a fundamental component of inter prediction in modern video coding standards. They are used to exploit temporal redundancy between successive frames by an estimation of the movement of objects or regions, reducing the amount of data required for encoding.

[0003] Inter prediction, also known as motion-compensated prediction (MCP), involves the following steps. In Motion Estimation (ME), the encoder searches for a bestmatching block in one or more reference pictures (previous or future pictures). This process is computationally intensive and often employs techniques such as full search, diamond search, or hierarchical block matching. In Motion Compensation (MC), once the best match is found, the motion vector (MV) is calculated. The MV represents the displacement from the current block’s position to its best match in the reference picture. The difference (residual) between the predicted block and the actual block is then encoded and transmitted along with the MV. At the decoder, the motion vector is used to fetch the corresponding predicted block, and the residual is added to reconstruct the final block.

[0004] Instead of transmitting the entire block’s pixel values, the encoder transmits the motion vector (MV) and the residual (difference between predicted and actual block). Since motion vectors typically exhibit spatial correlation (neighboring blocks have similar motion), they are usually compressed using differential coding (predictive coding of motion vectors).

[0005] The decoder receives the motion vector and residual, and retrieves the reference block using the MV and adds the residual to reconstruct the current block. Thissignificantly reduces the required bitrate compared to intra-coding, which encodes pictures independently.

[0006] While inter prediction and MVs are useful, there are times when motion vectors reference to parts of blocks that are outside the boundary of an image. These parts usually need to be “padded” to have information in them, which describes the process of extending the image beyond the boundary. It has been found that processing the out-of-bounds and padded information yields benefits, but this process could be improved.BRIEF SUMMARY

[0007] This section is intended to include examples and is not intended to be limiting.

[0008] In an example, a method is disclosed that includes performing operations comprising the following for an encoding process for input video: selecting one of multiple padding methods to perform padding of an outer region defined outside a region, the region from a picture used in the encoding process, and the selecting performed based on information providing a ranking of the multiple padding methods; and performing the selected padding method to form an extension region defined outside the region.

[0009] An additional example includes a computer program, comprising instructions for performing the method of the previous paragraph, when the computer program is run on an apparatus. The computer program according to this paragraph, wherein the computer program is a computer program product comprising a computer-readable medium bearing the instructions embodied therein for use with the apparatus. Another example is the computer program according to this paragraph, wherein the program is directly loadable into an internal memory of the apparatus.

[0010] An example of an apparatus includes one or more processors and one or more memories storing instructions that, when executed by the one or more processors, cause the apparatus at least to perform: performing operations comprising the following for an encoding process for input video: selecting one of multiple padding methods to perform padding of an outer region defined outside a region, the region from a picture used in the encoding process, and the selecting performed based on information providing a ranking of the multiple padding methods; and performing the selected padding method to form an extension region defined outside the region.

[0011] An example of a computer program product includes a computer-readable storage medium bearing instructions that, when executed by an apparatus, cause the apparatus to perform at least the following: performing operations comprising the following for an encoding process for input video: selecting one of multiple padding methods to perform padding of an outer region defined outside a region, the region from a picture used in the encoding process, and the selecting performed based on information providing a ranking of the multiple padding methods; and performing the selected padding method to form an extension region defined outside the region.

[0012] In another example, an apparatus comprises means for: performing operations comprising the following for an encoding process for input video: selecting one of multiple padding methods to perform padding of an outer region defined outside a region, the region from a picture used in the encoding process, and the selecting performed based on information providing a ranking of the multiple padding methods; and performing the selected padding method to form an extension region defined outside the region.

[0013] In an example, a method is disclosed that includes performing operations comprising the following for a decoding process using a bitstream: determining a selected padding method to use in relation to a region of a picture from the bitstream; and applying the selected padding method to the region to form an extension region that is defined outside the region.

[0014] An additional example includes a computer program, comprising instructions for performing the method of the previous paragraph, when the computer program is run on an apparatus. The computer program according to this paragraph, wherein the computer program is a computer program product comprising a computer-readable medium bearing the instructions embodied therein for use with the apparatus. Another example is the computer program according to this paragraph, wherein the program is directly loadable into an internal memory of the apparatus.

[0015] An example of an apparatus includes one or more processors and one or more memories storing instructions that, when executed by the one or more processors, cause the apparatus at least to perform: performing operations comprising the following for a decoding process using a bitstream: determining a selected padding method to use in relation to a region ofa picture from the bitstream; and applying the selected padding method to the region to form an extension region that is defined outside the region.

[0016] An example of a computer program product includes a computer-readable storage medium bearing instructions that, when executed by an apparatus, cause the apparatus to perform at least the following: performing operations comprising the following for a decoding process using a bitstream: determining a selected padding method to use in relation to a region of a picture from the bitstream; and applying the selected padding method to the region to form an extension region that is defined outside the region.

[0017] In another example, an apparatus comprises means for: performing operations comprising the following for a decoding process using a bitstream: determining a selected padding method to use in relation to a region of a picture from the bitstream; and applying the selected padding method to the region to form an extension region that is defined outside the region.BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings use reference numerals, where the same reference numerals may be used to refer to like parts throughout, but parts having the same reference numeral can differ in operation and components. In the attached drawings:

[0019] FIG. 1 is a block diagram illustrating a system in accordance with an example;

[0020] FIG. 2 illustrates an encoder such as a WC encoder where the encoder examples can be implemented;

[0021] FIG. 2A illustrates an example of motion compensation with a motion vector pointing outside a boundary for a picture for inter prediction;

[0022] FIG. 3 illustrates a decoder such as a VVC decoder where the decoder examples can be implemented;

[0023] FIG. 4 illustrates an overview example of two padding methods, their corresponding outer regions, and a defined continuation based on a metric calculation for the two padding methods;

[0024] FIG. 4A is an example of a scenario used to implement FIG. 4;

[0025] FIG. 4B is another example of a scenario used to implement FIG. 4;

[0026] FIG. 4C illustrates a picture partitioned into subpictures and corresponding possible cropping for this picture;

[0027] FIG. 5 illustrates a flowchart of an encoding method for selection of an applicable padding method in a coding scheme for inter prediction;

[0028] FIG. 5A illustrates a flowchart of an encoding method for selection of an applicable padding method in a coding scheme for inter prediction based on region attributes;

[0029] FIG. 5B illustrates a flowchart of an encoding method for selection of an applicable padding method in a coding scheme for a loop filter;

[0030] FIG. 6 illustrates a flowchart of a decoding method for selection and use of an applicable padding method in a coding scheme;

[0031] FIG. 7 illustrates a flowchart of a method of signaling a selected method to the decoder side; and

[0032] FIG. 8 is an example of a block diagram of an apparatus suitable for implementing any of the encoders or decoders described herein.DETAILED DESCRIPTION OF THE DRAWINGS

[0033] Abbreviations that may be found in the specification and / or the drawing figures are defined below, at the end of the detailed description section.

[0034] The word “exemplary” is used herein to mean “serving as an example, instance, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments. All of the embodiments described in this Detailed Description are exemplary embodiments provided to enable persons skilled in the art to make or use the examples.

[0035] When more than one drawing reference numeral, word, or acronym is used within this description withand in general as used within this description, the “ / ” may be interpreted as “or”, “and”, or “both”. As used herein, “at least one of the following: ” and “at least one of ” and similar wording, where the list of two or more elements are joined by “and” or “or,” mean at least any one of the elements, or at least any two or more of the elements, or at least all the elements.

[0036] As used herein, the singular forms “a”, “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be furtherunderstood that the terms “comprises”, “comprising”, “has”, “having”, “includes” and / or “including”, when used herein, specify the presence of stated features, elements, and / or components etc., but do not preclude the presence or addition of one or more other features, elements, components and / or combinations thereof.

[0037] It is noted that capital and lowercase words or phrases are considered to be the same herein. For instance, the words Slice, slice, and SLICE are the same, as are the phrases Network Repository Function, network repository function, and NETWORK REPOSITORY FUNCTION.

[0038] Any flow diagram or signaling diagram herein is considered to be a logic flow diagram, and illustrates the operation of an exemplary method, results of execution of computer program instructions embodied on a computer readable memory, and / or functions performed by logic implemented in circuitry. For methods, flow diagrams, and signaling diagrams, the orders of method steps, blocks in the flow, or signaling are not critical and instead are examples.

[0039] Technical context is now provided for technical areas related to the understanding of the examples. This is provided as brief overviews of possibly related technical areas.

[0040] The description herein relates to coding and decoding of digital video material. As such, a video codec includes an encoder that transforms the input video into a compressed representation suited for storage / transmission and a decoder that can decompress the compressed video representation back into a viewable form. Typically, the encoder discards some information in the original video sequence to be able to represent the video in a more compact form (that is, at lower bitrate).

[0041] Referring to FIG. 1, this figure is used as a simple overview of a possible system. This figure is a block diagram illustrating a system 100 in accordance with an example. In the example, the encoder 130 is used to encode input video 110-1 from the scene 15, and the encoder 130 is implemented in a transmitting apparatus 180-1. There is a capture of input video at a viewpoint 10 of a scene 15, which includes a human being 20. The encoder 130 produces a bitstreams 101, using the encoding process 131 on the input video 110-1, that is received by the receiving apparatus 180-2. The receiving apparatus 180-2 implements a decoder 140, which performs a decoding process 141. The decoder 140, using the decoding process 141 on the bitstreams 101, forms the output video 110-2 (as a representation of the input video 110-1) forthe scene 15-1, and the receiving apparatus 180-2 would present this to the user, e.g., via a smartphone, television, or projector among many other options. The scene 15-1 has a viewpoint 10-1 and contains representations of at least a human being 20-1. The encoder 130 and decoder 140 may be applied to multiple coding standards.

[0042] One such standard is Versatile Video Coding (WC), which is a new international video coding standard. Enhanced Compression Model (ECM) is built on top of WC and is a test model for a potential future video coding standard. ECM is currently under the development sponsored by JVET (Joint video experts team). Both WC and HEVC (High Efficiency Video Coding) are block-based video coding standards, where an input picture is divided into CTUs (coding tree units), and each CTU may be further split into CUs (coding units). A CU (as one type of block) is coded in either inter-coding mode or intra-coding mode. If the block is in inter-coding mode, the encoder 130 searches for a temporal prediction block in reference picture(s), and may signal the decoder 140 how to find the same prediction block in reference picture(s) at the decoder end. If the block is in intra coding mode, the encoder 130 constructs a spatial prediction block from the current picture, and may signal the decoder 140 how to form the same spatial prediction block from the current picture at the decoder end.

[0043] At the encoder 130 end, the residual block between a current CU and its prediction block is transformed and quantized. The quantized transform coefficients are entropy coded. The decoder 140, on the other hand, performs inverse operations, such as entropy decoding, dequantization, and inverse transform, to reconstruct the residual block, and reconstructs the CU (or block) by adding the reconstructed residual block to the prediction block.

[0044] In more detail, typical hybrid video codecs, such as H.264 / AVC (Advanced video coding), H.265 / HEVC (High Efficiency Video Coding), and H.266 / WC, encode the video information in two phases. Firstly, pixel values in a certain picture area (or “block”) are predicted for example by motion compensation means (finding and indicating an area in one of the previously coded pictures that corresponds closely to the block being coded) or by spatial means (using the pixel values around the block to be coded in a specified manner). Secondly the prediction error, i.e., the difference between the predicted block of pixels and the original block of pixels, is coded. This is typically performed by transforming the difference in pixel values using a specified transform (e.g., Discrete Cosine Transform (DCT) or a variant of it), quantizing the resulting transform coefficients and entropy coding the quantized coefficients. By varying thefidelity of the quantization process, encoder can control the balance between the accuracy of the pixel representation (picture quality) and size of the resulting coded video representation (file size or transmission bitrate). The encoding process is illustrated in FIG. 2, using WC as an example.

[0045] Referring to FIG. 2, this figure illustrates a VVC encoder 130 where the encoder examples can be implemented. The examples herein can be implemented in at least a process concerning the inter-prediction block 206 and / or the filter 216, possibly the ALF (Adaptive loop filter) 216-4, which may perform padding and involve boundary handling as described below. It is noted that the terms “image”, “picture”, and “frame” are considered to be the same herein. That is, in the description below, the term “picture” is mainly used, but any other term encompassing what the “picture” is may be used. The encoder 130 is a base layer and other layers (not shown) may be added to the base layer. Concerning the other layers, it is noted that the examples below can also be relevant to a scalable codec. For the examples below, in a scalable codec, the padding area that is generated in the base layer could also be used to derive information for an enhancement layer, for example taking the padding into account in the upsampling process.

[0046] FIG. 2 shows in part an inter-prediction block 206, an intra-prediction block 208, a mode selector 255, a filter 216, and a decoded picture buffer 218. The input video 110-1 is sent as individual pictures to be encoded at both the inter-predictor 206 (which determines the difference between the picture and a motion compensated reference frame from the decoded picture buffer 218) and the intra-predictor 208 (which determines a prediction for an image block based only on the already processed parts of the current picture from the input video 110). The mode selector 255 selects which output of the inter-predictor 206 or the intra-prediction block 208 is passed to the summing devices 221 and 239. The intra-prediction block 208 may have more than one intra-prediction mode. Hence, each mode may perform the intra-prediction and provide the predicted signal to the summing devices 221 and 239.

[0047] Depending on which encoding mode is selected to encode the current block, the output of the inter-prediction block 206, or the output of one of the intra-predictor 208 modes is passed as a predictor output 212 to the summing devices 221 and 239. This predictor output 212 is passed to first summing device 221 and comprises the prediction representation of an image block. The first summing device 221 may subtract the prediction representation of theimage block to produce a first prediction error signal 220, which is input to the transform / quantization block 203.

[0048] The quantized transform coefficients (“quant, transf. coeff”) are output to the entropy coding block 230 and to the inverse quantization (“inv. quant”) and inverse transform block 204. Output 292 of second summing device 239, which acts as a preliminary reconstructor, there is a combination of the prediction representation of the image block at the predictor output 212 and the prediction error signal 238 of the inverse quantization / inverse transform block 204. The preliminary reconstructed picture 214 may be passed to the filter 216. The filter 216 receiving the preliminary representation may filter the preliminary representation and output a final reconstructed picture into the decoded picture buffer 218. The filter 216 in this example includes LMCS (luma mapping with chroma scaling) 216-1, DBF (deblocking filter) 216-2, SAG (Sample adaptive offset) 216-3, and ALF (Adaptive loop filter) 216-4. It is noted that not all of these filters may be used, and other filters may be used. Pictures from the decoded picture buffer 218 may be connected to the inter-prediction block 206 to be used as the reference picture against which a future base layer picture from the input video 110-1 is compared in interprediction operations.

[0049] The transform / quantization block 203 comprises a transform unit and a quantizer. The transform unit transforms the first prediction error signal 220 to a transform domain. The transform is, for example, the DCT (discrete cosine transform). The quantizer quantizes the transform domain signal, e.g., the DCT coefficients, to form quantized coefficients.

[0050] The inverse quantization / inverse transform block 204 receives the output from the transform / quantization block 203, and performs the opposite processes of the transform / quantization block 203 to produce a decoded prediction error signal 238 which, when combined with the prediction representation of the image block at the predictor output 212 at the second summing device 239, produces the preliminary reconstructed picture 214. The inverse quantization / inverse transform block 204 may be considered to comprise a dequantizer, which dequantizes the quantized coefficient values, e.g., DCT coefficients, to reconstruct the transform signal and an inverse transformation unit, which performs the inverse transformation to the reconstructed transform signal wherein the output of the inverse transformation unit contains reconstructed block(s). The inverse quantization / inverse transform block 204 may also comprisea block filter (not shown) which may filter the reconstructed block(s) according to further decoded information and filter parameters.

[0051] The entropy coding block 230 receives the output 295, and may receive filter control data 296 from the filter 216, and may perform a suitable entropy encoding / variable length encoding to provide error detection and correction capability. The output (bitstreams 101) of the entropy coding block 230 can be influenced by signaling (not shown) as to whether inter prediction or intra prediction is being used. The output of the entropy coding block 230 may be inserted into a bitstreams 101. Entropy coding / decoding may be performed in many ways. For example, context-based coding / decoding may be applied, where in both the encoder and the decoder modify the context state of a coding parameter based on previously coded / decoded coding parameters. Context based coding may for example be context adaptive binary arithmetic coding (CAB AC), or context-based variable length coding (CAVLC) or any similar entropy coding. Entropy coding / decoding may alternatively or additionally be performed using a variable length coding scheme, such as Huffman coding / decoding or Exp-Golomb coding / decoding. Decoding of coding parameters from an entropy - coded bitstream or codewords may be referred to as parsing.

[0052] In some video codecs, such as H.265 / HEVC and H.266 / WC, the video pictures are divided into smaller blocks, called coding units (CU) for H.265 / HEVC and H.266 / WC as well as other standards, covering the area of the picture. Those blocks are referred to as CU in the following but always also imply other terms for those smaller blocks. Such a blocks consists of one or more units, called prediction units (PUs) for H.265 / HEVC andH.266 / VVC, defining the prediction process for the samples within the CU and one or more units defining the prediction error coding process for the samples, called transform units (TUs) for H.265 / HEVC and H.266 / WC, in the said CU. Typically, a CU includes a rectangular block of samples with a size selectable from a predefined set of possible CU sizes. A CU with the maximum allowed size may be named as LCU (largest coding unit) or CTU (coding tree unit) as example, and the video picture is divided into non-overlapping CTUs. Those blocks are referred to as CTUs in the following while addressing the general concept. A CTU can be further split into a combination of smaller CUs, e.g., by recursively splitting the CTU and resultant CUs. Each resulting CU typically has at least one PU and at least one TU associated with it. Each PU and TU can be further split into smaller PUs and TUs to increase granularity of the predictionand prediction error coding processes, respectively. Each PU has prediction information associated with the PU defining what kind of a prediction is to be applied for the pixels within that PU (e.g., motion vector information for inter predicted PUs and intra prediction directionality information for intra predicted PUs). Similarly, each TU is associated with information describing the prediction error decoding process for the samples within the TU (including, e.g., DCT coefficient information). It is typically signaled at CU level whether prediction error coding is applied or not for each CU. In the case there is no prediction error residual associated with the CU, it can be considered there are no TUs for the said CU. The division of the image into CUs, and division of CUs into PUs and TUs is typically signaled in the bitstream allowing the decoder to reproduce the intended structure of these units.

[0053] In terms of the examples herein, they address what happens with methods for padding in regions outside pictures (e.g., also referred to as frames) from video. As an introduction to these concepts, FIG. 2A illustrates an example of motion compensation with a motion vector pointing to a block that is partially outside a boundary of a picture for inter prediction. A current block 280 from a current picture (e.g., picture N, from the input video 110-1) 270-1 is being analyzed, e.g., in inter prediction block 206, for motion compensation. This uses the previous picture 270-2, e.g., picture N-l, from the decoded picture buffer 218. A motion vector (MV) 275 for the motion compensation points to part of the reference block 285 that is outside the boundary 290 of the picture 270-2, such that part of block 285 is in an out-of-bounds area 297. Using the reference blocks 285 that have areas outside the boundary 290, as compared to ignoring the parts of reference blocks 285 that have areas outside the boundary 290, leads to more efficient and better video coding. As described below, there are techniques for forming an extension region in the out-of-bounds area of reference block 285 to aid in this processing, and the instant methods improve upon this process.

[0054] Turning now to decoding, the decoder reconstructs the output video by applying prediction means similar to the encoder to form a predicted representation of the pixel blocks (using the motion or spatial information created by the encoder and stored in the compressed representation) and prediction error decoding (inverse operation of the prediction error coding recovering the quantized prediction error signal in spatial pixel domain). After applying prediction and prediction error decoding means the decoder sums up the prediction and prediction error signals (pixel values) to form the output video frame. The decoder (and encoder)can also apply additional filtering means to improve the quality of the output video before passing it for display and / or storing it as prediction reference for the forthcoming frames in the video sequence. The decoding process is illustrated in FIG. 3.

[0055] Turning to FIG. 3, this figure illustrates a decoder such as a WC decoder 140 where the decoder examples can be implemented. The examples herein can be implemented in processes involving the inter prediction block 355, the filters block 340, or for decoded pictures in the decoded picture buffer 350. For the decoded pictures, an extension region (as described below) may be added before storage in the decoded picture buffer 350, and the decoded picture would be stored with the extension region in the decoded picture buffer 350, or the extension region may be added to a decoded picture added before use by the inter prediction block 385. If the padding region is added before just before the inter prediction, the padding may be performed only partially (since it is not stored) to save computations.

[0056] As an overview of decoding, the decoding process starts with entropy decoding using, e.g., a context-adaptive binary arithmetic coding (CAB AC) engine, followed by inverse quantization and inverse transform that results in the decoded residue. The decoded residue is added to the prediction signal (intra, inter, or mix of both in case of combined intrainter prediction mode (CIIP)). The resulting reconstructed signal is then processed through different in-loop filtering steps.

[0057] In more detail, the (e.g., coded) bitstreams 101 are decoded initially by an entropy decoder 305, which typically uses CABAC (context adaptive binary arithmetic coding). An inverse quantization (Q1) 310 and an inverse transform (T-1) 315 are performed. This transforms the video information from the frequency domain to the time domain. This decoded residue is further processed as follows.

[0058] LMCS (luma mapping with chroma scaling) 320 is performed on the decoded residue, and this includes chroma residue scaling. LMCS contains two components: luma mapping (LM) and luma-dependent chroma residue scaling (CS). The basic idea of luma mapping is to make better use of the range of luma code values at a specified bit depth, as some luma code values may not be used in the input video. The CS is designed to compensate for the LM impact on the bit cost repartition between the luma signal and the chroma signal. For LMCS 320, this performs CS, which in general determines a chroma scaling factor and scales the chroma residue values according to the scaling factor. The following steps may be performed indecoding for CS: a) inverse quantization and inverse transform processes are applied to the decoded chroma transform coefficients to produce chroma residue-scaled values; b) the chroma residue values are obtained by multiplying the chroma residue-scaled values by an inverse scaling factor; c) reconstructed chroma sample values are obtained by summing the chroma residue values with the corresponding predicted chroma value.

[0059] The adder 325 adds output of the LMCS 320 to a prediction signal 376. Output of the adder 325 is operated on by LMCS 330, which performs inverse luma mapping. Inverse luma mapping maps the luma code values from the mapped sample domain back to the original sample domain. LMCS 360 is performed in a later pathway. Overall in decoding, the following steps are performed for LM: a) inverse quantization (310) and inverse transform (315) are applied to the decoded luma transform coefficients to produce the luma residues in the mapped sample domain, Y'res; b) reconstructed luma sample values in the mapped sample domain, Y'r, are obtained by summing Y'reswith the corresponding predicted luma values in the mapped sample domain, Y'pred(for intra prediction 335, Y'predis directly obtained by performing intra prediction in mapped sample domain, while for inter prediction 355, the predicted luma values in original sample domain, Ypred, are first obtained by motion compensation using reference pictures from the decoded picture buffer (DPB, 350), and then forward luma mapping in 360 is applied to produce the luma values in the mapped sample domain K'pred,); and c) the reconstructed values, which are the sum of Y'predand Y'res, are then inverse-mapped (see 330) and processed by other in-loop filters (see 340) before being stored in the DPB 350 (in the original sample domain).

[0060] Intra prediction 335 is applied to the output of LMCS 330. The corresponding output 336 is one input to the intra / inter / combined selection switch 375. Filters 340 are also applied to the output of the LMCS 330. The filters may include a deblocking filter, an SAG (Sample adaptive offset) filter (e.g., for luma), CC-SAO (cross-component sample adaptive offset filter) (e.g., for chroma), ALF (Adaptive loop filter), and CC-ALF (cross-component adaptive loop filter). Output of the filters 340 includes a reconstructed picture 345 (which could be part of output video, e.g., to a screen, video output, or the like), and be put into a decoded picture buffer (DPB) 350.

[0061] Pictures from the decoded picture buffer 350 are used for inter prediction 355, the output of which is operated on by the LMCS (with forward luma mapping) 360, which produces output 361 that is an input to the intra / inter / combined selection switch 375. The CIIP (combined intra-inter prediction mode) 365 provides another input via output 366 to the intra / inter / combined selection switch 375. The CIIP is introduced to take advantage of both an inter-prediction merge mode and intra prediction. In CIIP mode, the final prediction is a weighted combination of a merge mode inter-picture prediction and a planar mode intra-picture prediction. The combining weight is implicitly derived based on whether the above and left neighboring CUs are coded using an intra-picture prediction mode or not.

[0062] The intra / inter / combined selection switch 375 produces the prediction signal 376, which is added by adder 325 to the output of the LMCS 320. The intra / inter / combined selection switch 375 is controlled to select one of the inputs 361, 366, or 336. For additional information in this area, see Marta Karczewicz, et al., WC In-Loop Filters, IEEE Transactions on Circuits and Systems for Video Technology, Vol. 31, No. 10, October 2021.

[0063] As previously stated, by varying the fidelity of the quantization process, the encoder can control the balance between the accuracy of the pixel representation (picture quality) and size of the resulting coded video representation (file size or transmission bitrate). Instead, or in addition to approaches utilizing sample value prediction and transform coding for indicating the coded sample values, a color palette based coding can be used. Palette-based coding refers to a family of approaches for which a palette, i.e., a set of colors and associated indexes, is defined and the value for each sample within a coding unit is expressed by indicating its index in the palette. Palette based coding can typically achieve good coding efficiency in coding units with a relatively small number of colors (such as image areas which are representing computer screen content, like text or simple graphics). In order to improve the coding efficiency of palette coding different kinds of palette index prediction approaches can be utilized, or the palette indexes can be run-length coded to be able to represent larger homogenous image areas efficiently. Also, in the case the CU contains sample values that are not recurring within the CU, escape coding can be utilized. Escape coded samples are transmitted without referring to any of the palette indexes. Instead, their values are indicated individually for each escape coded sample.

[0064] In typical video codecs, the motion information is indicated with motion vectors associated with each motion compensated image block. Each of these motion vectorsrepresents the displacement of the image block in the picture to be coded (in the encoder side) or decoded (in the decoder side) and the prediction source block in one of the previously coded or decoded pictures. To represent motion vectors efficiently, those are typically coded differentially with respect to block-specific predicted motion vectors. In typical video codecs, the predicted motion vectors are created in a predefined way, for example calculating the median of the encoded or decoded motion vectors of the adjacent blocks. Another way to create motion vector predictions is to generate a list of candidate predictions from adjacent blocks and / or co-located blocks in temporal reference pictures and signaling the chosen candidate as the motion vector predictor. In addition to predicting the motion vector values, the reference index of previously coded / decoded picture can be predicted. The reference index is typically predicted from adjacent blocks and / or or co-located blocks in temporal reference picture. Moreover, typical high efficiency video codecs employ an additional motion information coding / decoding mechanism, often called merging or merge mode, where all the motion field information, which includes motion vector and corresponding reference picture index for each available reference picture list, is predicted and used, typically without any modification / correction. Similarly, predicting the motion field information is carried out using the motion field information of adjacent blocks and / or co-located blocks in temporal reference pictures and the used motion field information is signaled among a list of motion field candidate list filled with motion field information of available adjacent / co-located blocks. This type of prediction is typically referred to as inter prediction.

[0065] Typically, video codecs support motion compensated prediction from at least one source image (uni-prediction) and two sources (bi-prediction). In the case of uni-prediction a single motion vector is applied whereas in the case of bi-prediction two motion vectors are determined and the motion compensated predictions from two sources are combined to create the final sample prediction. In the case of weighted prediction the relative weights of the two predictions can be adjusted, or a signaled offset can be added to the prediction signal.

[0066] In addition to applying motion compensation for inter picture prediction, similar approach can be applied to intra picture prediction. In this case the displacement vector indicates where from the same picture a block of samples can be copied to form a prediction of the block to be coded or decoded. This kind of intra block copying (IBC) methods can improvethe coding efficiency substantially in presence of repeating structures within the frame - such as text or other graphics.

[0067] In typical video codecs, the prediction residual after motion compensation or intra prediction is first transformed with a transform kernel (such as DCT) and then coded. The reason for this is that often there still exists some correlation among the residual and transform can in many cases help reduce this correlation and provide more efficient coding.

[0068] Typical video encoders utilize Lagrangian cost functions to find optimal coding modes, e.g., the desired CTU mode and associated motion vectors. This kind of cost function uses a weighting factor A. to tie together the (exact or estimated) image distortion due to lossy coding methods and the (exact or estimated) amount of information that is required to represent the pixel values in an image area:C = D + R, (Eq. 1) where C is the Lagrangian cost to be minimized, D is the image distortion (e.g., Mean Squared Error) with the mode and motion vectors considered, and R the number of bits needed to represent the required data to reconstruct the image block in the decoder (including the amount of data to represent the candidate motion vectors).

[0069] Video codecs might also use artificial intelligence, such as neural networks, to find coding modes instead or in addition to Lagrangian cost optimization.

[0070] Typically, video is encoded in YUV or YCbCr color space as that is found to reflect some characteristics of human visual system and allows using lower quality representation for Cb and Cr channels as human perception is less sensitive to the chrominance fidelity those channels represent. Each of YUV / YCbCr is a color model based on one luminance (Y) and two chrominance / color difference channels (UV or CbCr) that is typically used in many video coding applications.

[0071] When inter prediction is applied, it is generally possible to allow for motion vectors that (partly) point beyond the picture boundary. See FIG. 2A and its description. In order to generate an intact prediction signal, the out-of-bound samples need to be defined. This process is referred to as reference picture padding in the following. One low-complex way to do this is to assign the spatially closest in-bound sample value to the out-of-bound position of the reference picture. This can be achieved by padding the reference picture with its border sample values,hence the name. The described method is widely used in standards such as AVC, HEVC and WC but alternative methods are possible as well.

[0072] Reference picture padding by repetitive padding, as previously described, is a low-complex way of defining out-of-bound samples. An alternative approach introduces an angular prediction, similar to what is performed in intra-prediction, which is used to predict the out-of-bound samples and thereby define them. The prediction angle is derived, based on the block content, and the angle is signaled in the bitstream.

[0073] Other techniques use template matching structure prediction to define the out-of-bound samples. The method has a number of parameters which can be varied and thereby alternate the padding result.

[0074] The previously described methods all only use information from within the reference picture itself and can therefore be categorized as intra reference picture padding. It has to be noted that other image extrapolation methods known from image processing are also suitable for the task of reference picture padding.

[0075] On the other hand, techniques for motion compensated reference picture padding use information from other reference pictures to define the out-of-bound samples of a current reference picture. This is performed by following the motion vectors of the boundary blocks to their corresponding other reference picture. If a continuation of the boundary block in the direction perpendicular to the boundary exists in the other reference picture, it is used as padding in the current reference picture. Area that cannot be padded with content from other reference pictures is padded with repetitive padding.

[0076] Current video coding schemes use a lookup mechanism, that selects the nearest in-bound sample value as the value for the out-of-bound sample. This mechanism cannot continue directionality and structure accurately outwards. Consider the following. Current video coding schemes use a lookup mechanism, that selects the nearest in-bound sample value as the value for the out-of-bound sample, thereby repeating the outmost sample in the direction perpendicular to the boundary. Directionality which is different from that direction cannot be continued accurately outwards with this lookup mechanism. Additionally, this scheme cannot continue structure accurately outward. This is described in more detail in reference to FIG. 4, see where the lookup method is presented as padding method A.

[0077] Therefore, a number of other methods have been developed to attempt to correct this. Currently, however, in video coding schemes, a single method with fixed parameters is used throughout decoding, which does not allow one to adapt the padding method on varying situations such as picture type or content of the picture.

[0078] The examples herein solve issues with this technique. In particular, the examples solve the problem of selecting an appropriate reference picture padding method and the corresponding parameters on the encoder side or decoder side, and implementing the signaling to the decoder side. Since a selection of different variants is possible with the examples, it is possible to apply different methods for different units, which can be flexible regions with different spatial and or temporal properties.

[0079] The examples are equally applicable to subpictures or other spatial partitioning where out-of-bound samples need to be defined. The examples are further applicable for loop filter operations where out-of-bound samples need to be defined. It is further described how a selection of the applicable method and its parameters can be performed on the encoder side or decoder side. Furthermore, a signaling scheme is described that allows for indication of the selected method and parameters to the decoder side.

[0080] In one example, a method is described that selects a padding method for a region, where the region can be, e.g., a picture or subpicture. This is performed by choosing a sub-region (which can be the full region / picture itself or a crop of the full region / picture) and performing the considered padding methods for a defined outer region. This can be performed on the reconstructed signal or on a related signal such as the prediction signal without residual, the output of a loop filter stage, or the original signal. In a second step, the performance of the methods is evaluated by comparison with a defined continuation for the outer region, which is referred to as comparison signal in the following. In this context, “defined” refers to comprising of defined samples. This continuation can be the (partly) defined continuation of the reconstructed signal or the (partly) defined continuation of a related signal such as the original signal. A distance metric, e.g., mean-squared-error (MSE), is used to rank the performance. The selection of the method is then performed based on the ranking or a rate distortion optimization, involving consideration of the signaling cost. It is noted that the term “signal” herein may be interpreted as a picture, depending on where the signal is being referred to in an encoder or decoder. For instance, a reconstructed signal could be a decoded picture in the decoded picturebuffer 218 of the encoder 130 of FIG. 2, and an original signal could be an original picture from the input video 110-1.

[0081] An example is given in FIG. 4. A discussion of one of the problems is described before proceeding with the details presented by FIG. 4. There are two “padded” regions 480-1 and 480-2 that have had individual and different padding methods applied to subregions in them, where the corresponding padding method has been applied to a sub-region 410. See padded region 480-1, where a padding method has been applied to sub-region 410 that selects the nearest in-bound sample value as the value for the out-of-bound sample and continues this process. This creates an extension region 460-1 that is obviously incorrect relative to the extension region 460-2, which uses a different padding method (e.g., some type of angular prediction) on sub-region 410. Thus, the padding method 415 being used affects the quality of the resultant extension region 460. This is addressed in the following.

[0082] FIG. 4 provides an overview, and FIGS. 4 A and 4B provide two different possible scenarios as to how FIG. 4 may be implemented. Here, in FIG. 4, the sub-region 410 is chosen to be a crop of a (sub)picture. The same sub-region 410 is used for two different padding methods 415, and the padding that is performed allows comparison to be performed with a comparison signal. The comparison 497 is indicated by the lines pointing toward the extension regions 460-1 and 460-2 and the defined continuation 470 that is an extension of the sub-region 475. In this example, the comparison signal is the defined continuation 470.

[0083] The two padding methods A 415-1 and B 415-2 are applied to the sub-region 410 to generate an extension region 460-1 and 460-2 for each method 415. The extension region 460 is an outer region defined around the sub-region 410, and extends from the outer boundary 420 of the sub-region 410 to outer boundary 440 of the extension region 460. While this shows padding to be performed on all four sides of the region, this may be performed by performing the padding on the top, then the right, then the left, then the bottom, or performing one or more of these in parallel. It may also be that not all four sides would be padded, such as one side being padded as in FIG. 2A, or two sides could be padded, and the like. Those extension regions 460 are compared 497 with a comparison signal that is the defined continuation 470. The metric calculation 425 is performed for a metric for the two padding methods 415-1 and 415-2. The mean-squared-error (MSE), although other metrics are possible as well, is used in this example to calculate the difference between the continuation of the sub-region 410 generated by themethods 415 and the extension of the sub-region 475 that results in the defined continuation 470 that is used as the comparison signal. The sub-region 475 has its own outer boundary 420-1, and the defined continuation 470 has its own outer boundary 441. In this example, the methods A 415-1 and B 415-2 are ranked by a metric, e.g., MSE, and the method with the lowest MSE is selected. Other implementations may involve the signaling cost (if applicable) and results of the comparison metric to form a joint cost criterion, as typically performed in Rate Distortion Optimization (RDO).

[0084] FIG. 4 illustrates an overview, and this can be implemented in multiple ways. In particular, the input that is used as the sub-region 475 and the input that is used as the defined continuation 470 can be different. FIGS. 4A and 4B are examples.

[0085] Referring to FIG. 4A, this figure is an example of a scenario used to implement FIG. 4. In this example, there is an original region 490, e.g., part or all of an original picture from the input video 110-1, having the sub-region 475 and the defined continuation 470. The encoder 130 performs part of the encoding process 131 on the original region 490 to create a coded version region 492 that is in the decoded picture buffer 218. See FIG. 2 for more explanation of the encoding process from the input video 110-1 to the decoded picture buffer 218. The original region 490 is considered to be related content to the coded version region 492. The original region 490 is part of the initial picture and the coded version region 492 shows the same content but with small differences (potentially lower quality and coding artifacts). They are (e.g., exactly) the same in terms of content. The coded version region 492 has the sub-region 410 and its own defined continuation 406. The encoder 130 performs 495 cropping and padding of the coded version region 492 to form the padded region 480-1 using method A 415-1 and to form the padded region 480-2 using method B 415-2. The comparison 497 is between the corresponding extension regions 460-1 and 460-2 and the defined continuation 470 (only the comparison between the extension region 460-2 and the defined continuation 470 is shown).

[0086] FIG. 4B is another example of a scenario used to implement FIG. 4. In this example, the original region 490 is still processed by the encoder 130 using the encoding process 131 to the coded version region 492. However, the sub-regions 410 and 475 are from the coded version region 492, and the comparison 497 is performed by comparing the extension regions 460 with a comparison signal that is the defined continuation 406 / 470 of the coded version region 492.

[0087] It is noted that the possible cropping and padding methods for individual subpictures may depend on the location of the subpicture in the picture. Consider FIG. 4C, which illustrates a picture 400 partitioned into regions 465-1 to 465-9. If the picture is partitioned into regions (which are subpictures in this example), then sometimes the crop is not needed because the continuation is known on the encoder side. This is described in the following:

[0088] 1) For region 1 465-1, the bottom and right side do not need to be cropped;

[0089] 2) For region 2465-2, only the top side needs to be cropped, since the continuation for all other sides is known;

[0090] 3) For region 3465-3, the left and bottom side do not need to be cropped;

[0091] 4) For region 4465-4, only the left side should be cropped;

[0092] 5) For region 5465-5, there is no need to crop at all; and

[0093] 6) These techniques would be applied to regions 6465-6, 7465-7, 8465-8, and 9465-9.

[0094] This should all be covered by selecting the appropriate region, which could be different from the region 465. For instance, the region may extend over the bottom and right side for region 1 465-1, and then a meaningful sub-region may be defined which defines the crop. Consider region 468, which extends over the right and bottom sides of region 1 may be selected as an appropriate regions, and the sub-region 467 could be selected, cropped from the region 465-1 (and also from region 468), and used as sub-region 410 in the previous examples. The defined continuation 470 is then defined by the regions that were not cropped and that exist in region 468.

[0095] Referring to FIG. 5, this figure illustrates a flowchart of an encoding method for selection of an applicable padding method in a coding scheme for inter prediction. The operations performed in FIG. 5 are assumed to be performed as part of the encoding process 131 performed by the encoder 130 in the transmitting apparatus 180-1. In operation 505, the encoder 130 accesses multiple padding methods for a region of a picture (or a related picture), where the region can be, e.g., the picture or a subpicture of the picture (or the related picture). A picture or a subpicture of the picture for the region are commonly used, but these are not the only possibilities. Instead, the region could be a coding tree unit (CTU); a macroblock; or an arbitrary shape. With respect to FIGS. 4, 4A, and 4B, the sub-region 410 could be the sub-region 410 thatis a subpicture cropped out of the region 492. In operation 507, a sub-region 410 of the region is chosen, and this sub-region could be, e.g., the full region or a crop of it.

[0096] In terms of the inter prediction process and the encoder 131 of FIG. 2, the subregion 410 from the coded version region 492 from the decoded picture buffer 218 is used, and this is used so that the subsequent selected padding method 415 that “wins” the evaluation is applied to the entire coded version region 492. It is also possible that the sub- region 475 of the original (and related) region 490 could be used for the evaluation and selecting of the padding method 415. This is a “decoupled” analysis, in the sense that the analysis would be performed using the sub-region 475 on the original (and related) region 490, but the result of the analysis (the “winning” padding method is selected) is then applied to the entire coded version region 492.

[0097] In operation 510, the encoder 130 performs the multiple padding methods 415 to perform padding of an outer region defined outside (e.g., an outer boundary 420 of) the subregion 410. Operation 515 is one example of operation 510. The multiple padding methods 415 are performed on the reconstructed signal (e.g., the coded version region 492) or on a related signal such as the prediction signal without residual, the output of a loop filter stage, or the original signal (e.g., the original region 490). The encoder 130, in operation 520, evaluates the performance of the multiple padding methods at least by comparison of output of individual ones of the multiple padding methods with a comparison signal (e.g., for a defined continuation 470) for the outer region, where the evaluating performs a ranking. For instance, the comparison signal was a defined continuation 470 from the original region 490 or from the coded version region 492. As another example, the ranking used above was MSE, but other metrics, e.g., of distortion, may be used.

[0098] Operation 530 involves the encoder 130 selecting one of the multiple padding methods 415 (e.g., and its parameters) to perform the padding based on the ranking. In terms of parameters, one parameter that could be used for a picture is the attribute that picture identifies as a Random Access Point (RAP). Another parameter could be that the picture itself has only reference pictures which are in the past (temporal direction: they are shown on the display before the picture itself is shown), making it a low-delay picture. A third parameter could further involve the minimum distance of those reference pictures to the picture itself. These are examples and other parameters are possible.1

[0099] There are multiple options for operation 530, which are described by operations 535, 540, and 545. The encoder in operation 535 uses a metric of a distortion approximation for the ranking, and in operation 540 uses the distortion approximation and signaling cost of the padding for the ranking. Another option is to perform an RDO process and test encode with different padding methods and derive the ranking. See operation 550. For instance, test encoding may be performed for each method and then the encoder uses an equation that unifies the rate and distortion into a single number and the ranking is then based on this single number.

[0100] In operation 555, the encoder 130 performs the selected padding method 415 on the region 492 to form a defined continuation 470 defined outside the region 492. The encoder signals (see operation 560) indication of the selected padding method 415 (e.g., and parameters) to the decoder side. More information on possible signaling techniques is described below. Further, operation 560 may be skipped in certain instances, e.g., where both the encoder and decoder select padding method based on a predefined logic (e.g., logic defined at least before the padding methods are selected).

[0101] In another embodiment, the padding method 415 is derived based on the region attributes. For example, if the sub-region 410 itself has a reference picture from two temporal directions in its reference picture list(s), e.g., resulting in structural delay, method A might be chosen, while otherwise method B is chosen. The method would be chosen because it generally works better than the other methods when there is no structural delay, and instead of signaling this one could fix this for the decoder. In another example, if the region only uses intra prediction, method A is chosen, while otherwise method B is chosen.

[0102] Turning to FIG. 5A, this figure illustrates a flowchart of an encoding method for selection of an applicable padding method in a coding scheme for inter prediction based on region attributes. FIG. 5 A is a modified version of FIG. 5 and is modified for the example where the padding method 415 is derived based on the region attributes.

[0103] One way to consider what is happening in FIG. 5A compared to FIG. 5 is that operations 507, 510 and 520 are not performed, and operation 565 in FIG. 5A is a version of the selection in operation 530 of FIG. 5. The selection in FIG. 5 A uses a ranking that is based on attributes of the region. That is, the attributes are information that provide a ranking to enable the selection.

[0104] Operation 505 is the same as before. In operation 565, the encoder 130 selects one of the multiple padding methods by deriving a padding method 415 based on attributes of the sub-region 410. Operation 565 can be considered to be also be a version of operation 520, in the sense that deriving the padding methods based on attributes of the region inherently evaluates and ranks the padding methods. Two examples of operation 565 are operations 570 or 575. In operation 570, the encoder derives the padding method based on the region itself having a reference picture from two temporal directions in its reference picture list(s), e.g., resulting in structural delay. The structural delay is used in deciding the method, and is one example of an attribute. Other attributes may be used. In operation 575, the encoder derives the padding method based on the region if the region only allows for intra prediction. This means the picture has no dependencies from other pictures (e.g., and is a Random Access Point RAP or Intra-frame). Intra prediction is one attribute of the region and provides the information that provides a ranking. Operations 555 and 560 have already been described.

[0105] In another embodiment, a selection process for loop filter padding methods, which define the out-of-bound samples in a loop filter process, is defined. This is performed by performing the method on a sub-region 410 (which typically is a sub-region of a region / picture, but can also be the region / picture itself) and performing the considered padding methods 415 for a defined outer region (e.g., an extension region 460). This can be performed on the input content for the loop filter stage or on related content such as an intermediate loop filter stage or the original content. In a second step, the performance of the padding methods is evaluated by comparison with a (partly) defined continuation for the outer region. In this context “defines” refers to comprising of defined samples. The continuation can be the (partly) defined continuation of the coded content (in the region) or the (partly) defined continuation of related content. The “(partly) defined continuation” means partly or completely defined continuation. A distance metric, e.g., mean-squared-error (MSE) may be used to rank the performance. The selection of the padding method is then performed based on the ranking or a rate distortion optimization, involving consideration of the signaling cost.

[0106] FIG. 5B illustrates a flowchart of an encoding method for selection of an applicable padding method in a coding scheme for a loop filter. FIG. 5B is a modified version of FIG. 5 and is modified for the example where the padding method is selected and used in acoding scheme for a loop filter such as the ALF 216-4. Many of the operations have already been described in reference to FIG. 5, and mainly the different operations are described here.

[0107] In operation 580, the encoder 130 selects multiple padding methods for a region of a picture used in a loop filter process, where the region can be, e.g., the picture or a subpicture of the picture. Operation 510 in this example may perform operation 585, where the multiple padding methods 415 are performed on the input content for the loop filter stage or on related content such as an intermediate loop filter stage or the original content (e.g., from the input video 110-1 as in the original region 490). The ALF 216-4 may have multiple stages, and an intermediates state is one of those stages. The operation 590 is similar to the operation in operation 520, but with some differences. In operation 590, the encoder evaluates the performance of the multiple padding methods 415 at least by comparison of output of individual ones of the multiple padding methods with a comparison signal 437 for a defined continuation for the outer region (e.g., extension region 460), the evaluating performing a ranking. In this case, the continuation may be a (partly) defined continuation of the coded content (in the region) or a (partly) defined continuation of related content. Related content could be the original signal which is not known to the decoder but is available. Since the related content shows the same content but is different in terms of pixel values, this is referred to as related content. The rest of the operations are the same as in FIG. 5.

[0108] Referring to FIG. 6, this figure illustrates a flowchart of a decoding method for selection and use of an applicable padding method in a coding scheme. This example is applicable to all of the encoding methods in FIGS. 5, 5A, and 5B. The operations in FIG. 6 are assumed to be performed as part of the decoding process 141 performed by the decoder 140 in the receiving apparatus 180-2. In operation 605, the decoder determines a selected padding method (e.g., and parameters) to use in relation to a region of a picture, where the region can be, e.g., the picture or a subpicture. In operation 610, the decoder 140 receives signaling from an encoder that provides indication of a selected padding method (e.g., and parameters). One example of operation 610 is provided in operation 620, where the indication could be for an inter prediction process with corresponding information (e.g., a selected padding method possibly with corresponding parameters, e.g., where the method should be applied in the decoder) for the inter prediction process. Another example is provided in operation 625, where the indication could be for a loop filter(s) process with corresponding information (e.g., the correspondingpadding method for a filter, indication of the filter or filter stage, indication of which data to use for the region). The information for operations 620 and 625 allows the decoder 140 to use the selected padding method 415 in the correct process (e.g., the inter prediction process, involving the inter prediction 355 and the decoded picture buffer 350, or the filtering process performed by filters 340 such as an ALF). Another option is in operation 630, where the decoder selects padding method based on a predefined logic. This is described in more detail below. The decoder 140 applies, in operation 640, the selected padding method 415 (e.g., and parameters) to the region (e.g., region 492 for inter prediction) to form an extension region that is defined outside the region.

[0109] In another embodiment, a method of signaling the selected method to the decoder side is described. This method of signaling may be combined with any of the methods above. First, a list is defined, that contains a padding method and its parameters (if applicable) for every entry. This list can be fixed or signaled initially, e.g., in the high-level-syntax of the coding scheme. The index of the list is then encoded in the bitstream.

[0110] This is described based on FIG. 7, which is a flowchart of a method of signaling a selected method to the decoder side. In operation 710, one of the two ends of the coding process defines a list of multiple padding methods 415 to perform padding of a region, where the region can be, e.g., a picture or subpicture. This is typically performed by the encoder, but may be performed by the decoder or possibly a human being. One example of operation 710 is to form a list to contain a padding method 415 (e.g., and corresponding parameters) for individual entries that are indexed. See operation 720. That is, knowing the index will allow one to use a specific padding method. In operation 730, the two ends (encoder and decoder) operate so each has the list. This could be performed by (see operation 740) fixing the list, e.g., in a standard. This is an example where each side accesses a programmed list. In operation 750, the list is signaled between the two ends, e.g., in high-level-syntax. This is typically performed from the encoder to the decoder, but it is possible for the signaling to be from the decoder to the encoder. In operation 760, one of the ends encodes indication (e.g., index) of a padding method to use and sends the indication to other end. This is typically performed from the encoder to the decoder, but tis possible to go from the decoder to the encoder.

[0111] Additional embodiments are as follows. For ease of reference, these provide categories by using letters.

[0112] a) In one embodiment, the selection is performed for a padding method for a video coding tool.

[0113] b) In one embodiment, the selection is performed for a padding method for reference picture padding.

[0114] c) In one embodiment, the selection is performed for a padding method for reference subpicture padding.

[0115] d) In one embodiment, the selection is performed for a padding method of a loop filter process.

[0116] e) In one embodiment, the selection of the method is performed on the decoder side. Consider a predefined logic that has, e.g., the picture parameters (or other parameters) as input. Encoder and decoder then apply the same logic. The encoder leaves out the last operation 560 in the flow chart. The decoder instead of getting the selection from the bitstream applies the predefined logic.

[0117] f) In one embodiment, the selection of the method is performed on the decoder side for reference picture padding.

[0118] g) In one embodiment, the selection of the method is performed on the decoder side for reference subpicture padding.

[0119] h) In one embodiment, the selection of the method is performed on the decoder side for a loop filter process.

[0120] i) In one embodiment, the selection of the method is performed on the encoder side and signaled to the decoder side.

[0121] j) In one embodiment, the selection of the method is performed on the encoder side and signaled to the decoder side for reference picture padding.

[0122] k) In one embodiment, the selection of the method is performed on the encoder side and signaled to the decoder side for reference subpicture padding.

[0123] 1) In one embodiment, the selection of the method is performed on the encoder side and signaled to the decoder side for a loop filter process.

[0124] m) In one embodiment, the selection is performed on a region basis.

[0125] n) In one embodiment, the selection is performed on a per-picture basis.

[0126] o) In one embodiment, the selection is performed on a per-subpicture basis, when subpictures are present.

[0127] p) In one embodiment, the selection is performed on a fixed region basis, such as a CTU.

[0128] q) In one embodiment, signaling of the selected method from the encoder to decoder side is performed.

[0129] r) In one embodiment, signaling of the selected method from the encoder to decoder side is performed by encoding the index a list of methods in the bitstream.

[0130] s) In one embodiment, signaling of the selected method from the encoder to decoder side is performed by encoding the index a list of methods in the bitstream for different pictures or regions.

[0131] t) In one embodiment, signaling of the selected method from the encoder to decoder side is performed by encoding the index a list of methods in the bitstream for different pictures or regions and encoding the list of methods in the bitstream.

[0132] u) In one embodiment, signaling of the selected method from the encoder to decoder side is performed for the reference picture padding method (combined with above embodiments q-t).

[0133] v) In one embodiment, signaling of the selected method from the encoder to decoder side is performed for the reference subpicture padding method (combined with above embodiments q-t).

[0134] w) In one embodiment, signaling of the selected method from the encoder to decoder side is performed for the padding method of a loop filter process (combined with above embodiments q-t).

[0135] An example for signaling is given in the following. There are three padding methods A, B and C that are used for padding in the coding scheme. Further, method C allows for adaption of the padding method by varying the parameter x. A list comprising the padding methods and parameters is defined as follows:

[0136] l. A

[0137] 2. B

[0138] 3. C: x=5

[0139] 4. C: x=10

[0140] The padding method is chosen adaptively on a picture basis. The coding scheme allows one to define the list in the bitstream. Therefore, first the list is encoded in thehigh-level syntax of the coding scheme. This could, e.g., be performed in the Sequence Parameter Set or a high-level syntax set with a similar purpose. The encoder 130 determines the applicable method from the given list for each picture and encodes the index of the list in the bitstream with unary encoding for this example. Coding with 2-bit fixed length, a golomb code or other binarization schemes are possible as well.

[0141] The example could look as follows using H.266 / VVC syntax and a signaling in the Picture header of H.266 / VVC utilizing unsigned integer fixed length binarization scheme.

[0142] The following shows a table for the changed picture header syntax which has now an entry of ph_padding_id which contains the index of the predefined list from the example

[0143] Semantics of the above include the following.

[0144] ph_padding_id specifies the index of the padding method in the reference picture padding list. It is coded with unsigned integer using 2 bits.

[0145] The predefined reference picture padding list Table could look as follows:

[0146] In another example, the list comprises of two methods A and B and is fixed for the coding scheme. The encoder decides to use method A for intra pictures and B for all other pictures. The choice of the padding method is encoded by a single bit in the high-level-syntax of the bitstream, e.g., in the picture parameter set.

[0147] The following shows a table for the changed picture header syntax which has now an entry of ph_alt_padding_flag which signals if an alternative padding should be used.

[0148] The logic of which method would be applied is given in pseudo code in the

[0149] Possible semantics include the following. Note that these values may be reversed or otherwise modified.

[0150] ph_alt_padding_flag equal to 0 (zero) specifies that padding method A is applied for the current picture. A value of 1 (one) specifies that padding method B is used for reference picture padding.

[0151] Turning to FIG. 8, this figure is an example of a block diagram of an apparatus 180 suitable for implementing any of the encoders or decoders described herein. The apparatus 180 includes circuitry comprising one or more processors 820, one or more memories 825, one or more transceivers 830, one or more network (N / W) interface(s) (I / F(s)) 855 and user interface (UI) circuitry and elements 857, interconnected through one or more buses 827.Depending on implementation, some apparatus may not have all of the circuitry. For example, an apparatus 180 might not have UI circuitry and elements 857. An apparatus may have additional circuitry, not described here. FIG. 8 is presented merely as an example.

[0152] Each of the one or more transceivers 830 includes a receiver, Rx, 832 and a transmitter, Tx, 833. The one or more buses 827 may be address, data, and / or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment, and the like. The one or more transceivers 830 are connected to one or more antennas 805, and may communicate using wireless link 811, which could implement any number of wireless communication interfaces such as Wi-Fi, cellular, or satellite.

[0153] The one or more memories 825 include computer program code 823. The apparatus 180 includes a program 840, comprising one of or both parts 840-1 and / or 840-2. The program 840 may implement an encoder 130, a decoder 140, or a codec (130 + 140), which implements both encoding and decoding. The program itself may be implemented in a number of ways. The program 840 may be implemented in circuitry as program 840-1, such as being implemented as part of the one or more processors 820, and contains instructions implemented in circuitry. The program 840-1 may be implemented also as an integrated circuit or through other circuitry such as a programmable gate array. In another example, the program 840 may be implemented as program 840-2, which is implemented as computer program code (having corresponding instructions) 823 and is executed by the one or more processors 820. For instance,the one or more memories 825 store instructions that, when executed by the one or more processors 820, cause the apparatus 180 to perform one or more of the operations as described herein.

[0154] The network interface(s) (N / W I / F(s)) 855 are wired interfaces communicating using link(s) 856, which could be fiber optic or other wired interfaces. The apparatus 180 could include only wireless transceiver(s) 830, only N / W I / Fs 855, or both wireless transceiver(s) 830 and N / W I / Fs 855.

[0155] The apparatus 180 may or may not include UI circuitry and elements 857. These could include a display such as a touchscreen, speakers, or interface elements such as for headsets. For instance, an apparatus 180 of a smartphone would typically include at least a touchscreen and speakers. The UI circuitry and elements 857 may also include circuity to communicate with external UI elements (not shown) such as displays, keyboards, mice, headsets, and the like.

[0156] The computer readable memories 825 may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, flash memory, firmware, magnetic memory devices and systems, optical memory devices and systems, fixed memory and removable memory. The processor(s) 820 may be of any type suitable to the local technical environment, and may include one or more of general-purpose computers, special purpose computers, microprocessors, digital signal processors (DSPs) and processors based on a multi-core processor architecture, as nonlimiting examples. The processor(s) 820 control the apparatus 180 to perform the operations as described herein. The processor(s) 820 may execute instructions, including microcode, but are not implemented solely in software.

[0157] Without in any way limiting the scope, interpretation, or application of the claims appearing below, a technical effect and / or advantage of one or more of the example embodiments disclosed herein is reduced coding artifact at boundaries, which can be visually disturbing. Another technical effect and / or advantage of one or more of the example embodiments disclosed herein is improved compression efficiency.

[0158] The following are additional examples.

[0159] Example 1. A method, comprising: performing operations comprising the following for an encoding process for input video: selecting one of multiple padding methods toperform padding of an outer region defined outside a region, the region from a picture used in the encoding process, and the selecting performed based on information providing a ranking of the multiple padding methods; and performing the selected padding method to form an extension region defined outside the region.

[0160] Example 2. The method according to example 1, further comprising signaling indication of the selected padding method toward a decoder.

[0161] Example 3. The method according to example 2, wherein there is a list of the multiple padding methods, the selecting selects one of the multiple padding methods in the list, and the signaling the indication signals the indication of the selected one of the multiple padding methods from the list.

[0162] Example 4. The method according to example 1, wherein selecting one of the multiple padding methods comprises selecting the selected padding method of the multiple padding methods based on a predefined logic.

[0163] Example 5. The method according to any of examples 1 to 4, wherein the region is one of the following: the picture; a subpicture of the picture; a coding tree unit (CTU); a macroblock; or an arbitrary shape.

[0164] Example 6. The method according to any of examples 1 to 5, wherein the method further comprises, prior to the selecting one of the multiple padding methods: selecting a sub-region of the region; accessing the multiple padding methods to be used to perform padding of the outer region for the sub-region; performing the multiple padding methods using the subregion to perform the padding of the outer region of the sub-region; and evaluating performance of the multiple padding methods that perform the padding of the outer region of the sub-region by comparing output of individual ones of the multiple padding methods with a comparison signal for the outer region of the sub-region, where the evaluating produces the information providing the ranking of the padding methods.

[0165] Example 7. The method according to example 6, wherein: the performing the multiple padding methods is performed for an inter prediction process.

[0166] Example 8. The method according to example 7, wherein the evaluating performance of the multiple padding methods that perform the padding of the outer region of the sub-region comprises performing one of the following: using a metric of a distortion approximation for the comparing output for the ranking; using the distortion approximation andsignaling cost of the padding for the comparing output for the ranking; or performing a rate distortion optimization process and test encoding with different padding methods of the multiple padding methods and deriving the ranking to perform the comparing output.

[0167] Example 9. The method according to any of examples 6 to 8, wherein: the sub-region is defined from within a picture from a decoded picture buffer; and the comparison signal is from a defined continuation of another sub-region in a region from an original picture from the input video.

[0168] Example 10. The method according to any of examples 6 to 8, wherein: the sub-region is defined from within a picture from a decoded picture buffer; and the comparison signal is from a defined continuation of the sub-region that was defined from within the picture from the decoded picture buffer.

[0169] Example 11. The method according to any of examples 6 to 8, wherein: the sub-region is defined from within an original picture from the input video, and the comparison signal is from a defined continuation of the sub-region that was defined from within the original picture from the input video.

[0170] Example 12. The method according to example 6, wherein: the performing the multiple padding methods is performed using a process performed by a loop filter, and performed on input content for a loop filter stage or on related content.

[0171] Example 13. The method according to example 12, wherein the related content comprises one of content from an intermediate loop filter stage or original content.

[0172] Example 14. The method according to example 12, wherein the comparison signal is for a defined continuation for the outer region, and the defined continuation is a partly or completely defined continuation of coded content in the sub-region or a partly or completely defined continuation of the related content.

[0173] Example 15. The method according to any of examples 12 to 14, wherein the evaluating performance of the multiple padding methods that perform the padding of the outer region of the sub-region comprises performing one of the following: using a metric of a distortion approximation for the comparing output for the ranking; using the distortion approximation and signaling cost of the padding for the comparing output for the ranking; or performing a rate distortion optimization process and test encoding with different paddingmethods of the multiple padding methods and deriving the ranking to perform the comparing output.

[0174] Example 16. The method according to any of examples 1 to 15, wherein the padding of the outer region is performed from an outer boundary of the region to an outer boundary of the outer region.

[0175] Example 17. The method according to example 1, wherein the selecting one of multiple padding methods comprises: deriving the selected padding method based on one or more attributes of the region.

[0176] Example 18. The method according to example 17, wherein the deriving comprises deriving the selected padding method based on the region having a reference picture from two temporal directions in its reference picture list or reference picture lists.

[0177] Example 19. The method according to example 17, wherein the deriving comprises deriving the selected padding method based on the region only using intra prediction.

[0178] Example 20. A method, comprising: performing operations comprising the following for a decoding process using a bitstream: determining a selected padding method to use in relation to a region of a picture from the bitstream; and applying the selected padding method to the region to form an extension region that is defined outside the region.

[0179] Example 21. The method according to example 20, wherein: the method further comprises receiving signaling from an encoder that provides indication of the selected padding method, wherein the indication is for an inter prediction process with corresponding information for the inter prediction process; and the applying comprises applying, using the information, the selected padding method to the region in the inter prediction process to form an extension region that is defined outside the region.

[0180] Example 22. The method according to example 20, further comprising: receiving signaling from an encoder that provides indication of the selected padding method, wherein the indication is for a loop filter process with corresponding information for the loop filter process; and the applying comprises applying, using the information, the selected padding method to the region in the loop filter process to form an extension region that is defined outside the region.

[0181] Example 23. The method according to example 20, wherein determining the selected padding method to use comprises determining the selected padding method to use based on a predefined logic.

[0182] Example 24. An apparatus, comprising means for: performing operations comprising the following for an encoding process for input video: selecting one of multiple padding methods to perform padding of an outer region defined outside a region, the region from a picture used in the encoding process, and the selecting performed based on information providing a ranking of the multiple padding methods; and performing the selected padding method to form an extension region defined outside the region.

[0183] Example 25. The apparatus according to example 24, wherein the means are further configured for signaling indication of the selected padding method toward a decoder.

[0184] Example 26. The apparatus according to example 25, wherein there is a list of the multiple padding methods, the selecting selects one of the multiple padding methods in the list, and the signaling the indication signals the indication of the selected one of the multiple padding methods from the list.

[0185] Example 27. The apparatus according to example 24, wherein selecting one of the multiple padding methods comprises selecting the selected padding method of the multiple padding methods based on a predefined logic.

[0186] Example 28. The apparatus according to any of examples 24 to 27, wherein the region is one of the following: the picture; a subpicture of the picture; a coding tree unit (CTU); a macroblock; or an arbitrary shape.

[0187] Example 29. The apparatus according to any of examples 24 to 28, wherein the means are further configured for , prior to the selecting one of the multiple padding methods: selecting a sub-region of the region; accessing the multiple padding methods to be used to perform padding of the outer region for the sub-region; performing the multiple padding methods using the sub-region to perform the padding of the outer region of the sub-region; and evaluating performance of the multiple padding methods that perform the padding of the outer region of the sub-region by comparing output of individual ones of the multiple padding methods with a comparison signal for the outer region of the sub-region, where the evaluating produces the information providing the ranking of the padding methods.

[0188] Example 30. The apparatus according to example 29, wherein: the performing the multiple padding methods is performed for an inter prediction process.

[0189] Example 31. The apparatus according to example 30, wherein the evaluating performance of the multiple padding methods that perform the padding of the outer region of the sub-region comprises performing one of the following: using a metric of a distortion approximation for the comparing output for the ranking; using the distortion approximation and signaling cost of the padding for the comparing output for the ranking; or performing a rate distortion optimization process and test encoding with different padding methods of the multiple padding methods and deriving the ranking to perform the comparing output.

[0190] Example 32. The apparatus according to any of examples 29 to 31, wherein: the sub-region is defined from within a picture from a decoded picture buffer; and the comparison signal is from a defined continuation of another sub-region in a region from an original picture from the input video.

[0191] Example 33. The apparatus according to any of examples 29 to 31, wherein: the sub-region is defined from within a picture from a decoded picture buffer; and the comparison signal is from a defined continuation of the sub-region that was defined from within the picture from the decoded picture buffer.

[0192] Example 34. The apparatus according to any of examples 29 to 31, wherein: the sub-region is defined from within an original picture from the input video, and the comparison signal is from a defined continuation of the sub-region that was defined from within the original picture from the input video.

[0193] Example 35. The apparatus according to example 29, wherein: the performing the multiple padding methods is performed using a process performed by a loop filter, and performed on input content for a loop filter stage or on related content.

[0194] Example 36. The apparatus according to example 35, wherein the related content comprises one of content from an intermediate loop filter stage or original content.

[0195] Example 37. The apparatus according to example 35, wherein the comparison signal is for a defined continuation for the outer region, and the defined continuation is a partly or completely defined continuation of coded content in the sub-region or a partly or completely defined continuation of the related content.

[0196] Example 38. The apparatus according to any of examples 35 to 37, wherein the evaluating performance of the multiple padding methods that perform the padding of the outer region of the sub-region comprises performing one of the following: using a metric of a distortion approximation for the comparing output for the ranking; using the distortion approximation and signaling cost of the padding for the comparing output for the ranking; or performing a rate distortion optimization process and test encoding with different padding methods of the multiple padding methods and deriving the ranking to perform the comparing output.

[0197] Example 39. The apparatus according to any of examples 24 to 38, wherein the padding of the outer region is performed from an outer boundary of the region to an outer boundary of the outer region.

[0198] Example 40. The apparatus according to example 24, wherein the selecting one of multiple padding methods comprises: deriving the selected padding method based on one or more attributes of the region.

[0199] Example 41. The apparatus according to example 40, wherein the deriving comprises deriving the selected padding method based on the region having a reference picture from two temporal directions in its reference picture list or reference picture lists.

[0200] Example 42. The apparatus according to example 40, wherein the deriving comprises deriving the selected padding method based on the region only using intra prediction.

[0201] Example 43. An apparatus, comprising means for: performing operations comprising the following for a decoding process using a bitstream: determining a selected padding method to use in relation to a region of a picture from the bitstream; and applying the selected padding method to the region to form an extension region that is defined outside the region.

[0202] Example 44. The apparatus according to example 43, wherein: the means are further configured for receiving signaling from an encoder that provides indication of the selected padding method, wherein the indication is for an inter prediction process with corresponding information for the inter prediction process; and the applying comprises applying, using the information, the selected padding method to the region in the inter prediction process to form an extension region that is defined outside the region.

[0203] Example 45. The apparatus according to example 43, wherein the means are further configured for: receiving signaling from an encoder that provides indication of the selected padding method, wherein the indication is for a loop filter process with corresponding information for the loop filter process; and the applying comprises applying, using the information, the selected padding method to the region in the loop filter process to form an extension region that is defined outside the region.

[0204] Example 46. The apparatus according to example 43, wherein determining the selected padding method to use comprises determining the selected padding method to use based on a predefined logic.

[0205] Example 47. An apparatus, comprising: one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, cause the apparatus at least to perform: performing operations comprising the following for an encoding process for input video: selecting one of multiple padding methods to perform padding of an outer region defined outside a region, the region from a picture used in the encoding process, and the selecting performed based on information providing a ranking of the multiple padding methods; and performing the selected padding method to form an extension region defined outside the region.

[0206] Example 48. The apparatus according to example 47, wherein the one or more memories further store instructions that, when executed by the one or more processors, cause the apparatus at least to perform signaling indication of the selected padding method toward a decoder.

[0207] Example 49. The apparatus according to example 48, wherein there is a list of the multiple padding methods, the selecting selects one of the multiple padding methods in the list, and the signaling the indication signals the indication of the selected one of the multiple padding methods from the list.

[0208] Example 50. The apparatus according to example 47, wherein selecting one of the multiple padding methods comprises selecting the selected padding method of the multiple padding methods based on a predefined logic.

[0209] Example 51. The apparatus according to any of examples 47 to 50, wherein the region is one of the following: the picture; a subpicture of the picture; a coding tree unit (CTU); a macroblock; or an arbitrary shape.

[0210] Example 52. The apparatus according to any of examples 47 to 51, wherein the one or more memories further store instructions that, when executed by the one or more processors, cause the apparatus at least to perform, prior to the selecting one of the multiple padding methods: selecting a sub-region of the region; accessing the multiple padding methods to be used to perform padding of the outer region for the sub-region; performing the multiple padding methods using the sub-region to perform the padding of the outer region of the subregion; and evaluating performance of the multiple padding methods that perform the padding of the outer region of the sub-region by comparing output of individual ones of the multiple padding methods with a comparison signal for the outer region of the sub-region, where the evaluating produces the information providing the ranking of the padding methods.

[0211] Example 53. The apparatus according to example 52, wherein: the performing the multiple padding methods is performed for an inter prediction process.

[0212] Example 54. The apparatus according to example 53, wherein the evaluating performance of the multiple padding methods that perform the padding of the outer region of the sub-region comprises performing one of the following: using a metric of a distortion approximation for the comparing output for the ranking; using the distortion approximation and signaling cost of the padding for the comparing output for the ranking; or performing a rate distortion optimization process and test encoding with different padding methods of the multiple padding methods and deriving the ranking to perform the comparing output.

[0213] Example 55. The apparatus according to any of examples 52 to 54, wherein: the sub-region is defined from within a picture from a decoded picture buffer; and the comparison signal is from a defined continuation of another sub-region in a region from an original picture from the input video.

[0214] Example 56. The apparatus according to any of examples 52 to 54, wherein: the sub-region is defined from within a picture from a decoded picture buffer; and the comparison signal is from a defined continuation of the sub-region that was defined from within the picture from the decoded picture buffer.

[0215] Example 57. The apparatus according to any of examples 52 to 54, wherein: the sub-region is defined from within an original picture from the input video, and the comparison signal is from a defined continuation of the sub-region that was defined from within the original picture from the input video.

[0216] Example 58. The apparatus according to example 52, wherein: the performing the multiple padding methods is performed using a process performed by a loop filter, and performed on input content for a loop filter stage or on related content.

[0217] Example 59. The apparatus according to example 58, wherein the related content comprises one of content from an intermediate loop filter stage or original content.

[0218] Example 60. The apparatus according to example 58, wherein the comparison signal is for a defined continuation for the outer region, and the defined continuation is a partly or completely defined continuation of coded content in the sub-region or a partly or completely defined continuation of the related content.

[0219] Example 61. The apparatus according to any of examples 58 to 60, wherein the evaluating performance of the multiple padding methods that perform the padding of the outer region of the sub-region comprises performing one of the following: using a metric of a distortion approximation for the comparing output for the ranking; using the distortion approximation and signaling cost of the padding for the comparing output for the ranking; or performing a rate distortion optimization process and test encoding with different padding methods of the multiple padding methods and deriving the ranking to perform the comparing output.

[0220] Example 62. The apparatus according to any of examples 47 to 61, wherein the padding of the outer region is performed from an outer boundary of the region to an outer boundary of the outer region.

[0221] Example 63. The apparatus according to example 47, wherein the selecting one of multiple padding methods comprises: deriving the selected padding method based on one or more attributes of the region.

[0222] Example 64. The apparatus according to example 63, wherein the deriving comprises deriving the selected padding method based on the region having a reference picture from two temporal directions in its reference picture list or reference picture lists.

[0223] Example 65. The apparatus according to example 63, wherein the deriving comprises deriving the selected padding method based on the region only using intra prediction.

[0224] Example 66. An apparatus, comprising: one or more processors; and one or more memories storing instructions that, when executed by the one or more processors, cause the apparatus at least to perform: performing operations comprising the following for a decodingprocess using a bitstream: determining a selected padding method to use in relation to a region of a picture from the bitstream; and applying the selected padding method to the region to form an extension region that is defined outside the region.

[0225] Example 67. The apparatus according to example 66, wherein: the one or more memories further store instructions that, when executed by the one or more processors, cause the apparatus at least to perform receiving signaling from an encoder that provides indication of the selected padding method, wherein the indication is for an inter prediction process with corresponding information for the inter prediction process; and the applying comprises applying, using the information, the selected padding method to the region in the inter prediction process to form an extension region that is defined outside the region.

[0226] Example 68. The apparatus according to example 66, wherein the one or more memories further store instructions that, when executed by the one or more processors, cause the apparatus at least to perform: receiving signaling from an encoder that provides indication of the selected padding method, wherein the indication is for a loop filter process with corresponding information for the loop filter process; and the applying comprises applying, using the information, the selected padding method to the region in the loop filter process to form an extension region that is defined outside the region.

[0227] Example 69. The apparatus according to example 66, wherein determining the selected padding method to use comprises determining the selected padding method to use based on a predefined logic.

[0228] Example 70. A computer program, comprising instructions which, when the program is executed by an apparatus, cause the apparatus to carry out the methods of any of examples 1 to 23.

[0229] Example 71. The computer program according to example 70, wherein the computer program is a computer program product comprising a computer-readable medium bearing the instructions embodied therein for use with the apparatus.

[0230] Example 72. The computer program according to example 70, wherein the computer program is directly loadable into an internal memory of the apparatus.

[0231] As used in this application, the term “circuitry” may refer to one or more or all of the following:

[0232] (a) hardware-only circuit implementations (such as implementations in analog, digital, and / or quantum circuitry) and

[0233] (b) combinations of hardware circuits and software such as (as applicable): (i) a combination of analog, digital, and / or quantum hardware circuit(s) with software / firmware and (ii) any or all portions of hardware processor(s) (including digital and / or quantum processor(s)) with software, and memory(ies) that work together to cause an apparatus, such as a mobile device, computing device, or server, to perform various functions) and

[0234] (c) any or all portions of hardware circuit(s), such as microprocessor(s), processor(s) and / or quantum processors, that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.

[0235] This definition of circuitry applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.

[0236] In an example embodiment, software (e.g., application logic, an instruction set) as used herein is maintained on any one of various conventional computer-readable media. In the context of this document, a “computer-readable medium” may be any media or means that can contain, store, communicate, propagate or transport the instructions for use by or in connection with an instruction execution system, apparatus, or device, such as a computer, with one example of a computer described and depicted, e.g., in FIG. 8. A computer-readable medium may comprise a computer-readable storage medium (e.g., memories 825 or other device) that may be any media or means that can contain, store, and / or transport the instructions for use by or in connection with an instruction execution system, apparatus, or device, such as a computer. A computer-readable storage medium does not comprise propagating signals, and therefore may be considered to be non-transitory. The term “non-transitory”, as used herein, is a limitation of the medium itself (i.e., tangible, not a signal) as opposed to a limitation on data storage persistency (e.g., RAM, random access memory, versus ROM, read-only memory).

[0237] If desired, the different functions discussed herein may be performed in a different order and / or concurrently with each other. Furthermore, if desired, one or more of the above-described functions may be optional or may be combined.

[0238] Although various aspects of the invention are set out in the independent claims, other aspects of the invention comprise other combinations of features from the described embodiments and / or the dependent claims with the features of the independent claims, and not solely the combinations explicitly set out in the claims.

[0239] It is also noted herein that while the above describes example embodiments of the invention, these descriptions should not be viewed in a limiting sense. Rather, there are several variations and modifications which may be made without departing from the scope of the present invention as defined in the appended claims.

[0240] The following abbreviations that may be found in the specification and / or the drawing figures are defined as follows:

[0241] ALF Adaptive loop filter

[0242] AVC Advanced video coding (ITU-T H.264 video coding standard)

[0243] BCW Bi-prediction with CU-level weight

[0244] CABAC Context-Adaptive Binary Arithmetic Coding

[0245] CCCM Convolutional Cross-Component Model

[0246] CCRM Cross-Component Residual Model / CrossComponent Reconstruction Model

[0247] CIIP combined intra-inter prediction mode

[0248] CS chroma residue scaling

[0249] CTU coding tree unit

[0250] CU coding unit

[0251] DBF deblocking filter

[0252] DCT discrete cosine transform

[0253] DPB decoded picture buffer

[0254] DST Discrete Sine Transform

[0255] ECM Enhanced Compression Model (JVET’s exploratory video codec)

[0256] EECM Enhanced Compression Model

[0257] HEVC High Efficiency Video Coding (ITU-T H.265 video coding standard)

[0258] IBC intra block copying

[0259] JVET Joint video experts team

[0260] LCU largest coding unit

[0261] LIC Local Illumination Compensation

[0262] LM luma mapping

[0263] LMCS luma mapping with chroma scaling

[0264] MSE Mean Squared Error

[0265] MV motion vector

[0266] PU prediction unit

[0267] RDO Rate Distortion Optimization

[0268] RGB red, green, blue

[0269] SAG Sample adaptive offset

[0270] TU transform unit

[0271] WC Versatile Video Coding (ITU-T H.266 video coding standard)

[0272] WP Weighted prediction

[0273] YUV / YCbCr a color model based on one luminance and two chrominance / color difference channels (typically used in many video coding applications)

Claims

Claims:

1. A method, comprising:performing operations comprising the following for an encoding process for input video: selecting one of multiple padding methods to perform padding of an outer region defined outside a region, the region from a picture used in the encoding process, and the selecting performed based on information providing a ranking of the multiple padding methods; andperforming the selected padding method to form an extension region defined outside the region.

2. The method according to claim 1, further comprising signaling indication of the selected padding method toward a decoder.

3. The method according to claim 2, wherein there is a list of the multiple padding methods, the selecting selects one of the multiple padding methods in the list, and the signaling the indication signals the indication of the selected one of the multiple padding methods from the list.

4. The method according to claim 1, wherein selecting one of the multiple padding methods comprises selecting the selected padding method of the multiple padding methods based on a predefined logic.

5. The method according to any of claims 1 to 4, wherein the region is one of the following:the picture; a subpicture of the picture; a coding tree unit (CTU); a macroblock; or an arbitrary shape.

6. The method according to any of claims 1 to 5, wherein the method further comprises, prior to the selecting one of the multiple padding methods:selecting a sub-region of the region;46accessing the multiple padding methods to be used to perform padding of the outer region for the sub-region;performing the multiple padding methods using the sub-region to perform the padding of the outer region of the sub-region; andevaluating performance of the multiple padding methods that perform the padding of the outer region of the sub-region by comparing output of individual ones of the multiple padding methods with a comparison signal for the outer region of the sub-region, where the evaluating produces the information providing the ranking of the padding methods.

7. The method according to claim 6, wherein:the performing the multiple padding methods is performed for an inter prediction process.

8. The method according to claim 7, wherein the evaluating performance of the multiple padding methods that perform the padding of the outer region of the sub-region comprises performing one of the following:using a metric of a distortion approximation for the comparing output for the ranking; using the distortion approximation and signaling cost of the padding for the comparing output for the ranking; orperforming a rate distortion optimization process and test encoding with different padding methods of the multiple padding methods and deriving the ranking to perform the comparing output.

9. The method according to any of claims 6 to 8, wherein:the sub-region is defined from within a picture from a decoded picture buffer; and the comparison signal is from a defined continuation of another sub-region in a region from an original picture from the input video.

10. The method according to any of claims 6 to 8, wherein:the sub-region is defined from within a picture from a decoded picture buffer; and47the comparison signal is from a defined continuation of the sub-region that was defined from within the picture from the decoded picture buffer.

11. The method according to any of claims 6 to 8, wherein:the sub-region is defined from within an original picture from the input video, and the comparison signal is from a defined continuation of the sub-region that was defined from within the original picture from the input video.

12. The method according to claim 6, wherein:the performing the multiple padding methods is performed using a process performed by a loop filter, and performed on input content for a loop filter stage or on related content.

13. The method according to claim 12, wherein the related content comprises one of content from an intermediate loop filter stage or original content.

14. The method according to claim 12, wherein the comparison signal is for a defined continuation for the outer region, and the defined continuation is a partly or completely defined continuation of coded content in the sub-region or a partly or completely defined continuation of the related content.

15. The method according to any of claims 12 to 14, wherein the evaluating performance of the multiple padding methods that perform the padding of the outer region of the subregion comprises performing one of the following:using a metric of a distortion approximation for the comparing output for the ranking; using the distortion approximation and signaling cost of the padding for the comparing output for the ranking; orperforming a rate distortion optimization process and test encoding with different padding methods of the multiple padding methods and deriving the ranking to perform the comparing output.4816. The method according to any of claims 1 to 15, wherein the padding of the outer region is performed from an outer boundary of the region to an outer boundary of the outer region.

17. The method according to claim 1, wherein the selecting one of multiple padding methods comprises:deriving the selected padding method based on one or more attributes of the region.

18. The method according to claim 17, wherein the deriving comprises deriving the selected padding method based on the region having a reference picture from two temporal directions in its reference picture list or reference picture lists.

19. The method according to claim 17, wherein the deriving comprises deriving the selected padding method based on the region only using intra prediction.

20. A method, comprising:performing operations comprising the following for a decoding process using a bitstream: determining a selected padding method to use in relation to a region of a picture from the bitstream; andapplying the selected padding method to the region to form an extension region that is defined outside the region.

21. The method according to claim 20, wherein:the method further comprises receiving signaling from an encoder that provides indication of the selected padding method, wherein the indication is for an inter prediction process with corresponding information for the inter prediction process; andthe applying comprises applying, using the information, the selected padding method to the region in the inter prediction process to form an extension region that is defined outside the region.

22. The method according to claim 20, further comprising:receiving signaling from an encoder that provides indication of the selected padding method, wherein the indication is for a loop filter process with corresponding information for the loop filter process; andthe applying comprises applying, using the information, the selected padding method to the region in the loop filter process to form an extension region that is defined outside the region.

23. The method according to claim 20, wherein determining the selected padding method to use comprises determining the selected padding method to use based on a predefined logic.

24. An apparatus, comprising means for:performing operations comprising the following for an encoding process for input video: selecting one of multiple padding methods to perform padding of an outer region defined outside a region, the region from a picture used in the encoding process, and the selecting performed based on information providing a ranking of the multiple padding methods; andperforming the selected padding method to form an extension region defined outside the region.

25. An apparatus, comprising means for:performing operations comprising the following for a decoding process using a bitstream: determining a selected padding method to use in relation to a region of a picture from the bitstream; andapplying the selected padding method to the region to form an extension region that is defined outside the region.

26. An apparatus, comprising:one or more processors; andone or more memories storing instructions that, when executed by the one or more processors, cause the apparatus at least to perform:performing operations comprising the following for an encoding process for input video: selecting one of multiple padding methods to perform padding of an outer region defined outside a region, the region from a picture used in the encoding process, and the selecting performed based on information providing a ranking of the multiple padding methods; andperforming the selected padding method to form an extension region defined outside the region.

27. The apparatus according to claim 26, wherein the one or more memories further store instructions that, when executed by the one or more processors, cause the apparatus at least to perform signaling indication of the selected padding method toward a decoder.

28. The apparatus according to claim 27, wherein there is a list of the multiple padding methods, the selecting selects one of the multiple padding methods in the list, and the signaling the indication signals the indication of the selected one of the multiple padding methods from the list.

29. The apparatus according to claim 26, wherein selecting one of the multiple padding methods comprises selecting the selected padding method of the multiple padding methods based on a predefined logic.

30. The apparatus according to any of claims 26 to 29, wherein the region is one of the following: the picture; a subpicture of the picture; a coding tree unit (CTU); a macroblock; or an arbitrary shape.

31. The apparatus according to any of claims 26 to 30, wherein the one or more memories further store instructions that, when executed by the one or more processors, cause the apparatus at least to perform, prior to the selecting one of the multiple padding methods: selecting a sub-region of the region;accessing the multiple padding methods to be used to perform padding of the outer region for the sub-region;performing the multiple padding methods using the sub-region to perform the padding of the outer region of the sub-region; andevaluating performance of the multiple padding methods that perform the padding of the outer region of the sub-region by comparing output of individual ones of the multiple padding methods with a comparison signal for the outer region of the sub-region, where the evaluating produces the information providing the ranking of the padding methods.

32. The apparatus according to claim 31, wherein:the performing the multiple padding methods is performed for an inter prediction process.

33. The apparatus according to claim 32, wherein the evaluating performance of the multiple padding methods that perform the padding of the outer region of the sub-region comprises performing one of the following:using a metric of a distortion approximation for the comparing output for the ranking; using the distortion approximation and signaling cost of the padding for the comparing output for the ranking; orperforming a rate distortion optimization process and test encoding with different padding methods of the multiple padding methods and deriving the ranking to perform the comparing output.

34. The apparatus according to any of claims 31 to 33, wherein:the sub-region is defined from within a picture from a decoded picture buffer; and the comparison signal is from a defined continuation of another sub-region in a region from an original picture from the input video.

35. The apparatus according to any of claims 31 to 33, wherein:the sub-region is defined from within a picture from a decoded picture buffer; and52the comparison signal is from a defined continuation of the sub-region that was defined from within the picture from the decoded picture buffer.

36. The apparatus according to any of claims 31 to 33, wherein:the sub-region is defined from within an original picture from the input video, and the comparison signal is from a defined continuation of the sub-region that was defined from within the original picture from the input video.

37. The apparatus according to claim 31, wherein:the performing the multiple padding methods is performed using a process performed by a loop filter, and performed on input content for a loop filter stage or on related content.

38. The apparatus according to claim 37, wherein the related content comprises one of content from an intermediate loop filter stage or original content.

39. The apparatus according to claim 37, wherein the comparison signal is for a defined continuation for the outer region, and the defined continuation is a partly or completely defined continuation of coded content in the sub-region or a partly or completely defined continuation of the related content.

40. The apparatus according to any of claims 37 to 39, wherein the evaluating performance of the multiple padding methods that perform the padding of the outer region of the subregion comprises performing one of the following:using a metric of a distortion approximation for the comparing output for the ranking; using the distortion approximation and signaling cost of the padding for the comparing output for the ranking; orperforming a rate distortion optimization process and test encoding with different padding methods of the multiple padding methods and deriving the ranking to perform the comparing output.5341. The apparatus according to any of claims 26 to 40, wherein the padding of the outer region is performed from an outer boundary of the region to an outer boundary of the outer region.

42. The apparatus according to claim 26, wherein the selecting one of multiple padding methods comprises:deriving the selected padding method based on one or more attributes of the region.

43. The apparatus according to claim 42, wherein the deriving comprises deriving the selected padding method based on the region having a reference picture from two temporal directions in its reference picture list or reference picture lists.

44. The apparatus according to claim 42, wherein the deriving comprises deriving the selected padding method based on the region only using intra prediction.

45. An apparatus, comprising:one or more processors; andone or more memories storing instructions that, when executed by the one or more processors, cause the apparatus at least to perform:performing operations comprising the following for a decoding process using a bitstream: determining a selected padding method to use in relation to a region of a picture from the bitstream; andapplying the selected padding method to the region to form an extension region that is defined outside the region.

46. The apparatus according to claim 45, wherein:the one or more memories further store instructions that, when executed by the one or more processors, cause the apparatus at least to perform receiving signaling from an encoder that provides indication of the selected padding method, wherein the indication is for an inter prediction process with corresponding information for the inter prediction process; and54the applying comprises applying, using the information, the selected padding method to the region in the inter prediction process to form an extension region that is defined outside the region.

47. The apparatus according to claim 45, wherein the one or more memories further store instructions that, when executed by the one or more processors, cause the apparatus at least to perform:receiving signaling from an encoder that provides indication of the selected padding method, wherein the indication is for a loop filter process with corresponding information for the loop filter process; andthe applying comprises applying, using the information, the selected padding method to the region in the loop filter process to form an extension region that is defined outside the region.

48. The apparatus according to claim 45, wherein determining the selected padding method to use comprises determining the selected padding method to use based on a predefined logic.55

Images