Extended merge candidates for inter coding units

Abstract

An apparatus including at least one processor; and at least one memory storing instructions that, when executed with the at least one processor, cause the apparatus to perform: determining, for a coding unit in a current picture, a first merge candidate based on a first reference picture or a first pair of reference pictures; determining, for the coding unit in the current picture, a second merge candidate based on a second reference picture or a second pair of reference pictures; and selecting, for use with a motion information prediction, at least one of the determined first merge candidate or the determined second merge candidate.

Description

EXTENDED MERGE CANDIDATES FOR INTER CODING UNITSBACKGROUNDTechnical Field

[0001] The example and non-limiting embodiments relate generally to video coding and decoding and, more particularly, to merge prediction.Brief Description of Prior Developments

[0002] Block-based processing is widely used in video coding, as it provides a good tradeoff between coding efficiency and computational complexity. Intra block copy tools are known to be able to generate a prediction for a current block.SUMMARY

[0003] The following summary is merely intended to be an example. The summary is not intended to limit the scope of the claims.

[0004] In accordance with one aspect, an apparatus is provided comprising: at least one processor; and at least one memory storing instructions that, when executed with the at least one processor, cause the apparatus to perform: determining, for a coding unit in a current picture, a first merge candidate based on a first reference picture or a first pair of reference pictures; determining, for the coding unit in the current picture, a second merge candidate based on a second reference picture or a second pair of reference pictures; and selecting, for use with a motion information prediction, at least one of the determined first merge candidate or the determined second merge candidate.

[0005] In accordance with another aspect, an example method is provided comprising: determining, for a coding unit in a current picture, a first merge candidate based on a first reference picture or a first pair of reference pictures; determining, for the coding unit in the current picture, a second merge candidate based on a second reference picture or a second pairof reference pictures; and selecting, for use with a motion information prediction, at least one of the determined first merge candidate or the determined second merge candidate.

[0006] In accordance with another aspect, an example apparatus is provided comprising: means for determining, for a coding unit in a current picture, a first merge candidate based on a first reference picture or a first pair of reference pictures; means for determining, for the coding unit in the current picture, a second merge candidate based on a second reference picture or a second pair of reference pictures; and means for selecting, for use with a motion information prediction, at least one of the determined first merge candidate or the determined second merge candidate.

[0007] In accordance with another aspect, an example apparatus is provided with a non-transitory computer readable medium comprising program instructions that, when executed by an apparatus, cause the apparatus to perform at least the following: determining, for a coding unit in a current picture, a first merge candidate based on a first reference picture or a first pair of reference pictures; determining, for the coding unit in the current picture, a second merge candidate based on a second reference picture or a second pair of reference pictures; and selecting, for use with a motion information prediction, at least one of the determined first merge candidate or the determined second merge candidate.

[0008] In accordance with another aspect, an apparatus is provided comprising: at least one processor; and at least one memory storing instructions that, when executed with the at least one processor, cause the apparatus to perform: determining, for a coding unit in a current picture, a first merge candidate based on a first reference picture or a first pair of reference pictures; determining, for the coding unit in the current picture, a second merge candidate based on a second reference picture or a second pair of reference pictures; and receiving information for selecting, for use with a motion information prediction, at least one of the determined first merge candidate or the determined second merge candidate.

[0009] In accordance with another aspect, an example method is provided comprising: determining, for a coding unit in a current picture, a first merge candidate based on a first reference picture or a first pair of reference pictures; determining, for the coding unit in the current picture, a second merge candidate based on a second reference picture or a second pair of reference pictures; and receiving information for selecting, for use with a motion informationprediction, at least one of the determined first merge candidate or the determined second merge candidate.

[0010] In accordance with another aspect, an example apparatus is provided comprising: means for determining, for a coding unit in a current picture, a first merge candidate based on a first reference picture or a first pair of reference pictures; means for determining, for the coding unit in the current picture, a second merge candidate based on a second reference picture or a second pair of reference pictures; and means for receiving information for selecting, for use with a motion information prediction, at least one of the determined first merge candidate or the determined second merge candidate.

[0011] In accordance with another aspect, an example apparatus is provided with a non-transitory computer readable medium comprising program instructions that, when executed by an apparatus, cause the apparatus to perform at least the following: determining, for a coding unit in a current picture, a first merge candidate based on a first reference picture or a first pair of reference pictures; determining, for the coding unit in the current picture, a second merge candidate based on a second reference picture or a second pair of reference pictures; and receiving information for selecting, for use with a motion information prediction, at least one of the determined first merge candidate or the determined second merge candidate.

[0012] According to some aspects, there is provided the subject matter of the independent claims. Some further aspects are provided in subject matter of the dependent claims.BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The foregoing aspects and other features are explained in the following description, taken in connection with the accompanying drawings, wherein:

[0014] FIG. 1 is a block diagram of one possible and non-limiting example system in which the example embodiments may be practiced;

[0015] FIG. 2 illustrates a block diagram of an example apparatus (a terminal device) used to implement one or more of the entities in FIG. 1;

[0016] FIG. 3 is a diagram illustrating an example of spatial merge candidates of a current block X;

[0017] FIGs. 4A and 4B are diagrams respectively illustrating an example of a collocated block in a collated picture and a current block in a current picture;

[0018] FIG. 5 is a diagram illustrating an example of HMVP merge candidates which may hold motion information from the past coded inter blocks;

[0019] FIG. 6 is a diagram illustrating, for the current block, an example of non-adjacent spatial merge candidates which may hold motion information from the non-adjacent spatial positions;

[0020] FIG. 7A is a diagram illustrating an example of a current picture having a coding unit, where the coding unit has a merge candidate list of merge candidates;

[0021] FIG. 7B is a diagram illustrating an example of two reference pictures which may be used for two different merge candidates for the coding unit shown in FIG 7A;

[0022] FIG. 8 is a diagram illustrating an example, with reference to two reference pictures, where a motion vector of a merge candidate is scaled down;

[0023] FIG. 9 is a diagram illustrating an example, with reference to two reference pictures, where a motion vector of a merge candidate is stretched;

[0024] FIG. 10 is a diagram illustrating an example, with reference to two reference pictures, where two motion vectors of a merge candidate are respectively scaled down and stretched;

[0025] FIG. 11 is a diagram illustrating an example method; and

[0026] FIG. 12 is a diagram illustrating an example method.DETAILED DESCRIPTION

[0027] Turning to FIG. 1, this figure shows a block diagram of one possible and nonlimiting example system 100 in which the example embodiments may be practiced. An encoder 10 analyzes a source video sequence 5 and produces a coded video sequence 15, which is communicated over a network 25 to a decoder 40. The decoder 40 analyzes the coded video sequence 15 and produces a reconstructed video sequence 35, which may be similar to the source video sequence 5.

[0028] The network 25 may be a singular network, such as a local area network for example. As other examples, the network 25 could include multiple networks, such as a local area network to get to the Internet, through the Internet to another local area network; a cellular network to another cellular network; or other combinations of networks.

[0029] The encoder 10 and decoder 40 may be implemented in many different apparatuses. Referring also to FIG. 2, this figure illustrates a block diagram of an example apparatus (a terminal device 110 in this example) used to implement one or more of the entities in FIG. 1. The term “terminal device” is used since the example of FIG. 1 has a communication that terminates on both ends. The term terminal device is not meant to be limiting, and such a device can be any electronic device able to receive or transmit coded or uncoded video.

[0030] The terminal device 110 includes circuitry comprising one or more processors 120, one or more memories 125, one or more transceivers 130 (or receiver(s) and transmitter(s)), one or more network (N / W) interface(s) (I / Fs) 150, and one or more user I / F(s) 155 interconnected through one or more buses 127. Each of the one or more transceivers 130 includes a receiver, Rx, 132 and a transmitter, Tx, 133. The one or more buses 127 may be address, data, or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment, and the like. The one or more transceivers 130 are connected to one or more antennas 128, which may communicate using wireless link 111 with other devices.

[0031] The one or more memories 125 include computer program code 123 (comprising instructions). The terminal device 110 includes a control module 140. The control module140 may implement the encoder 10, the decoder 40, or a codec 60, which includes both an encoder 10 and a decoder 60.

[0032] The control module 140 comprises one of or both parts 140-1 and / or 140-2, which may be implemented in a number of ways. The control module 140 may be implemented in hardware as control module 140-1, such as being implemented as part of the one or more processors 120. The control module 140-1 may be implemented also as an integrated circuit or through other hardware such as a programmable gate array. In another example, the control module 140 may be implemented as control module 140-2, which is implemented as computer program code 123 and is executed by the one or more processors 120. For instance, the one or more memories 125 and (e.g., instructions in) the computer program code 123 may be configured to, in response to the one or more processors 120 executing the instructions, cause the user equipment 110 to perform one or more of the operations as described herein.

[0033] A wired network connection 160 may be used by the one or more network interfaces 150. The user interface(s) 155 may interface with user interface elements 165 such as, for example, a display, a keyboard, a headset (e.g., only earphones or earphones and microphone), speakers, a microphone, and / or mouse. Some of these can be internal, some can be internal, or there could be a combination of internal and external elements 165. For example, a smartphone as the terminal device 110 could have an internal display and internal speakers as elements 165, or a personal computer as the terminal device 110 could have a display, keyboard, mouse, and speakers that are external to the personal computer.

[0034] Hybrid video codecs, for example ITU-T H.263, H.264 / AVC and HEVC, may encode the video information in two phases. At first, pixel values in a certain picture are (or “block”) are predicted for example by motion compensation means (finding and indicating an area in one of the previously coded video frames that corresponds closely to the block being coded) or by spatial means (using the pixel values around the block to be coded in a specified manner). In the first phase, predictive coding may be applied, for example, as so-called sample prediction and / or so-called syntax prediction.

[0035] In the sample prediction, pixel or sample values in a certain picture area or "block" are predicted. These pixel or sample values can be predicted, for example, using one or more of motion compensation or intra prediction mechanisms.

[0036] Motion compensation mechanisms (which may also be referred to as inter prediction, temporal prediction or motion-compensated temporal prediction or motion-compensated prediction or MCP) involve finding and indicating an area in one of the previously encoded video frames that corresponds closely to the block being coded. Inter prediction may reduce temporal redundancy.

[0037] Intra prediction, where pixel or sample values can be predicted by spatial mechanisms, involves finding and indicating a spatial region relationship. Intra prediction utilizes the fact that adjacent pixels within the same picture are likely to be correlated. Intra prediction can be performed in spatial or transform domain, i.e., either sample values or transform coefficients can be predicted. Intra prediction is typically exploited in intra coding, where no inter prediction is applied.

[0038] In the syntax prediction, which may also be referred to as parameter prediction, syntax elements and / or syntax element values and / or variables derived from syntax elements are predicted from syntax elements (de)coded earlier and / or variables derived earlier. Nonlimiting examples of syntax prediction are provided below.

[0039] In motion vector prediction, motion vectors e.g. for inter and / or inter-view prediction may be coded differentially with respect to a block-specific predicted motion vector. In many video codecs, the predicted motion vectors are created in a predefined way, for example by calculating the median of the encoded or decoded motion vectors of the adjacent blocks. Another way to create motion vector predictions, sometimes referred to as advanced motion vector prediction (AMVP), is to generate a list of candidate predictions from adjacent blocks and / or co-located blocks in temporal reference pictures and signaling the chosen candidate as the motion vector predictor. In addition to predicting the motion vector values, the reference index of previously coded / decoded picture can be predicted. The reference index is typically predicted from adjacent blocks and / or co-located blocks in temporal reference picture. Differential coding of motion vectors is typically disabled across slice boundaries.

[0040] The block partitioning, e.g. from CTU to CUs and down to PUs, may be predicted.

[0041] In filter parameter prediction, the filtering parameters e.g. for sample adaptive offset may be predicted.

[0042] Prediction approaches using image information from a previously coded image can also be called as inter prediction methods which may also be referred to as temporal prediction and motion compensation.

[0043] Prediction approaches using image information within the same image can also be called as intra prediction methods.

[0044] Secondly, the prediction error, i.e. the difference between the predicted block of pixels and the original block of pixels, is coded. This may be done by transforming the difference in pixel values using a specified transform (e.g. Discrete Cosine Transform (DCT) or a variant of it), quantizing the coefficients and entropy coding the quantized coefficients. By varying the fidelity of the quantization process, encoder can control the balance between the accuracy of the pixel representation (picture quality) and size of the resulting coded video representation (file size of transmission bitrate).

[0045] In many video codecs, including H.264 / AVC and HEVC, motion information is indicated by motion vectors associated with each motion compensated image block. Each of these motion vectors represents the displacement of the image block in the picture to be coded (in the encoder) or decoded (at the decoder) and the prediction source block in one of the previously coded or decoded images (or pictures). H.264 / AVC and HEVC, as many other video compression standards, a picture is divided into a mesh of rectangles, for each of which a similar block in one of the reference pictures is indicated for inter prediction. The location of the prediction block is coded as a motion vector that indicates the position of the prediction block relative to the block being coded.

[0046] Versatile Video Coding (VVC) is an international video coding standard, and Enhanced Compression Model (ECM), built on top of VVC, is potentially a future video coding standard currently under development sponsored by joint video experts team (JVET). Both VVC and ECM are block-based video coding standards, where an input picture is divided into Coding Tree Units (CTUs), and each CTU may be further split into Coding Units (CUs). A CU (or block) is coded in either inter-coding mode or intra-coding mode. If the block is in inter-coding mode, the encoder searches for a temporal prediction block in reference picture(s) and signals the decoder on how to find the same prediction block in reference picture(s) at the decoder end. If the block is in intra-coding mode, the encoder constructs a spatial predictionblock from the current picture and signals the decoder on how to form the same spatial prediction block from the current picture at the decoder end.

[0047] In developing Versatile Video Codec (VVC), there are the following new coding tools:• Intra prediction- 67 intra mode with wide angles mode extension- Block size and mode dependent 4 tap interpolation filter- Position dependent intra prediction combination (PDPC)- Cross component linear model intra prediction (CCLM)- Multi-reference line intra prediction- Intra sub-partitions- Weighted intra prediction with matrix multiplication• Inter-picture prediction- Block motion copy with spatial, temporal, history-based, and pairwise average merging candidates- Affine motion inter prediction- sub-block based temporal motion vector prediction- Adaptive motion vector resolution- 8x8 block-based motion compression for temporal motion prediction - High precision (1 / 16 pel) motion vector storage and motion compensation with 8-tap interpolation filter for luma component and 4-tap interpolation filter for chroma component- Triangular partitions- Combined intra and inter prediction- Merge with MVD (MMVD)- Symmetrical MVD coding- Bi-directional optical flow- Decoder side motion vector refinement- Bi-prediction with CU-level weight• Transform, quantization and coefficients coding- Multiple primary transform selection with DCT2, DST7 and DCT8- Secondary transform for low frequency zone- Sub-block transform for inter predicted residual- Dependent quantization with max QP increased from 51 to 63- Transform coefficient coding with sign data hiding- Transform skip residual coding• Entropy Coding- Arithmetic coding engine with adaptive double windows probability update • In loop filter- In-loop reshaping- Deblocking filter with strong longer filter- Sample adaptive offset- Adaptive Loop Filter• Screen content coding:- Current picture referencing with reference region restriction• 360-degree video coding- Horizontal wrap-around motion compensation• High-level syntax and parallel processing- Reference picture management with direct reference picture list signalling - Tile groups with rectangular shape tile groups

[0048] In VVC, each picture is divided into coding tree units (CTUs) similar to HEVC. A picture may also be divided into slices, tiles, bricks and sub-pictures. CTU may be split into smaller CUs using quaternary tree structure. Each CU may be divided using quad-tree and nested multi-type tree including ternary and binary split.

[0049] For a current inter-CU in a current picture, the associated temporal prediction block in reference pictures is represented by motion information (e.g. motion vectors, reference pictures, reference picture lists) with respect to the current CU in the current picture. The encoder signals the motion information to the decoder, and the decoder uses the motion information to form the temporal prediction block from reference pictures.

[0050] In VVC and ECM, for a current CU, its motion information may consist of two parts: motion information prediction (e.g. motion vector prediction - MVP) and motion information delta (e.g. motion vector difference - MVD). For a current CU, its motion information prediction is derived from the motion information of the past inter coded CUs in the current picture or in reference pictures and, on the other hand, the motion information delta is often coded in an explicit manner.

[0051] VVC and ECM supports many new and refined coding tools for deriving the motion information prediction for a current CU. One of the coding tools is merge prediction in which, for a current CU, both the encoder and the decoder construct a same list of merge candidates. The merge candidates hold the motion information of the past inter coded CUs around the current CU both spatially and temporally. The encoder selects a merge candidate (motion information) from the merge candidate list for the current CU, and signals the decoder which merge candidate in the merge candidate list to be used for the current CU.

[0052] In the current design of ECM, for a current CU in a current picture, both encoder and decoder should have a same merge candidate list, which may include the following types of candidates:1) Spatial merge candidates,2) Temporal merge candidates (such as temporal motion vector predictor (TMVP) for example),3) Non-adjacent spatial merge candidates,4) History-based merge candidates (such as history-based motion vector prediction (HMVP) for example),5) Pairwise merge candidates, and6) Others.

[0053] Spatial merge candidates for a current block in a current picture hold the motion information from the spatial neighboring blocks of the current block. FIG. 3 shows an example where, for a current block X, its spatial merge candidates may hold the motion information from the neighboring positions of A0, B0, Al, Bl and / or AB.

[0054] Temporal merge candidates (such as a TMVP for example) for a current block in a current picture hold the motion information around the collocated block in the collocated picture. FIGs. 4A and 4B shows an example where, for a current block X in a current picture, its TMVP may hold the motion information from the right-bottom position cO and the center position cl of the collocated block in the collocated picture. The collocated picture is one of the past coded pictures, and the collocated block is the block with the same geometric position in the collocated picture as the current block in the current picture.

[0055] History-based merge candidates (such as HMVP merge candidates for example) for a current block in a current picture hold the motion information of the past coded inter blocks. FIG. 5 shows a CTU (coding tree unit), which is further partitioned into small coding units or blocks. In the current ECM design, coding of CUs within a CTU follows a zigzag path starting with the top-left CU, as shown in FIG. 5. For the current block X in a current picture shown in FIG. 5, its HMVP candidates may hold the motion information from the past-coded inter blocks 13, 12, 11,....

[0056] Non-adjacent spatial merge candidates for a current picture in a current picture hold the motion information around, but not immediately next to, the current block. FIG. 6 shows an example of non-adjacent spatial merge candidates, where for a current block X (gray) in a current picture, non-adjacent spatial candidates contain the motion information from the spatial non-adjacent positions 6, 7, 8,....

[0057] Referring also to FIG. 7A, in current video coding standards, such as VVC and ECM for example, the encoder and the decoder construct a same merge candidate list for a current block (or CU) 704 in a current picture 706. A merge candidate holds motion information of past coded inter blocks. With features as described herein, there may be provided two reference picture lists: a reference picture list 0 (such as including a reference picture L0) and a reference picture list 1 (such as including a reference picture LI). Each reference picture list may contain multiple reference pictures, and each picture may be assigned a reference index (refldx). Motion information may comprise, for example, motion vectors (mv), reference picture indexes (refldx), and lists of reference picture(s) (L0 / L1)).

[0058] Fig. 7A illustrated, for example, an L0 reference picture 702 with the merge candidate list. Merge candidates 708 in a merge candidate list, for a current block in a currentpicture, hold specific motion information 710 around the current block, either spatially or temporally. The motion information 710 may consist of motion vectors, reference picture indices and reference picture lists for example. In the current video standards, such as VVC and ECM, for a current CU the encoder may evaluate merge candidates based upon a certain coding cost and select a merge candidate with the smallest coding cost. Rate-Distortion (RD) cost has been widely used as the coding cost for video coding, which balances the trade-off between compression efficiency (bitrate) and visual quality (distortion). FIG. 7B is a diagram illustrating an example of features as used herein which may use two reference pictures 706’ (such as L0 as noted below for example) and 706” (such as LI as noted below for example) with two different merge candidates 710 for the coding unit 704 shown in FIG 7A.

[0059] The RD cost J for encoding a current block in a current picture using a merge candidate evaluation may defined as:J = D + Rwhere D is a difference measure between the current block and the reconstructed block, where R is the number of bits required to encode the current block by using the merge candidate, and where is Lagrange multiplier (a parameter that controls the trade-off between rate and distortion).

[0060] In current video coding standards, such as VVC and ECM for example, when evaluating a merge candidate for a current block, the encoder only evaluates the motion vectors over the reference pictures with reference indices held by the merge candidate. To sync with the decoder, for a current block in a current picture, the encoder needs to signal the selected merge candidate to the decoder so that encoder and decoder can build a same prediction block for the current block using the same merge candidate from the same merge candidate list.

[0061] In the current video coding standards, such as VVC and ECM for example, for a current CU in a current picture, there may be up to two reference pictures lists (such as reference picture list 0 and reference picture list 1). Each reference picture list may contain more than one reference pictures. These multiple reference pictures in a reference picture list are indexed, starting with index 0. For example, L0 may represent a reference picture “0”, andLI may represent a reference picture “1”. A merge candidate might contain only LO motion information, or only LI motion information, or both LO and LI motion information.

[0062] With features as described herein, a method and apparatus is describer where a merge candidate may be extended to other available reference pictures for a current block in a current picture. With features as described herein, the encoder and the decoder can use both the original merge candidates and the extended merge candidates for prediction with regard to a current block. With features as described herein, examples are described with three types of merge candidates: merge candidates have only LO motion information (motion information regarding an LO reference picture; L0(mv,refIdx)), merge candidates have only LI motion information (motion information regarding an LI reference picture; Ll(mv,refldx)), and merge candidates have both LO motion information and LI motion information (L0(mv,refIdx), Ll(mv,refldx)). The firsts two types with either only LO motion or only LI motion may be referred to as unidirectional merge candidates, and the third type with both LO and LI motion may be referred to as bidirectional merge candidates.

[0063] Unidirectional

[0064] With reference to the above noted nomenclature, if a merge candidate has only L0 or LI motion information, it is a unidirectional merge candidate. A unidirectional merge candidate holds one motion vector and one reference index for either L0 or L 1. With the motion vector and reference picture index held in a unidirectional merge candidate, the encoder and the decoder can find a same prediction block. Thus, with the above noted nomenclature, the encoder and the decoder can find a prediction block pointed by the motion vector in the indexed reference picture in either L0 or LI.

[0065] Bidirectional

[0066] With reference to the above noted nomenclature, if a merge candidate has both L0 and LI motion information, it is a bidirectional merge candidate. A bidirectional merge candidate holds one motion vector and one reference picture index for L0, and one motion vector and one reference picture index for LI. A bidirectional merge candidate may be based on two (or a pair of) reference pictures: one from reference picture list L0 and another one from reference picture list LI.

[0067] With the motion vectors and reference picture indices held in a bidirectional merge candidate, the encoder and the decoder can find a same prediction block, which may be a (weighted) average of two prediction blocks for example. The two prediction blocks may comprise one prediction block pointed by the L0 motion vector over the reference picture indexed in L0, and the other prediction block pointed by the LI motion vector over the reference picture indexed in LI.

[0068] Extension

[0069] Features as described herein may be used to extend potential candidates to candidates besides original merge candidates. Thus, features as described herein may be used to extend the merge candidates to other reference pictures for a current block in a current picture. The encoder and the decoder may have both a same set of the original merge candidates and the extended merge candidates for the current block. The encoder has the freedom to select one merge candidate from among the original merge candidates and the extended merge candidates. The encoder may also to signal the decoder which merge candidate the encoder has selected.

[0070] Assume that, for a current block in a current picture, the encoder and the decoder have constructed a same list of (original) merge candidates with the motion information of the past coded blocks around the current block, either spatially or temporally. With this extension, merge candidates are now extended, from the original merge candidates, to other reference pictures.

[0071] Unidirectional Example 1

[0072] In one example embodiment, a unidirectional (original) merge candidate contains only L0 motion information, holding:one L0 motion vector of mvL0-MC, andone L0 reference picture index of refIdxL0-MC.The merge candidate may be denoted as MC(mvLQ-MC,refIdxLQ-MC), where MC stands for merge candidate, and mvL0-MCis the motion vector, and refIdxL0-MCis the reference pictureindex held in the merge candidate. With refIdxLQ-MC, the encoder and the decoder can locate the indexed reference picture and its specific POC (Picture Order Count) number. With features as described herein, the merge candidate may be extended to other LO reference pictures of refIdxL0= 0, 1,.... That extension does not include refIdxL0-MC. The extended merge candidate will hold one LO reference picture index refIdxL0and one projected motion vector mvL0-refIdx, denoted as:MC (_mvL0-refidx, re fl dxL0).

[0073] The reference picture index refIdxL0can be equal to 0, 1,..., but not to refIdxL0-MC. The projected motion vector mvL0-refIdxon the reference picture with index refIdxL0may be calculated based upon the temporal distance between the current picture and the reference picture indexed by refIdxL0, and by the temporal distance between the current picture and the reference picture indexed by refIdxL0-MC, as: / POC (cur Pic) — POC(refIdxL0) \ mvL0-refldx= mvL0-MCx (^0C(-cltrPic) _ poC(refIdxLQ-MC) )where:• POC cur Pic) is the picture order count (POC) number of the current picture,• POC(refIdxL0) is the picture order count (POC) number of the reference picture indexed by refIdxL0, and• POC(refIdxL0-MC) is the picture order count (POC) number of the reference picture indexed by refIdxL0-MC.

[0074] With reference to the above, FIG. 8 shows an example of a L0 motion vector of a merge candidate MC(mvL0-MC,refIdxL0-MC). In this example the L0 motion vectormvLo-Mc 802 of the merge candidate MC(mvL0-MC,refIdxL0-MC) is scaled down with the motion vector TnvLQ-refIdx804 from the L0 reference picture of POC(refIdxL0-MC) to the L0 reference picture of POC(refIdxL0).

[0075] Unidirectional Example 2

[0076] In one example embodiment, a unidirectional (original) merge candidate contains only LI motion information, holding:one LI motion vector of mvL1-MC, andone LI reference picture index of refIdxL1-MC.The merge candidate may be denoted as MC(mvL1-MC, refIdxL1-MC), where MC stands for merge candidate, and mvL1-MCis the motion vector and refIdxL1-MCis the reference picture index held in the merge candidate. With refIdxL1-MC, the encoder and the decoder can locate the indexed reference picture and its specific POC (Picture Order Count) number.

[0077] The merge candidate may be extended to other LI reference pictures of refIdxL1= 0, 1,..., but not to refIdxL1-MC. The extended merge candidate will hold one LI reference picture index refIdxL1and one projected motion vector mvL1-refIdx, denoted as:MC(mvL1-refIdx, refIdxL1).

[0078] The reference picture index refIdxL1can be equal to 0, 1,..., but not to refIdxL1-MC. The projected motion vector mvL1-refIdxon the reference picture with index refIdxL1may be calculated based upon the temporal distance between the current picture and the reference picture indexed by refIdxL1, and the temporal distance between the current picture and the reference picture indexed by refIdxL1-MC, as: / POC (cur Pic) — POC(refIdxL1) \ mvL1-refldx= mvL1-MCx \p0C(curPic) _ poC(refIdxL1-MC) )where• POC cur Pic) is the POC number of the current picture,• POC(refIdxL1) is the POC number of the reference picture indexed by refIdxL1, and• POC(refIdxL1-MC) is the POC number of the reference picture indexed by refIdxL1-MC.

[0079] With reference to the above, FIG. 9 shows an example of a LI motion vector of a merge candidate MC(mvL1-MC,refIdxL1-MC). In this example the LI motion vectormvLi-Mc 902 of the merge candidate MC(mvL1-MC,refIdxL1-MC) is stretched with the motion vector nivL1-refIdx904 from the LI reference picture of POC(refIdxL1-MC) to the LI reference picture of POC(refIdxL1).

[0080] Bidirectional Example

[0081] In one example embodiment, a bidirectional (original) merge candidate contains both L0 and LI motion information, holding:one L0 motion vector of mvL0-MC, andone L0 reference picture index of refIdxL0-MC, andone LI motion vector of mvL1-MC, andone LI reference picture index of refIdxL1-MC.The bidirectional merge candidate may be denoted as:MC((mvL0-Mc< re fl dxL0-MCf (mvL1-MC, re fl dxL1-MCffwhere• MC stands for merge candidate,• mvL0-MCis the L0 motion vector,• refIdxL0-MCis the L0 reference picture index,• mvL -MCis the LI motion vector, and• refIdxL1-MCis the LI reference picture index held in the bidirectional merge candidate.

[0082] With refIdxL0-MCand refIdxL1-MC, the encoder and the decoder can locate the indexed reference pictures and their specific POC (Picture Order Count) numbers. The bidirectional merge candidate may be extended to other L0 and LI reference pictures of:refIdxLQ= 0, 1, but not to refIdxLQ-MC, andrefIdxL1= 0, 1, but not to refIdxL1-MC.

[0083] The extended merge candidate may hold or comprise:one L0 reference picture index refIdxLQ, andone projected motion vector mvL0-refIdxandone LI reference picture index refIdxL1, andone projected motion vector mvL1-refIdx.It may be denoted as:MC(^mvL0-refidx, refIdxL0), (mvL1-refidx, refIdxL1))where the reference picture index refIdxL0can be equal to 0, 1,.. but not to refIdxL0-MC.

[0084] The projected motion vector mvL0-refIdxon the reference picture with index refIdxL0may be calculated based upon the temporal distance between the current picture and the reference picture indexed by refIdxL0and the temporal distance between the current picture and the reference picture indexed by refIdxL0-MC, as: / POC (cur Pic) — POC(refIdxL0) \ mvL0-refldx= mvL0-MCx (^0C(-cltrPic) _ poc(refIdxL0-MC') )where• POC (cur Pic) is the POC number of the current picture,• POC(refIdxL0) is the POC number of the reference picture indexed by refIdxLQ, and • POC(refIdxL0-MC) is the POC number of the reference picture indexed by refIdxLQ-MC.

[0085] The reference picture index refIdxL1can be equal to 0, 1, but not to refIdxL1-MC. The projected motion vector mvL1-refIdxon the reference picture with index refIdxL1may be calculated based upon the temporal distance between the current picture andthe reference picture indexed by refIdxL1, and the temporal distance between the current picture and the reference picture indexed by refIdxL1-MC, as: / POC (cur Pic) — POC(refIdxL1) \ mvL1-refldx= mvL1-MCx \p0C(curPic) _ poC(refIdxL1-MC) )where• POC cur Pic) is the POC number of the current picture,• POC(refIdxL1) is the POC number of the reference picture indexed by refIdxL1, and • POC(refIdxL1-MC) is the POC number of the reference picture indexed by refIdxL1-MC.

[0086] FIG. 10 shows an example, where the L0 motion vector of a bidirectional merge candidateMC^(mvLQ-refidx,refIdxLQ), (jrtvL1-refidx,refIdxL1)^ is scaled down from L0 reference picture of POC(refIdxL0-MC) to L0 reference picture of POC(refIdxL0), and its LI motion vector is stretched from LI reference picture of POC(refIdxL1-MC) to LI reference picture of POC(refIdxL1).

[0087] With features as described herein, both the encoder and the decoder may be adapted to perform the same process for extending merge candidates to other reference pictures for a current block in a current picture. With features as described herein, both the encoder and the decoder may be adapted to construct the same extended merge candidate list, including both the original merge candidates and the extended merge candidates, for a current block in a current picture. With features as described herein, the encoder may be configured to select a merge candidate among both the original merge candidates and extended merge candidates for a current block in a current picture. With features as described herein, to sync with the decoder, the encoder may signal the decoder which merge candidate has been selected by the encoder for a current block in a current picture.

[0088] In one example embodiment, a unique merge index may be assigned for each (original or extended) merge candidate in the extended merge candidate list. For a selected (original or extended) merge candidate, the encoder may signal the assigned index to the decoder.

[0089] In one example embodiment, a unique merge index might only be assigned for each original merge candidate in the extended merge candidate list. For a selected (original or extended) merge candidate, the encoder may signal the decoder the assigned index of the original merge candidate from which the selected merge candidate is projected, and also signal the reference picture index of the reference picture where the original merge candidate is projected to.

[0090] Features as described herein may be provided with one or more of the following:1. For a merge candidate (MC) with only L0 motion, its L0 motion vector (mvL0-MC) may be projected to other reference pictures of reference picture list L0, and each of the other reference pictures (refIdxL0) of L0 and the associated projected motion vector (mvL0-refIdx) may form a new extended unidirectional merge candidate MC(mvL0-refIdx, refIdxL0).2. For a merge candidate (MC) with only LI motion, its LI motion vector (mvL1-MC) may be projected to other reference pictures of reference picture list LI, and each of the other reference pictures (refIdxL1) of LI and the associated projected motion vector mvL1-refIdx) may form a new extended unidirectional merge candidate MC(mvL1-refIdx, refIdxL1).3. For a merge candidate (MC) with both L0 motion and LI motion, its L0 motion vector (mvL0-MC) may be projected to other reference pictures of reference picture list L0 and its LI motion vector (mvL1-MC) may be projected to other reference pictures of reference picture list LI, and each of the other reference pictures (refIdxL0) of L0 and the associated projected motion vector (mvL0-refIdx) and each of the other reference pictures (refIdxL1) of LI and the associated projected motion vector (mvL1-refIdx) may form a new extended bidirectional merge candidate MC((mvL0-refIdx, refIdxL0), (mvL1-refIdx, refIdxL1)).refIdxL0-MCmay be used for reference to the L0 reference picture index of the original merge candidate, and refIdxL1-MCmay be used for reference to the LI reference picture index of the original merge candidate.

[0091] An example embodiment may be provided with an apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed with the at least one processor, cause the apparatus to perform: determining, for a coding unit in a current picture, a first merge candidate based on a first reference picture; determining, for the coding unit in the current picture, a second merge candidate based on a different second reference picture; and selecting, for use with a motion information prediction, one of the determined first merge candidate or the determined second merge candidate. The second reference picture is different from the first reference picture, and the second reference picture may be different from the first picture.

[0092] An example embodiment may be provided with an apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed with the at least one processor, cause the apparatus to perform: determining, for a coding unit in a current picture, a first merge candidate based on a first reference picture or a first pair of reference pictures; determining, for the coding unit in the current picture, a second merge candidate based on a second reference picture or a second pair of reference pictures; and selecting, for use with a motion information prediction, at least one of the determined first merge candidate or the determined second merge candidate.

[0093] The instructions, when executed with the at least one processor, may cause the apparatus to perform: determining, for the coding unit in the current picture, a third merge candidate based on the first pair of reference pictures, and determining, for the coding unit in the current picture, a fourth merge candidate based on the second pair of reference pictures; and selecting, for use with the motion information prediction, at least one of the determined third merge candidate or the determined fourth merge candidate. The first merge candidate may comprise one motion vector and one reference picture index. The first merge candidate may comprise more than one motion vector and more than one reference picture index. The instructions, when executed with the at least one processor, may cause the apparatus to perform: sending selected merge candidate information for a decoder, where the selected merge candidate information comprises information regarding the selected merge candidate(s). The instructions, when executed with the at least one processor, may cause the apparatus to perform: forming a first list of merge candidate(s), comprising the first merge candidate with motion information of at least one past coding unit; and forming a second list of merge candidate(s),comprising the second merge candidate with extended motion information. The extended motion information may comprise the motion vector for the extended case such as mvL0-refIdxfor example. The forming of the second list merge candidate(s) may exclude the first merge candidate from the second list of merge candidate(s). The instructions, when executed with the at least one processor, may cause the apparatus to perform: determining a motion vector based on a temporal distance between the current picture and the first reference picture, and a temporal distance between the current picture and the second reference picture. The motion ( POC(curPic)—POC(refIdxi0') \. vector mayJcomp1rise: x - - - - - - - - where:refIdxL0-MCmvL0-refIdxis the motion vector, mvL0-MCis a motion vector of a first merge candidate, POC(curPic) is the picture order count number of the current picture, POC(refIdxL0) is the picture order count number of the second reference picture, and POC(refIdxL0-MC) is the picture order count number of the first reference picture. The instructions, when executed with the at least one processor, may cause the apparatus to perform: determining the another motion vector based on a temporal distance between the current picture and a third reference picture, and a temporal distance between the current picture and a fourth reference picture. The determining of the another motion vector may comprise: mvL1-refIdx= mvL1-MC× (POC(curPic)−POC(refIdxL1)where: mvL1-refIdxis the another motion vector, mvL1-MCis POC(curPic)−POC(refIdxL1-MC)a motion vector of the second merge candidate, POC(curPic) is the picture order count number of the current picture, POC(refIdxL1) is the picture order count number of the third reference picture, and POC(refIdxL1-MC)' is the picture order count number of the fourth reference picture. The instructions, when executed with the at least one processor, may cause the apparatus to perform, based at least partially on the selecting, at least one of: scaling down a motion vector, or stretching a motion vector. The merge candidates may comprise more than three merge candidates. The instructions, when executed with the at least one processor, may cause the apparatus to perform: assigning an individual different merge index for the first merge candidate and the second merge candidate. The instructions, when executed with the at least one processor, may cause the apparatus to perform: sending merge index information for a decoder, where the sent merge index information comprises the assigned merge index(es).

[0094] Referring also to FIG. 11, an example embodiment may be provided with a method comprising: determining, for a coding unit in a current picture, a first merge candidate basedon a first reference picture or a first pair of reference pictures as illustrated with block 1102; determining, for the coding unit in the current picture, a second merge candidate based on a second reference picture or a second pair of reference pictures as illustrated with block 1104; and selecting, for use with a motion information prediction, at least one of the determined first merge candidate or the determined second merge candidate as illustrated with block 1106. The method may comprise determining, for the coding unit in the current picture, a third merge candidate based on the first pair of reference pictures, and determining, for the coding unit in the current picture, a fourth merge candidate based on the second pair of reference pictures; and selecting, for use with the motion information prediction, at least one of the determined third merge candidate or the determined fourth merge candidate. The first merge candidate may comprise one motion vector and one reference picture index. The first merge candidate may comprise more than one motion vector and more than one reference picture index. The method may comprise sending selected merge candidate information for a decoder, where the selected merge candidate information comprises information regarding the selected merge candidate(s). The method may comprise forming a first list of merge candidate(s), comprising the first merge candidate with motion information of at least one past coding unit; and forming a second list of merge candidate(s), comprising the second merge candidate with extended motion information. The second list merge candidate(s) may exclude the first merge candidate from the second list of merge candidate(s). The method may comprise determining a motion vector based on: a temporal distance between the current picture and the first reference picture, and a temporal distance between the current picture and the second reference picture. The motion vector may comprise: mvL0-refIdx= mvL0-MC×POC(curPic)−POC(refIdxL0-MC) where: mvL0-refIdxis the motion vector, mvL0-MCis a motion vector of a first merge candidate, POC(curPic) is the picture order count number of the current picture, POC(refIdxL0) is the picture order count number of the second reference picture, and POC(refIdxL0-MC) is the picture order count number of the first reference picture. The method may comprise determining the another motion vector based on a temporal distance between the current picture and a third reference picture, and a temporal distance between the current picture and a fourth reference picture. The determining of the another motion vector may comprise: mvL1-refIdx= mvL1-MC×POC(curPic)−POC(refIdxL1-MC) where: mvL1-refIdxis the another motion vector, mvL1-MCis a motion vector of the second mergecandidate, POC(curPic) is the picture order count number of the current picture, POC(refIdxL) is the picture order count number of the third reference picture, and POC(refIdxL1-MC) is the picture order count number of the fourth reference picture. The method may comprise, based at least partially on the selecting, at least one of: scaling down a motion vector, or stretching a motion vector. The merge candidates may comprise more than three merge candidates. The method may comprise assigning an individual different merge index for the first merge candidate and the second merge candidate. The method may comprise sending merge index information for a decoder, where the sent merge index information comprises the assigned merge index(es).

[0095] An example embodiment may be provided with an apparatus comprising: means for determining, for a coding unit in a current picture, a first merge candidate based on a first reference picture or a first pair of reference pictures; means for determining, for the coding unit in the current picture, a second merge candidate based on a second reference picture or a second pair of reference pictures; and means for selecting, for use with a motion information prediction, at least one of the determined first merge candidate or the determined second merge candidate.

[0096] An example embodiment may be provided with an apparatus comprising a non-transitory computer readable medium comprising program instructions that, when executed by an apparatus, cause the apparatus to perform at least the following: determining, for a coding unit in a current picture, a first merge candidate based on a first reference picture or a first pair of reference pictures; determining, for the coding unit in the current picture, a second merge candidate based on a second reference picture or a second pair of reference pictures; and selecting, for use with a motion information prediction, at least one of the determined first merge candidate or the determined second merge candidate.

[0097] An example embodiment may be provided with an apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed with the at least one processor, cause the apparatus to perform: determining, for a coding unit in a current picture, a first merge candidate based on a first reference picture or a first pair of reference pictures; determining, for the coding unit in the current picture, a second merge candidate based on a second reference picture or a second pair of reference pictures; and receiving informationfor selecting, for use with a motion information prediction, at least one of the determined first merge candidate or the determined second merge candidate.

[0098] The instructions, when executed with the at least one processor, may cause the apparatus to perform: determining, for the coding unit in the current picture, a third merge candidate based on the first pair of reference pictures, and determining, for the coding unit in the current picture, a fourth merge candidate based on the second pair of reference pictures; and selecting, for use with the motion information prediction, at least one of the determined third merge candidate or the determined fourth merge candidate. The first merge candidate may comprise one motion vector and one reference picture index. The first merge candidate may comprise more than one motion vector and more than one reference picture index. The instructions, when executed with the at least one processor, may cause the apparatus to perform: receiving selected merge candidate information from an encoder, where the selected merge candidate information comprises information regarding the selected merge candidate(s). The instructions, when executed with the at least one processor, may cause the apparatus to perform: forming a first list of merge candidate(s), comprising the first merge candidate, with motion information of at least one past coding unit; and forming a second list of merge candidate(s), comprising the second merge candidate, with extended motion information. The forming of the second list merge candidate(s) may exclude the first merge candidate from the second list of merge candidate(s). The instructions, when executed with the at least one processor, may cause the apparatus to perform: determining a motion vector based on: a temporal distance between the current picture and the first reference picture, and a temporal distance between the current picture and the second reference picture. The motion vector may comprise:mvL0-refIdx= mvL0-MC× (POC(curPic)−POC(refIdxL0)) / (POC(curPic)−POC(refIdxL0-MC)) where: mvL0-refIdxPOC(curPic)−POC(refIdxL0-MC) where: mvL0-refIdxis the motion vector, mvL0-MCis a motion vector of a first merge candidate, POC(curPic) is the picture order count number of the current picture, POC(refIdxL0) is the picture order count number of the second reference picture, and POC(refIdxL0-MC) is the picture order count number of the first reference picture. The instructions, when executed with the at least one processor, may cause the apparatus to perform: determining the another motion vector based on a temporal distance between the current picture and a third reference picture, and a temporal distance between the current picture and a fourth reference picture. The determining of the secondmotion vector may comprise: mvL1-refIdx= mvL1-MC× (POC(curPic)−POC(refIdxL1)) / (POC(curPic)−POC(refIdxL1-MC)) where: mvL1-refIdxis the another motion vector, mvL1-MCis a motion vector of the second merge candidate, POC(curPic) is the picture order count number of the current picture, POCe is the picture order count number of the third reference picture, and POC(refIdxL1-MC) is the picture order count number of the fourth reference picture. The instructions, when executed with the at least one processor, may cause the apparatus to perform, based at least partially on the selecting, at least one of: scaling down a motion vector, or stretching a motion vector. The merge candidates may comprise more than three merge candidates. The instructions, when executed with the at least one processor, may cause the apparatus to perform: receiving an individual different merge index assigned for the first merge candidate and the second merge candidate.

[0099] Referring also to FIG. 12, an example embodiment may be provided with a method comprising: determining, for a coding unit in a current picture, a first merge candidate based on a first reference picture or a first pair of reference pictures as illustrated with block 1202; determining, for the coding unit in the current picture, a second merge candidate based on a second reference picture or a second pair of reference pictures as illustrated with block 12004; and receiving information for selecting, for use with a motion information prediction, at least one of the determined first merge candidate or the determined second merge candidate as illustrated with block 1206. The method may comprise determining, for the coding unit in the current picture, a third merge candidate based on the first pair of reference pictures, and determining, for the coding unit in the current picture, a fourth merge candidate based on the second pair of reference pictures; and selecting, for use with the motion information prediction, at least one of the determined third merge candidate or the determined fourth merge candidate. The first merge candidate may comprise one motion vector and one reference picture index. The first merge candidate may comprise more than one motion vector and more than one reference picture index. The method may comprise receiving selected merge candidate information from an encoder, where the selected merge candidate information comprises information regarding the selected merge candidate(s). The method may comprise forming a first list of merge candidate(s), comprising the first merge candidate, with motion information of at least one past coding unit; and forming a second list of merge candidate(s), comprising the second merge candidate, with extended motion information. The forming of the second listmerge candidate(s) may exclude the first merge candidate from the second list of merge candidate(s). The method may comprise determining a motion vector based on a temporal distance between the current picture and the first reference picture, and a temporal distance between the current picture and the second reference picture. The motion vector may comprise:mvL0-refIdx= mvL0-MC× (POC(curPic)−POC(refIdxL0)) / (POC(curPic)−POC(refIdxL0-MC)) where: mvL0-refIdxis the motionvector, mvL0-MCis a motion vector of a first merge candidate, POC(curPic) is the picture order count number of the current picture, POC(refIdxL0) is the picture order count number of the second reference picture, and POC(refIdxL0-MC) is the picture order count number of the first reference picture. The method may comprise determining the another motion vector based on a temporal distance between the current picture and a third reference picture, and a temporal distance between the current picture and a fourth reference picture. The determining of the second motion vector may comprise: mvL1-refIdx= mvL1-MC× (POC(curPic)−POC(refIdxL1))where: mvL1-refIdxis the another motion vector, mvL1-MCisPOC(curPic)−POC(refIdxL1-MC)a motion vector of the second merge candidate, POC(curPic) is the picture order count number of the current picture, POC(refIdxL1) is the picture order count number of the third reference picture, and POC(refIdxL1-MC)' is the picture order count number of the fourth reference picture. The method may comprise, based at least partially on the selecting, at least one of: scaling down a motion vector, or stretching a motion vector. The merge candidates may comprise more than three merge candidates. The method may comprise receiving an individual different merge index assigned for the first merge candidate and the second merge candidate.

[0100] An example embodiment may be provided with an apparatus comprising: means for determining, for a coding unit in a current picture, a first merge candidate based on a first reference picture or a first pair of reference pictures; means for determining, for the coding unit in the current picture, a second merge candidate based on a second reference picture or a second pair of reference pictures; and means for receiving information for selecting, for use with a motion information prediction, at least one of the determined first merge candidate or the determined second merge candidate.

[0101] An example embodiment may be provided with an apparatus comprising a non- transitory computer readable medium comprising program instructions that, when executed by an apparatus, cause the apparatus to perform at least the following: determining, for a codingunit in a current picture, a first merge candidate based on a first reference picture or a first pair of reference pictures; determining, for the coding unit in the current picture, a second merge candidate based on a second reference picture or a second pair of reference pictures; and receiving information for selecting, for use with a motion information prediction, at least one of the determined first merge candidate or the determined second merge candidate.

[0102] The term “non-transitory,” as used herein, is a limitation of the medium itself (i.e., tangible, not a signal) as opposed to a limitation on data storage persistency (e.g., RAM vs. ROM).

[0103] As used in this application, the term “circuitry” may refer to one or more or all of the following:(a) hardware-only circuit implementations (such as implementations in only analog and / or digital circuitry) and(b) combinations of hardware circuits and software, such as (as applicable):(i) a combination of analog and / or digital hardware circuit(s) with software / firmware and(ii) any portions of hardware processor(s) with software (including digital signal processor(s)), software, and memory(ies) that work together to cause an apparatus, such as a mobile phone or server, to perform various functions) and(iii) hardware circuit(s) and or processor(s), such as a microprocessor(s) or a portion of a microprocessor(s), that requires software (e.g., firmware) for operation, but the software may not be present when it is not needed for operation.”

[0104] This definition of circuitry applies to all uses of this term in this application, including in any claims. As a further example, as used in this application, the term circuitry also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portion of a hardware circuit or processor and its (or their) accompanying software and / or firmware. The term circuitry also covers, for example, and if applicable to the particular claim element,a baseband integrated circuit or processor integrated circuit for a mobile device or a similar integrated circuit in server, a cellular network device, or other computing or network device.

[0105] The following abbreviations that may be found in the specification and / or the drawing figures are defined as follows:ID one dimension(al)2D two dimension(al)AM VP advanced MVPBiMVD bidirectional MVDCCCM convolutional cross-component modelCTU coding tree unitCU coding unitcurPic current pictureDC a mode where all samples in a prediction block have the same value ECM enhanced compression modeHMVP history-based motion vector predictionIBC intra block copyI / F interfaceJVET joint video experts teamMC merge candidateMIP matrix weighted intra predictionMV motion vectorMVD motion vector differencemv motion vectorMVP motion vector predictionN / W or NW networkPDPC position dependent intra prediction combinationPOC picture order countRD rate-distortionrefldx reference picture indexSMVD symmetric MVDTMVP temporal motion vector predictorVVC versatile video coding

[0106] It should be understood that the foregoing description is only illustrative. Various alternatives and modifications can be devised by those skilled in the art. For example, features recited in the various dependent claims could be combined with each other in any suitable combination(s). In addition, features from different embodiments described above could be selectively combined into a new embodiment. Accordingly, the description is intended to embrace all such alternatives, modifications and variances which fall within the scope of the appended claims.

Claims

ClaimsWhat is claimed is:

1. An apparatus comprising:at least one processor; andat least one memory storing instructions that, when executed with the at least one processor, cause the apparatus to perform:determining, for a coding unit in a current picture, a first merge candidate based on a first reference picture or a first pair of reference pictures;determining, for the coding unit in the current picture, a second merge candidate based on a second reference picture or a second pair of reference pictures; andselecting, for use with a motion information prediction, at least one of the determined first merge candidate or the determined second merge candidate.

2. The apparatus as claimed in claim 1 where the instructions, when executed with the at least one processor, cause the apparatus to perform:determining, for the coding unit in the current picture, a third merge candidate based on the first pair of reference pictures, anddetermining, for the coding unit in the current picture, a fourth merge candidate based on the second pair of reference pictures; andselecting, for use with the motion information prediction, at least one of the determined third merge candidate or the determined fourth merge candidate.

3. The apparatus as claimed in claim 1 where the first merge candidate comprisesone motion vector and one reference picture index.

4. The apparatus as claimed in claim 1 where the first merge candidate comprises more than one motion vector and more than one reference picture index.

5. The apparatus as claimed in any one of claims 1-4 where the instructions, when executed with the at least one processor, cause the apparatus to perform:sending selected merge candidate information for a decoder, where the selected merge candidate information comprises information regarding the selected merge candidate(s).

6. The apparatus as claimed in any one of claims 1-5 where the instructions, when executed with the at least one processor, cause the apparatus to perform:forming a first list of merge candidate(s), comprising the first merge candidate with motion information of at least one past coding unit; andforming a second list of merge candidate(s), comprising the second merge candidate with extended motion information.

7. The apparatus as claimed in claim 6 where the forming of the second list merge candidate(s) excludes the first merge candidate from the second list of merge candidate(s).

8. The apparatus as claimed in any one of claims 1-7 where the instructions, when executed with the at least one processor, cause the apparatus to perform:determining a motion vector based on:a temporal distance between the current picture and the first reference picture, anda temporal distance between the current picture and the second reference picture.

9. The apparatus as claimed in claim 8 where the motion vector comprises: / POC(curPic) — POC(refIdxL0) \ mvL0-refldx= mvL0-MCx (^0C(curPic) _ poc(refIdxL0-MC') )where:mvLo-refidx is the motion vector,mvLo-Mc isamotion vector of the first merge candidate,POC(curPic) is the picture order count number of the current picture,POC(refIdxL0) is the picture order count number of the second reference picture, andPOC(refIdxL0-MC) is the picture order count number of the first reference picture.

10. The apparatus as claimed in any one of claims 8-9, where the instructions, when executed with the at least one processor, cause the apparatus to perform:determining the another motion vector based on:a temporal distance between the current picture and a third reference picture, anda temporal distance between the current picture and a fourth reference picture.

11. The apparatus as claimed in claim 10 where the determining of the another motion vector comprises: / POC(curPic) — POC(refIdxL1) \ mvL1-refldx= mvL1-MCx \p0C(curPic) _ poc(refIdxL1-MC') )where:mvLi-refidx is the another motion vector,mvLi-Mc isamotion vector of a further merge candidate,POC(curPic) is the picture order count number of the current picture,POC(refIdxL1) is the picture order count number of the third reference picture, andPOC(refIdxL1-MC)' is the picture order count number of the fourth reference picture.

12. The apparatus as claimed in any one of claims 1-11 where the instructions, when executed with the at least one processor, cause the apparatus to perform, based at least partially on the selecting, at least one of:scaling down a motion vector, orstretching a motion vector.

13. The apparatus as claimed in any one of claims 1-12 where the merge candidates comprise more than three merge candidates.

14. The apparatus as claimed in any one of claims 1-13 where the instructions, when executed with the at least one processor, cause the apparatus to perform:assigning an individual different merge index for the first merge candidate and the second merge candidate.

15. The apparatus as claimed in claim 14 where the instructions, when executed with the at least one processor, cause the apparatus to perform:sending merge index information for a decoder, where the sent merge index information comprises the assigned merge index(es).

16. A method comprising:determining, for a coding unit in a current picture, a first merge candidate based on a first reference picture or a first pair of reference pictures;determining, for the coding unit in the current picture, a second merge candidate based on a second reference picture or a second pair of reference pictures; andselecting, for use with a motion information prediction, at least one of the determined first merge candidate or the determined second merge candidate.

17. The method as claimed in claim 16 comprising:determining, for the coding unit in the current picture, a third merge candidate based on the first pair of reference pictures, anddetermining, for the coding unit in the current picture, a fourth merge candidate based on the second pair of reference pictures; andselecting, for use with the motion information prediction, at least one of the determined third merge candidate or the determined fourth merge candidate.

18. The method as claimed in claim 16 where the first merge candidate comprisesone motion vector and one reference picture index.

19. The method as claimed in claim 16 where the first merge candidate comprises more than one motion vector and more than one reference picture index.

20. The method as claimed in any one of claims 1-19 comprising:sending selected merge candidate information for a decoder, where the selected merge candidate information comprises information regarding the selected merge candidate(s).

21. The method as claimed in any one of claims 1-20 comprising:forming a first list of merge candidate(s), comprising the first merge candidate with motion information of at least one past coding unit; andforming a second list of merge candidate(s), comprising the second merge candidate with extended motion information.

22. The method as claimed in claim 21 where the forming of the second list merge candidate(s) excludes the first merge candidate from the second list of merge candidate(s).

23. The method as claimed in any one of claims 1-22 comprising:determining a motion vector based on:a temporal distance between the current picture and the first reference picture, anda temporal distance between the current picture and the second reference picture.

24. The method as claimed in claim 23 where the motion vector comprises: / POC(curPic) — POC(refIdxL0) \ mvL0-refldx= mvL0-MCx (^0C(curPic) _ poc(refIdxL0-MC') )where:mvLo-refidx is the motion vector,mvLo-Mc isamotion vector of a first merge candidate,POC(curPic) is the picture order count number of the current picture,POC(refIdxL0) is the picture order count number of the second reference picture, andPOC(refIdxL0-MC) is the picture order count number of the first reference picture.

25. The method as claimed in any one of claims 23-24, comprising:determining the another motion vector based on:a temporal distance between the current picture and a third reference picture, anda temporal distance between the current picture and a fourth reference picture.

26. The method as claimed in claim 25 where the determining of the another motion vector comprises: / POC(curPic) — POC(refIdxL}) \ mvL1-refldx= mvL1-MCx \p0C(curPic) _ poc(refIdxL1-MC') )where:mvLi-refidx is the another motion vector,mvLi-Mc isamotion vector of the second merge candidate,POC(curPic) is the picture order count number of the current picture,POC(refIdxL1) is the picture order count number of the third reference picture, andPOC(refIdxL1-MC)' is the picture order count number of the fourth reference picture.

27. The method as claimed in any one of claims 16-26 comprising, based at least partially on the selecting, at least one of:scaling down a motion vector, orstretching a motion vector.

28. The method as claimed in any one of claims 16-27 where the merge candidates comprise more than three merge candidates.

29. The method as claimed in any one of claims 16-28 comprising:assigning an individual different merge index for the first merge candidate and the second merge candidate.

30. The method as claimed in claim 29 comprising:sending merge index information for a decoder, where the sent merge index information comprises the assigned merge index(es).

31. An apparatus comprising:means for determining, for a coding unit in a current picture, a first merge candidate based on a first reference picture or a first pair of reference pictures;means for determining, for the coding unit in the current picture, a second merge candidate based on a second reference picture or a second pair of reference pictures; andmeans for selecting, for use with a motion information prediction, at least one of the determined first merge candidate or the determined second merge candidate.

32. A non-transitory computer readable medium comprising program instructions that, when executed by an apparatus, cause the apparatus to perform at least the following:determining, for a coding unit in a current picture, a first merge candidate based on a first reference picture or a first pair of reference pictures;determining, for the coding unit in the current picture, a second merge candidate based on a second reference picture or a second pair of reference pictures; andselecting, for use with a motion information prediction, at least one of the determined first merge candidate or the determined second merge candidate.

33. An apparatus comprising:at least one processor; andat least one memory storing instructions that, when executed with the at least one processor, cause the apparatus to perform:determining, for a coding unit in a current picture, a first merge candidate based on a first reference picture or a first pair of reference pictures;determining, for the coding unit in the current picture, a second merge candidate based on a second reference picture or a second pair of reference pictures; andreceiving information for selecting, for use with a motion information prediction, at least one of the determined first merge candidate or the determined second merge candidate.

34. The apparatus as claimed in claim 33 where the instructions, when executed with the at least one processor, cause the apparatus to perform:determining, for the coding unit in the current picture, a third merge candidate based on the first pair of reference pictures, anddetermining, for the coding unit in the current picture, a fourth merge candidate based on the second pair of reference pictures; andselecting, for use with the motion information prediction, at least one of the determined third merge candidate or the determined fourth merge candidate.

35. The apparatus as claimed in claim 33 where the first merge candidate comprises one motion vector and one reference picture index.

36. The apparatus as claimed in claim 33 where the first merge candidate comprises more than one motion vector and more than one reference picture index.

37. The apparatus as claimed in any one of claims 33-36 where the instructions, when executed with the at least one processor, cause the apparatus to perform:receiving selected merge candidate information from an encoder, where the selected merge candidate information comprises information regarding the selected merge candidate(s).

38. The apparatus as claimed in any one of claims 33-37 where the instructions, when executed with the at least one processor, cause the apparatus to perform:forming a first list of merge candidate(s), comprising the first merge candidate, with motion information of at least one past coding unit; andforming a second list of merge candidate(s), comprising the second merge candidate, with extended motion information.

39. The apparatus as claimed in claim 38 where the forming of the second list merge candidate(s) excludes the first merge candidate from the second list of merge candidate(s).

40. The apparatus as claimed in any one of claims 33-39 where the instructions, when executed with the at least one processor, cause the apparatus to perform:determining a motion vector based on:a temporal distance between the current picture and the first reference picture, anda temporal distance between the current picture and the second reference picture.

41. The apparatus as claimed in claim 40 where the motion vector comprises: / POC(curPic) — POC(refIdxL0) \ mvL0-refldx= mvL0-MCx (^0C(curPic) _ poc(refIdxL0-MC') )where:mvLo-refidx is the motion vector,mvLo-Mc isamotion vector of a first merge candidate,POC(curPic) is the picture order count number of the current picture,POC(refIdxL0) is the picture order count number of the second reference picture, andPOC(refIdxL0-MC) is the picture order count number of the first reference picture.

42. The apparatus as claimed in claim 41, where the instructions, when executed with the at least one processor, cause the apparatus to perform:determining the another motion vector based on:a temporal distance between the current picture and a third reference picture, anda temporal distance between the current picture and a fourth reference picture.

43. The apparatus as claimed in claim 42 where the determining of the second motion vector comprises: / POC(curPic) — POC(refIdxL1) \ mvL1-refldx= mvL1-MCx \p0C(curPic) _ poc(refIdxL1-MC') )where:mvLi-refidx is the another motion vector,mvLi-Mc isamotion vector of a further merge candidate,POC(curPic) is the picture order count number of the current picture,POC(refIdxL1) is the picture order count number of the third reference picture, andPOC(refIdxL1-MC)' is the picture order count number of the fourth reference picture.

44. The apparatus as claimed in any one of claims 33-43 where the instructions, when executed with the at least one processor, cause the apparatus to perform, based at least partially on the selecting, at least one of:scaling down a motion vector, orstretching a motion vector.

45. The apparatus as claimed in any one of claims 33-44 where the merge candidates comprise more than three merge candidates.

46. The apparatus as claimed in any one of claims 33-45 where the instructions, when executed with the at least one processor, cause the apparatus to perform:receiving an individual different merge index assigned for the first merge candidate and the second merge candidate.

47. A method comprising:determining, for a coding unit in a current picture, a first merge candidate based on a first reference picture or a first pair of reference pictures;determining, for the coding unit in the current picture, a second merge candidate based on a second reference picture or a second pair of reference pictures; andreceiving information for selecting, for use with a motion information prediction, at least one of the determined first merge candidate or the determined second merge candidate.

48. The method as claimed in claim 47 comprising:determining, for the coding unit in the current picture, a third merge candidate based on the first pair of reference pictures, anddetermining, for the coding unit in the current picture, a fourth merge candidate based on the second pair of reference pictures; andselecting, for use with the motion information prediction, at least one of the determined third merge candidate or the determined fourth merge candidate.

49. The method as claimed in claim 47 where the first merge candidate comprises one motion vector and one reference picture index.

50. The method as claimed in claim 47 where the first merge candidate comprises more than one motion vector and more than one reference picture index.

51. The method as claimed in any one of claims 47-50 comprising:receiving selected merge candidate information from an encoder, where the selected merge candidate information comprises information regarding the selected merge candidate(s).

52. The method as claimed in any one of claims 47-51 comprising:forming a first list of merge candidate(s), comprising the first merge candidate, with motion information of at least one past coding unit; andforming a second list of merge candidate(s), comprising the second merge candidate, with extended motion information.

53. The method as claimed in claim 52 where the forming of the second list merge candidate(s) excludes the first merge candidate from the second list of merge candidate(s).

54. The method as claimed in any one of claims 47-53 comprising:determining a motion vector based on:a temporal distance between the current picture and the first reference picture, anda temporal distance between the current picture and the second reference picture.

55. The method as claimed in claim 54 where the motion vector comprises: / POC(curPic) — POC(refIdxL0) \ mvL0-refldx= mvL0-MCx (^0C(curPic) _ poc(refIdxL0-MC') )where:mvLo-refidx is the motion vector,mvLo-Mc isamotion vector of a first merge candidate,POC(curPic) is the picture order count number of the current picture,POC(refIdxL0) is the picture order count number of the second reference picture, andPOC(refIdxL0-MC) is the picture order count number of the first reference picture.

56. The method as claimed in claim 55, comprising:determining the another motion vector based on:a temporal distance between the current picture and a third reference picture, anda temporal distance between the current picture and a fourth reference picture.

57. The method as claimed in claim 56 where the determining of the second motion vector comprises: / POC(curPic) — POC(refIdxL}) \ mvL1-refldx= mvL1-MCx \p0C(curPic) _ poc(refIdxL1-MC') )where:mvLi-refidx is the another motion vector,mvLi-Mc isamotion vector of the second merge candidate,POC(curPic) is the picture order count number of the current picture,POC(refIdxL1) is the picture order count number of the third reference picture, andPOC(refIdxL1-MC) is the picture order count number of the fourth reference picture.

58. The method as claimed in any one of claims 47-57 comprising, based at least partially on the selecting, at least one of:scaling down a motion vector, orstretching a motion vector.

59. The method as claimed in any one of claims 47-58 where the merge candidates comprise more than three merge candidates.

60. The method as claimed in any one of claims 47-59 comprising:receiving an individual different merge index assigned for the first merge candidate and the second merge candidate.

61. An apparatus comprising:means for determining, for a coding unit in a current picture, a first merge candidate based on a first reference picture or a first pair of reference pictures;means for determining, for the coding unit in the current picture, a second merge candidate based on a second reference picture or a second pair of reference pictures; andmeans for receiving information for selecting, for use with a motion information prediction, at least one of the determined first merge candidate or the determined second merge candidate.

62. A non-transitory computer readable medium comprising program instructions that, when executed by an apparatus, cause the apparatus to perform at least the following:determining, for a coding unit in a current picture, a first merge candidate based on a first reference picture or a first pair of reference pictures;determining, for the coding unit in the current picture, a second merge candidate based on a second reference picture or a second pair of reference pictures; andreceiving information for selecting, for use with a motion information prediction, at least one of the determined first merge candidate or the determined second merge candidate.

Images