Data-driven intra-inter-prediction

Abstract

A data-driven intra-inter-prediction mode associated with a matrix is identified. An inter-prediction for a current block is generated. An input vector of features is extracted from source regions associated with the current block, including at least a first source region from the inter-prediction and at least a second source region from a neighboring block spatially adjacent to the current block. The input vector is multiplied by the matrix to obtain an output vector. The output vector is converted into an output matrix. The output matrix is resized to match dimensions of the current block to generate a prediction block for the current block.

Description

Atty. Doc. No. GOGL-2288-A-WODATA-DRIVEN INTRA-INTER-PREDICTIONCROSS-REFERENCE TO RELATED APPLICATION(S)

[0001] This disclosure claims the benefit of U.S. Provisional Patent Application No. 63 / 762,377 filed February 24, 2025, the disclosure of which is incorporated by reference herein in its entirety.BACKGROUND

[0002] Digital video streams may represent video using a sequence of frames or still images. Digital video can be used for various applications including, for example, video conferencing, high definition video entertainment, video advertisements, or sharing of usergenerated videos. A digital video stream can contain a large amount of data and consume a significant amount of computing or communication resources of a computing device for processing, transmission, or storage of the video data. Various approaches have been proposed to reduce the amount of data in video streams, including encoding or decoding techniques.SUMMARY

[0003] This disclosure relates generally to encoding and decoding video data and more particularly relates to data-driven intra-inter-prediction.

[0004] One aspect of the disclosure is a method for identifying a data-driven intra-inter-prediction mode associated with a prediction matrix, generating an inter-prediction for a current block, extracting an input vector of features from source regions associated with the current block, the source regions including at least a first source region from the interprediction and at least a second source region from a neighboring block spatially adjacent to the current block, multiplying the input vector by the prediction matrix to obtain an output vector, converting the output vector into an output matrix, and resizing the output matrix to match dimensions of the current block to generate a prediction block for the current block.

[0005] In an aspect, identifying the data-driven intra-inter-prediction mode comprises decoding the data-driven intra-inter-prediction mode from a compressed bitstream.

[0006] In an aspect, identifying the data-driven intra-inter-prediction mode comprises selecting the data-driven intra-inter-prediction mode based on rate-distortion optimization.

[0007] In an aspect, the prediction matrix is one of a plurality of pre-trained matrices, each prediction matrix corresponding to at least one respective data-driven intra-interprediction mode.

[0008] In an aspect, the plurality of pre-trained prediction matrices comprises six prediction matrices generated using machine learning techniques.

[0009] In an aspect, the method further comprises determining whether the dimensions of the current block are greater than or equal to a threshold size; and performing the extracting, multiplying, converting, and resizing in response to determining that the dimensions are greater than or equal to the threshold size.

[0010] In an aspect, the threshold size is 128 samples.

[0011] In an aspect, extracting the input vector comprises determining a feature for each respective source region associated with the current block.

[0012] In an aspect, the source regions include source regions above the current block, source regions left of the current block, and source regions within an inter-prediction of the current block.

[0013] In an aspect, the data-driven intra-inter-prediction mode has a pre- determined number of source regions each encompassing a respective pre- determined region of pixels that are proportionate to a size of the current block and either adjacent to the current block or within the inter-prediction block generated for the current block.

[0014] In an aspect, at least one of the features in the input vector is obtained as an average of samples in a source region.

[0015] In an aspect, converting the output vector into the output matrix comprises reshaping the output vector into the output matrix.

[0016] In an aspect, resizing the output matrix comprises performing dimension-by-dimension processing of the output matrix to match the dimensions of the current block.

[0017] In an aspect, performing the dimension-by-dimension processing comprises applying up sampling when a target dimension is greater than a corresponding output matrix dimension, applying down sampling when the target dimension is less than the corresponding output matrix dimension, and maintaining a dimension of the output matrix when the target dimension equals the corresponding output matrix dimension.

[0018] In an aspect, the method includes encoding the data-driven intra-interprediction mode in a compressed bitstream.

[0019] In an aspect, encoding the data-driven intra-inter-prediction mode in the compressed bitstream comprises encoding a flag indicating whether data-driven intra-inter-prediction is used for the current block, and encoding a syntax element identifying which of a plurality of available data-driven intra-inter-prediction modes is used.

[0020] In an aspect, the prediction matrix has dimensions PxQ or QxP, where P corresponds to a number of input features and Q is a perfect square.

[0021] In an aspect, P is 12 and Q is 64.

[0022] In an aspect, generating an inter-prediction for the current block includes utilizing at least one of bi-prediction or an implicit inter prediction technique.

[0023] One aspect of the disclosure is a method for predicting a current block in a video frame. The method includes extracting a first plurality of features based on pixels in first source regions adjacent to the current block, extracting a second plurality of features based on pixels in second source regions in an inter-prediction of the current block, generating an intermediate prediction block for the current block by applying a respective set of weights to the first and second plurality of features at intermediate prediction locations in the intermediate prediction block, and generating a data-driven prediction block for the current block by resampling the intermediate prediction block.

[0024] In an aspect, the current block is a matrix of pixels, the intermediate prediction block is a matrix of intermediate predictions, a height or width of the matrix of pixels is greater than a height or width of the intermediate prediction block, and resampling the intermediate prediction block includes up sampling the intermediate prediction block.

[0025] In an aspect, the respective set of weights and locations of the first source regions and second source regions are defined according to a data-driven intra-interprediction mode.

[0026] In an aspect, the method further comprises encoding an identification of the intra-inter-prediction mode into an encoded bitstream.

[0027] In an aspect, the method further comprises decoding an identification of the intra-inter-prediction mode from an encoded bitstream.

[0028] In an aspect, generating the intermediate prediction block includes performing a matrix multiplication operation between a first matrix including the first plurality of features and the second plurality of features and a second matrix including respective sets of weights for intermediate prediction locations corresponding to the intermediate prediction block.

[0029] In an aspect, the respective set of weights is determined by a training process performed on a training set of videos.

[0030] In an aspect, the respective set of weights is pre- determined and made available to encoder and decoder prior to encoding or decoding.

[0031] An aspect is a device including a processor that is configured to perform one of the foregoing methods. An aspect is a device including a memory and a processor, the processor configured to execute instructions stored in the memory to perform one of the foregoing methods.

[0032] An aspect is a video decoding apparatus according to one of the foregoing methods. An aspect is a video encoding apparatus according to one of the foregoing methods.

[0033] An aspect is a computer readable medium comprising instructions which, when the program is executed by a computer, cause the computer to carry out steps according to one of the foregoing methods. An aspect is a computer program product comprising instructions which, when the program is executed by a computer, cause the computer to carry out steps according to one of the foregoing methods.

[0034] An aspect is a non-transitory computer readable medium including an encoded bitstream that was encoded using any of the foregoing methods or is decodable using any of the foregoing methods. An aspect is a computer media product including an encoded bitstream encoded using any of the foregoing methods or that is decodable using any of the foregoing methods.

[0035] These and other aspects of the present disclosure are disclosed in the following detailed description of the embodiments, the appended claims and the accompanying figures.

[0036] It will be appreciated that aspects can be implemented in any convenient form. For example, aspects may be implemented by appropriate computer programs which may be carried on appropriate carrier media which may be tangible carrier media (e.g. disks) or intangible carrier media (e.g. communications signals). Aspects may also be implemented using suitable apparatus which may take the form of programmable computers running computer programs arranged to implement the methods and / or techniques disclosed herein. Aspects can be combined such that features described in the context of one aspect may be implemented in another aspect.BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The description herein makes reference to the accompanying drawings described below, wherein like reference numerals refer to like parts throughout the several views.

[0038] FIG. 1 is a schematic of a video encoding and decoding system.

[0039] FIG. 2 is a block diagram of an example of a computing device that can implement a transmitting station or a receiving station.

[0040] FIG. 3 is a diagram of a typical video stream to be encoded and subsequently decoded.

[0041] FIG. 4 is a block diagram of an encoder according to implementations of this disclosure.

[0042] FIG. 5 is a block diagram of a decoder according to implementations of this disclosure.

[0043] FIG. 6 is a flowchart of a technique for generating a prediction block for a current block using a data-driven intra-inter-prediction.

[0044] FIGS. 7A-D illustrates examples of sample areas useable for extracting a feature vector.

[0045] FIG. 8 illustrates an example of signaling of data-driven intra-inter-prediction modes.

[0046] FIG. 9 is a flowchart of a technique for generating a prediction block for a current block using a data-driven intra-inter-prediction.

[0047] FIG. 10 is a flowchart of another technique for generating a prediction block for a current block using a data-driven intra-inter-prediction.DETAILED DESCRIPTION

[0048] As mentioned above, compression schemes related to coding video streams may include breaking images into blocks and generating a digital video output bitstream (i.e., an encoded bitstream) using one or more techniques to limit the information included in the output bitstream. A received bitstream can be decoded to re-create the blocks and the source images from the limited information. Encoding a video stream, or a portion thereof, such as a frame or a block, can include using temporal or spatial similarities in the video stream to improve coding efficiency. For example, a current block of a video stream may be encoded based on identifying a difference (residual) between the previously coded pixel values, or between a combination of previously coded pixel values, and those in the current block.

[0049] Intra prediction involves predicting the pixel values of a block of a frame of a video stream using pixels spatially peripheral to the block; that is, using pixels that are in the same frame as the block but that are outside the block. A prediction block resulting from intra prediction is referred to herein as an intra predictor. Intra prediction can be performed along a direction of prediction where a direction can correspond to an intra prediction mode. Non-directional intra prediction modes are also possible. The intra prediction mode can be signaled by an encoder to a decoder.

[0050] Inter prediction involves predicting the pixel values of a block of a frame of a video stream using pixels temporally peripheral to the block. For example, an encoder may identify one or more corresponding blocks in one or more previously decoded video frames that are similar to the current block. The similar block(s) may be identified by a motion vector (e.g., which may identify the spatial offset of the corresponding block as compared to the current block) and may be used to generate an inter-prediction block of pixels as a prediction for the current block. Various implementations of inter-prediction are possible that provide different ways of predicting the pixels of the current block based on pixels from previously decoded video frames.

[0051] The prediction technique (e.g., intra, inter, or a combination thereof) utilized for the current block may be selected by a video encoder and may be communicated to the decoder in an encoded bitstream utilizing a prediction mode. In some cases, the mode indicates the technique utilized. In other cases, additional information is provided in the compressed bitstream depending on the prediction mode to provide information necessary for the decoder to implement the selected prediction technique.

[0052] Existing inter and intra prediction techniques, while numerous, do not result in perfectly accurate predictions. The prediction error (e.g., the difference between the actual pixel values and the predicted pixel values) result in increased bitrate, reduced quality, or a combination thereof. New prediction techniques that provide more accurate predictions (at least for certain video inputs) as compared to conventional prediction techniques are desirable in order to permit the storage and transmission of video using fewer bits (thus resulting in consumption of less bandwidth and power for the same video) and / or to provide higher quality video (e.g., where the decoded video pixels are objectively or subjectively closer to the original video pixel values).

[0053] Implementations according to this disclosure introduce data-driven intra-interprediction modes that utilize an intra-inter feature based approach to prediction. These modes may provide a more accurate prediction for certain blocks of a video. The data-driven intra-inter prediction modes described herein utilize machine learning-derived matrices to generate a prediction block. Specifically, when generating a prediction block for a current block, input features are extracted both from pixels spatially near the current block and pixels in an interprediction block generated by performing an inter-prediction process on the current block. The input features are numerical values that include information representative of pixelvalues or characteristics of the inter-prediction block, pixels spatially near the current block, or combinations thereof. The input features are used to perform a matrix multiplication with pre-trained weights from a matrix of the machine learning-derived matrices, resulting in an intermediate prediction block that may need to be subsequently resampled to match the dimensions of the current block.

[0054] Further details of techniques for data-driven intra-inter-prediction are described herein with initial reference to a system in which they can be implemented. FIG. 1 is a schematic of a video encoding and decoding system 100. A transmitting station 102 can be, for example, a computer having an internal configuration of hardware such as that described in FIG. 2. However, other implementations of the transmitting station 102 are possible. For example, the processing of the transmitting station 102 can be distributed among multiple devices.

[0055] A network 104 can connect the transmitting station 102 and a receiving station 106 for encoding and decoding of the video stream. Specifically, the video stream can be encoded in the transmitting station 102, and the encoded video stream can be decoded in the receiving station 106. The network 104 can be, for example, the Internet. The network 104 can also be a local area network (LAN), wide area network (WAN), virtual private network (VPN), cellular telephone network, or any other means of transferring the video stream from the transmitting station 102 to, in this example, the receiving station 106.

[0056] The receiving station 106, in one example, can be a computer having an internal configuration of hardware such as that described in FIG. 2. However, other suitable implementations of the receiving station 106 are possible. For example, the processing of the receiving station 106 can be distributed among multiple devices.

[0057] Other implementations of the video encoding and decoding system 100 are possible. For example, an implementation can omit the network 104. In another implementation, a video stream can be encoded and then stored for transmission at a later time to the receiving station 106 or any other device having memory. In one implementation, the receiving station 106 receives (e.g., via the network 104, a computer bus, and / or some communication pathway) the encoded video stream and stores the video stream for later decoding. In an example implementation, a real-time transport protocol (RTP) is used for transmission of the encoded video over the network 104. In another implementation, a transport protocol other than RTP may be used (e.g., a Hypertext Transfer Protocol-based (HTTP-based) video streaming protocol).

[0058] When used in a video conferencing system, for example, the transmitting station 102 and / or the receiving station 106 may include the ability to both encode and decode a video stream as described below. For example, the receiving station 106 could be a video conference participant who receives an encoded video bitstream from a video conference server (e.g., the transmitting station 102) to decode and view and further encodes and transmits his or her own video bitstream to the video conference server for decoding and viewing by other participants.

[0059] FIG. 2 is a block diagram of an example of a computing device 200 that can implement a transmitting station or a receiving station. For example, the computing device 200 can implement one or both of the transmitting station 102 and the receiving station 106 of FIG. 1. The computing device 200 can be in the form of a computing system including multiple computing devices, or in the form of one computing device, for example, a mobile phone, a tablet computer, a laptop computer, a notebook computer, a desktop computer, and the like.

[0060] A processor 202 in the computing device 200 can be a conventional central processing unit. Alternatively, the processor 202 can be another type of device, or multiple devices, capable of manipulating or processing information now existing or hereafter developed. For example, although the disclosed implementations can be practiced with one processor as shown (e.g., the processor 202), advantages in speed and efficiency can be achieved by using more than one processor.

[0061] A memory 204 in computing device 200 can be a read only memory (ROM) device or a random access memory (RAM) device in an implementation. However, other suitable types of storage device can be used as the memory 204. The memory 204 can include code and data 206 that is accessed by the processor 202 using a bus 212. The memory 204 can further include an operating system 208 and application programs 210, the application programs 210 including at least one program that permits the processor 202 to perform the techniques described herein. For example, the application programs 210 can include applications 1 through N, which further include a video coding application that performs the techniques described herein. The computing device 200 can also include a secondary storage 214, which can, for example, be a memory card used with a mobile computing device.Because the video communication sessions may contain a significant amount of information, they can be stored in whole or in part in the secondary storage 214 and loaded into the memory 204 as needed for processing.

[0062] The computing device 200 can also include one or more output devices, such as a display 218. The display 218 may be, in one example, a touch sensitive display that combines a display with a touch sensitive element that is operable to sense touch inputs. The display 218 can be coupled to the processor 202 via the bus 212. Other output devices that permit a user to program or otherwise use the computing device 200 can be provided in addition to or as an alternative to the display 218. When the output device is or includes a display, the display can be implemented in various ways, including by a liquid crystal display (LCD), a cathode-ray tube (CRT) display, or a light emitting diode (LED) display, such as an organic LED (OLED) display.

[0063] The computing device 200 can also include or be in communication with an image-sensing device 220, for example, a camera, or any other image-sensing device 220 now existing or hereafter developed that can sense an image such as the image of a user operating the computing device 200. The image-sensing device 220 can be positioned such that it is directed toward the user operating the computing device 200. In an example, the position and optical axis of the image-sensing device 220 can be configured such that the field of vision includes an area that is directly adjacent to the display 218 and from which the display 218 is visible.

[0064] The computing device 200 can also include or be in communication with a sound-sensing device 222, for example, a microphone, or any other sound-sensing device now existing or hereafter developed that can sense sounds near the computing device 200. The sound-sensing device 222 can be positioned such that it is directed toward the user operating the computing device 200 and can be configured to receive sounds, for example, speech or other utterances, made by the user while the user operates the computing device 200.

[0065] Although FIG. 2 depicts the processor 202 and the memory 204 of the computing device 200 as being integrated into one unit, other configurations can be utilized. The operations of the processor 202 can be distributed across multiple machines (wherein individual machines can have one or more processors) that can be coupled directly or across a local area or other network. The memory 204 can be distributed across multiple machines such as a network-based memory or memory in multiple machines performing the operations of the computing device 200. Although depicted here as one bus, the bus 212 of the computing device 200 can be composed of multiple buses. Further, the secondary storage 214 can be directly coupled to the other components of the computing device 200 or can be accessed via a network and can comprise an integrated unit such as a memory card ormultiple units such as multiple memory cards. The computing device 200 can thus be implemented in a wide variety of configurations.

[0066] FIG. 3 is a diagram of an example of a video stream 300 to be encoded and subsequently decoded. The video stream 300 includes a video sequence 302. At the next level, the video sequence 302 includes a number of adjacent frames 304. While three frames are depicted as the adjacent frames 304, the video sequence 302 can include any number of adjacent frames 304. The adjacent frames 304 can then be further subdivided into individual frames, for example, a frame 306. At the next level, the frame 306 can be divided into a series of planes or segments 308. The segments 308 can be subsets of frames that permit parallel processing, for example. The segments 308 can also be subsets of frames that can separate the video data into separate colors. For example, a frame 306 of color video data can include a luminance plane and two chrominance planes. The segments 308 may be sampled at different resolutions.

[0067] Whether or not the frame 306 is divided into segments 308, the frame 306 may be further subdivided into blocks 310, which can contain data corresponding to, for example, 16x16 pixels in the frame 306. The blocks 310 can also be arranged to include data from one or more segments 308 of pixel data. The blocks 310 can also be of any other suitable size such as 4x4 pixels, 8x8 pixels, 16x8 pixels, 8x16 pixels, 16x16 pixels, or larger. Unless otherwise noted, the terms block and macroblock are used interchangeably herein.

[0068] FIG. 4 is a block diagram of an encoder 400 according to implementations of this disclosure. The encoder 400 can be implemented, as described above, in the transmitting station 102, such as by providing a computer software program stored in memory, for example, the memory 204. The computer software program can include machine instructions that, when executed by a processor such as the processor 202, cause the transmitting station 102 to encode video data in the manner described in FIG. 4. The encoder 400 can also be implemented as specialized hardware included in, for example, the transmitting station 102. In one particularly desirable implementation, the encoder 400 is a hardware encoder.

[0069] The encoder 400 has the following stages to perform the various functions in a forward path (shown by the solid connection lines) to produce an encoded or compressed bitstream 420 using the video stream 300 as input: an intra / inter prediction stage 402, a transform stage 404, a quantization stage 406, and an entropy encoding stage 408. The encoder 400 may also include a reconstruction path (shown by the dotted connection lines) to reconstruct a frame for encoding of future blocks. In FIG. 4, the encoder 400 has the following stages to perform the various functions in the reconstruction path: a dequantizationstage 410, an inverse transform stage 412, a reconstruction stage 414, and a loop filtering stage 416. Other structural variations of the encoder 400 can be used to encode the video stream 300.

[0070] When the video stream 300 is presented for encoding, respective adjacent frames 304, such as the frame 306, can be processed in units of blocks. At the intra / inter prediction stage 402, respective blocks can be encoded using intra-frame prediction (also called intra-prediction) or inter- frame prediction (also called inter-prediction). In any case, a prediction block can be formed. In the case of intra-prediction, a prediction block may be formed from samples in the current frame that have been previously encoded and reconstructed. In the case of inter-prediction, a prediction block may be formed from samples in one or more previously constructed reference frames.

[0071] Next, the prediction block can be subtracted from the current block at the intra / inter prediction stage 402 to produce a residual block (also called a residual). The transform stage 404 transforms the residual into transform coefficients in, for example, the frequency domain using block-based transforms. The quantization stage 406 converts the transform coefficients into discrete quantum values, which are referred to as quantized transform coefficients, using a quantizer value or a quantization level. For example, the transform coefficients may be divided by the quantizer value and truncated.

[0072] The quantized transform coefficients are then entropy encoded by the entropy encoding stage 408. The entropy-encoded coefficients, together with other information used to decode the block (which may include, for example, syntax elements such as used to indicate the type of prediction used, transform type, motion vectors, a quantizer value, or the like), are then output to the compressed bitstream 420. The compressed bitstream 420 can be formatted using various techniques, such as variable length coding (VLC) or arithmetic coding. The compressed bitstream 420 can also be referred to as an encoded video stream or encoded video bitstream, and the terms will be used interchangeably herein.

[0073] The reconstruction path (shown by the dotted connection lines) can be used to ensure that the encoder 400 and a decoder 500 (described below with respect to FIG. 5) use the same reference frames to decode the compressed bitstream 420. The reconstruction path performs functions that are similar to functions that take place during the decoding process (described below with respect to FIG. 5), including dequantizing the quantized transform coefficients at the dequantization stage 410 and inverse transforming the dequantized transform coefficients at the inverse transform stage 412 to produce a derivative residual block (also called a derivative residual). At the reconstruction stage 414, the prediction blockthat was predicted at the intra / inter prediction stage 402 can be added to the derivative residual to create a reconstructed block. The loop filtering stage 416 can be applied to the reconstructed block to reduce distortion such as blocking artifacts.

[0074] Other variations of the encoder 400 can be used to encode the compressed bitstream 420. In some implementations, a non-transform based encoder can quantize the residual signal directly without the transform stage 404 for certain blocks or frames. In some implementations, an encoder can have the quantization stage 406 and the dequantization stage 410 combined in a common stage.

[0075] FIG. 5 is a block diagram of a decoder 500 according to implementations of this disclosure. The decoder 500 can be implemented in the receiving station 106, for example, by providing a computer software program stored in the memory 204. The computer software program can include machine instructions that, when executed by a processor such as the processor 202, cause the receiving station 106 to decode video data in the manner described in FIG. 5. The decoder 500 can also be implemented in hardware included in, for example, the transmitting station 102 or the receiving station 106.

[0076] The decoder 500, similar to the reconstruction path of the encoder 400 discussed above, includes in one example the following stages to perform various functions to produce an output video stream 516 from the compressed bitstream 420: an entropy decoding stage 502, a dequantization stage 504, an inverse transform stage 506, an intra / inter prediction stage 508, a reconstruction stage 510, a loop filtering stage 512, and a deblocking filtering stage 514. Other structural variations of the decoder 500 can be used to decode the compressed bitstream 420.

[0077] When the compressed bitstream 420 is presented for decoding, the data elements within the compressed bitstream 420 can be decoded by the entropy decoding stage 502 to produce a set of quantized transform coefficients. The dequantization stage 504 dequantizes the quantized transform coefficients (e.g., by multiplying the quantized transform coefficients by the quantizer value), and the inverse transform stage 506 inverse transforms the dequantized transform coefficients to produce a derivative residual that can be identical to that created by the inverse transform stage 412 in the encoder 400. Using header information decoded from the compressed bitstream 420, the decoder 500 can use the intra / inter prediction stage 508 to create the same prediction block as was created in the encoder 400 (e.g., at the intra / inter prediction stage 402).

[0078] At the reconstruction stage 510, the prediction block can be added to the derivative residual to create a reconstructed block. The loop filtering stage 512 can be appliedto the reconstructed block to reduce blocking artifacts. Other filtering can be applied to the reconstructed block. In this example, the deblocking filtering stage 514 is applied to the reconstructed block to reduce blocking distortion, and the result is output as the output video stream 516. The output video stream 516 can also be referred to as a decoded video stream, and the terms will be used interchangeably herein. Other variations of the decoder 500 can be used to decode the compressed bitstream 420. In some implementations, the decoder 500 can produce the output video stream 516 without the deblocking filtering stage 514.

[0079] FIG. 6 is a flowchart of a technique 600 for generating a prediction block for a current block using a data-driven intra-inter-prediction. The technique 600 can be implemented, for example, as a software program that may be executed by computing devices such as transmitting station 102 or receiving station 106. The software program can include machine-readable instructions that may be stored in a memory such as the memory 204 or the secondary storage 214, and that, when executed by a processor, such as the processor 202, may cause the computing device to perform the technique 600. The software program may be stored in a non-transitory computer readable medium and / or may take the form of a computer program product. The technique 600 may be implemented in whole or in part in the intra / inter prediction stage 402 of the encoder 400 of FIG. 4 and / or the intra / inter prediction stage 508 of the decoder 500 of FIG. 5. When implemented by an encoder, coding means encoding, as described with respect to FIG. 4; and when implemented by a decoder, coding means decoding, as described with respect to FIG. 5. The technique 600 can be implemented using specialized hardware or firmware. Multiple processors, memories, or both, may be used. Technique 600 may be used to encode and / or decode a compressed bitstream. The compressed bitstream may be stored on a non-transitory computer readable medium and / or may take the form of a computer media product.

[0080] At 602, the current block is identified as the block for which a prediction block is to be generated. For example, a video frame may be subdivided into blocks, some or all of which may be processed as a current block by technique 600. At 604, a data-driven intra-inter-prediction mode is identified (e.g., selected) for generating the prediction block.

[0081] When implemented by an encoder, the technique 600 may identify the data-driven intra-inter-prediction mode based on rate-distortion optimization. Rate-distortion (R-D) optimization represents a trade-off between compression efficiency (rate, measured in bits) and quality loss (distortion) in video encoding. This optimization process is typically formulated using a Lagrangian cost function: J = D + 1R, where J is the cost to minimize, D is the distortion (quality loss), R is the rate (bits), and A (lambda) is the Lagrangian multiplierthat weights the relative importance of rate versus distortion. The optimization process may occur at multiple levels in the encoding pipeline, including coding unit decisions, transform selections, and quantization parameters.

[0082] When selecting between available coding modes (such as intra prediction, inter prediction, and data-driven intra-prediction), the encoder may calculate distortion using metrics like Sum of Absolute Differences (SAD), determines the rate including both mode signaling and residual coding bits, and computes the combined R-D cost. For computational efficiency, the encoder may be designed to employ various optimization strategies including early termination of mode searches, content-aware mode filtering, adaptive thresholds to skip testing unlikely modes, and statistical tracking of mode usage patterns, ultimately selecting the mode that minimizes the R-D cost function while respecting bitrate and quality constraints. With respect to data-driven intra-inter-prediction modes, the encoder may select one (i.e., the data-driven intra- inter-prediction mode) of multiple available data-driven intra-inter-prediction modes based on the rate-distortion optimization.

[0083] A data-driven intra-inter-prediction mode may be associated with (e.g., identify) a matrix 606, which is a set of weights. The encoder may include several matrices, each corresponding to a respective data-driven intra-inter-prediction mode. For example, the codec (e.g., the encoder and the decoder) may include, by way of example and not limitation, six matrices. In such an implementation, six data-driven intra- inter-prediction modes are available. The data-driven matrices may have been empirically derived. The data-driven matrices may be generated using machine learning techniques, such as by using a training process over a corpus of sample videos. In some cases, videos which have similar characteristics may be grouped together to produce a set of weights. Each matrix may have dimensions selected to accommodate a fixed number of input features while producing an output of predetermined dimensions (e.g., generating an 8x8 output matrix). The matrices may be generated by a machine learning model that is trained using a dataset of frames to optimize prediction accuracy while maintaining a compact representation. More generally, each matrix may be of size PxQ, where P corresponds to the number of input features derived from the neighboring samples of the current block and Q corresponds to a number of positions in an output matrix to be predicted. In an example, and as further explained herein, by way of illustration and not limitation, twelve input features (P=12) can be derived from pixels in twelve sample areas neighboring the current block and in an inter-prediction of the current block. In an example, sixty-four output values (Q=64) are generated which, for example, may correspond to a 8x8 output matrix.

[0084] The encoder encodes, as described with respect to FIG. 8, the identified data-driven intra-inter-prediction mode in a compressed bitstream, such as the compressed bitstream 420 of FIG. 4. As such, when the technique 600 is implemented by the decoder, the technique 600 identifies the data-driven intra-inter-prediction mode by decoding the mode from a compressed bitstream, such as the compressed bitstream 420 of FIG. 5.

[0085] At 608, an input vector of features is extracted (e.g., derived) from samples (e.g., pixels) neighboring the current block. The number of features is based on the size of the data-driven matrix. As mentioned above, the data-driven matrix can have a size of PxQ. As such, P features are derived. The input vector can be implemented as a IxP row vector. Deriving the features for the input vector is further described with respect to FIGS. 7A-D.

[0086] FIGS. 7A-D illustrates examples 700, 740, 750, 760 of sample areas useable for extracting a feature vector. The number of features is equal to P (e.g., 12). The example 700 include a current block 702 and an inter-predicted block 704. Inter-predicted block 704 is generated by performing an inter-prediction process with respect to the current block 702. Source regions are identified in cross-hatch as demonstrated by source region legend 706.

[0087] Example 700 includes source regions 710, 712, 714, 716, 718, 720, 722, 724, 730, 732, 734, and 736. A source region encompasses an area of pixels which may be proportionate to a block size of the current block. For example, if the current block as shown is a matrix of 8x8 pixels, the shown source regions each encompass 2x2 pixels. For example, if the current block as shown is a matrix of 16x16 pixels, the shown source regions each encompass 4x4 pixels. By making the source regions proportionate to block size instead of fixed to a certain number of pixels, the provided source regions may be utilized in a similar manner for different block sizes. Some of the source regions (e.g., 710, 712, 714, 716, 718, 720, 722, 724) correspond with pixels in blocks adjacent to the current block (e.g., intra regions) and other source regions (e.g., 730, 732, 734, 736) correspond with pixels in interpredicted block 704 (e.g., inter-predicted regions).

[0088] Example 740 provides an alternative configuration of source regions that omits region 730 and adds region 742. Example 750 provides an alternative configuration of source regions that utilizes region 752 instead of region 730. Example 760 provides an alternative configuration of source regions that utilizes region 762 instead of region 730. Depending on the implementation, different configurations of source regions are possible other than what is depicted in examples 700, 740, 750, or 760, including those that utilize additional, fewer, or different source regions. While the shown source regions are all of the same size, implementations using differently sized source regions are possible. For example, sourceregions adjacent to the current block may encompass rectangular areas, which may be limited to pixels in a row of pixels immediately above the current block or a column of pixels immediately to the left of the current block.

[0089] While the shown source regions include a certain number of source regions corresponding to pixels adjacent to the current block (the intra regions) and a certain number of source regions corresponding to pixels in an inter predicted block (the inter-predicted regions), the number of source regions corresponding to each particular type of region may vary. For example, in some implementations, there may be multiple inter predicted blocks, a bi-directionally predicted block, one or more blocks or other groupings of pixels predicted using other techniques, or combinations thereof for which one or more source regions correspond to (e.g., for particular region(s) of pixels in such blocks or groupings).

[0090] While the figures show source regions that include intra source regions from above and to the left of the current block and inter-predicted source regions towards the bottom right of the inter-predicted block, the locations of the source regions may vary. As an example, in some implementations, the locations of inter-predicted source regions and intra source regions may be situated such that there is a distribution or variation in location so as to provide more varied sources of data. For example, when the intra source regions are above and to the left of the current block, inter-predicted source regions may be located towards the bottom-right of the inter-predicted block because they are situated further from the boundary of the current block with the intra source regions.

[0091] The features of the input vector can be obtained from the source regions in any number of ways. In an example, a feature can be obtained as a weighted average of pixels within a source region, an average of pixels within a source region, a pixel value from with a source region (e.g., if the region encompasses a single pixel), or combinations thereof. For example, each source region may have a corresponding feature extracted from the pixels in that region, resulting in the production of one feature per source region.

[0092] Referring again to FIG. 6, at 610, the input vector of features is multiplied by the data-driven matrix (which may also be referred to as a prediction matrix) to obtain an output vector 612. Given an input vector v of features of size IxP and a data-driven matrix M of size PxQ, the matrix multiplication produces an output vector o of size IxQ. For example, if the data-driven matrices are each of size 12x64, and the input vector is of size 1x12, then the output vector would be of size 1x64.

[0093] At 614, the output vector is converted to an output matrix. The output vector is converted to the output matrix by reshaping the IxQ vector into a square matrix. Forexample, when Q=64, the 1x64 output vector is reshaped into an 8x8 matrix by sequentially filling the matrix row by row with elements from the vector. Specifically, the first 8 elements of the vector become the first row of the output matrix, the next 8 elements become the second row, and so on, until all 64 elements have been arranged into the 8x8 output matrix. This reshaping process preserves the order of elements while transforming them into a two-dimensional prediction block format suitable for subsequent processing. As such, Q may be a perfect square (e.g., 64 = 82) to enable a square matrix conversion.

[0094] At 616, the output matrix is resized into the size of the current block (e.g., WxH). The resizing is performed separately along each dimension, with one axis being processed before the other. For the horizontal dimension, when the width W is greater than the output matrix width, upsampling is performed using interpolation techniques such as linear or higher-order interpolation. When the width W is less than the output matrix width, downsampling is applied. When the width W equals the output matrix width, the horizontal dimension remains unmodified. Similarly, for the vertical dimension, when the height H is greater than the output matrix height, upsampling is performed using interpolation. When the height H is less than the output matrix height, downsampling is applied. When H equals the output matrix height, the vertical dimension remains unmodified. This dimension-by-dimension approach allows for flexible resizing of the prediction block to match any target block dimensions while maintaining prediction quality through appropriate sampling techniques.

[0095] In an example, data-driven intra-inter-prediction modes are only available for current blocks that are larger than or equal to a certain threshold size. Assuming the current block has a size of WxH (i.e., a width of W and a height of H), then the data-driven intra-inter-prediction modes may only be available for blocks where W*H > Threshold size (e.g., 128). This size-based restriction may have certain benefits. First, it aids hardware implementation efficiency, as processing smaller blocks typically represents a bottleneck in hardware implementations. By limiting data-driven modes to larger blocks, the implementation avoids adding computational complexity to these performance-critical paths. Second, experimental results indicate that the coding gains from these data-driven modes are most significant for larger block sizes, making this restriction both practically efficient and coding-effective.

[0096] FIG. 8 illustrates an example 800 of signaling of data-driven intra-inter-prediction modes. Signaling, in this context, refers to what an encoder encodes, in a compressed bitstream, regarding the prediction mode of a current block and what a decoder decodes fromthe compressed bitstream. The signaling can include one or more syntax elements. The syntax elements can be encoded in a header of the current block. The syntax elements described with respect to the example 800 include a USE_DDIIP flag 802 indicating whether data-driven prediction is used for the current block, a DDIIP_MODE syntax element 804 identifying which of the available data-driven intra-inter-prediction modes (i.e., matrices) is used when USE_DDIIP is set.

[0097] In some implementations, there may be M DDIIP modes in a codec and a DDIIP syntax element includes a mode index m in [0, M-l]. In some implementations, each mode m could use a different configuration of features for adaptability. For example, each mode may use different examples from FIGS. 7A-D, different combinations of source regions described in those examples, or other source regions not shown. Each mode may be characterized by a single prediction matrix of size (bwbh) x Fm, where Fmis the total number of intra and inter features for mode m.

[0098] FIG. 9 is a flowchart of a technique 900 for generating a prediction block for a current block using a data-driven intra-prediction. The technique 900 can be implemented, for example, as a software program that may be executed by computing devices such as transmitting station 102 or receiving station 106. The software program can include machine-readable instructions that may be stored in a memory such as the memory 204 or the secondary storage 214, and that, when executed by a processor, such as the processor 202, may cause the computing device to perform the technique 900. A processor may be configured to execute instructions stored in the memory to perform technique 900. The software program may be stored in a non-transitory computer readable medium and / or may take the form of a computer program product. The technique 900 may be implemented in whole or in part in the intra / inter prediction stage 402 of the encoder 400 of FIG. 4 and / or the intra / inter prediction stage 508 of the decoder 500 of FIG. 5. When implemented by an encoder, coding means encoding, as described with respect to FIG. 4; and when implemented by a decoder, coding means decoding, as described with respect to FIG. 5. The technique 900 can be implemented using specialized hardware or firmware. Multiple processors, memories, or both, may be used. Technique 900 may be used to encode and / or decode a compressed bitstream. The compressed bitstream may be stored on a non-transitory computer readable medium and / or may take the form or a computer media product.

[0099] At 902, a data-driven intra-inter-prediction mode associated with a prediction matrix is identified. This identification can occur through decoding the mode from a compressed bitstream, when implemented by the decoder; or by selecting the mode based onrate-distortion optimization, when implemented by the encoder. The prediction matrix may be one of several pre-trained matrices, such as six matrices generated using machine learning techniques, each corresponding to at least one respective data-driven intra-inter-prediction mode. For example, the matrices, and for a given mode, the prediction matrix for that mode, may be determined for and included within a video decoding specification, encoder implementation, decoder implementation, or combinations thereof.

[0100] At 903, an inter-prediction for a current block is generated by using one or more inter-prediction techniques. For example, such techniques may include using uni-prediction (e.g., using a single motion vector), bi-prediction (e.g., where two or more motion vectors are used and the resulting predictions are combined to obtain an inter-prediction), varying forms of implicit prediction, such as an implicit inter prediction technique (e.g., where a motion vector, reference frame, other inter-prediction parameter, or combinations thereof are predetermined or inferred based on context instead of being explicitly signaled), or combinations thereof. In some implementations, the inter-prediction techniques or parameters available may be constrained (as compared to what is available when inter-predicting a block outside of a data-driven intra-inter-prediction mode), for example, to the closest (e.g., most recent) reference frame, a motion vector may be limited to a top-ranked motion vector candidate in a dynamic list of motion vector reference candidates obtained from a spatial and / or temporal neighborhood, or combinations thereof. For example, 903 may include constraining a reference frame to a closest reference picture and constraining a motion vector used for inter prediction to a first motion vector in a dynamic list of reference candidates generated from a spatial and temporal neighborhood.

[0101] At 904, an input vector of features is extracted from source regions for a current block. For example, the source regions may be arranged such as described in one of the examples depicted in FIGS. 7A-D. For example, one feature may be extracted for each source region. In such an implementation, 904 may include determining a feature for each respective source region associated with the current block. The extraction may be performed by computing an average, a weighted average, or combination thereof on pixels within a source region. In some implementations, a pixel value may be selected as the feature for a particular source region. Other techniques may be used to obtain features for source regions based on pixels in the respective source region.

[0102] In some implementations, the data-driven intra-inter-prediction mode has a predetermined number of source regions each encompassing a respective pre-determined regionof pixels that are proportionate to a size of the current block and either adjacent to the current block or within the inter-prediction block generated for the current block.

[0103] At 906, the input vector is multiplied by the prediction matrix to obtain an output vector. The prediction matrix has dimensions PxQ or QxP, where P corresponds to the number of input features and Q is a perfect square, with specific implementations using P=12 and Q=64. For example, the multiplication may be implemented as a vector-matrix multiplication (e.g., a IxP input vector multiplied by a PxQ matrix) or a matrix-vector multiplication (e.g., a QxP matrix multiplied by a Pxl input vector), depending on whether the input features are arranged as a row vector or a column vector. At 908, the output vector is converted into an output matrix through a reshaping process.

[0104] At 910, the output matrix is resized to match dimensions of the current block to generate a prediction block. This resizing may include up sampling when a target dimension is greater than the corresponding output matrix dimension, down sampling when it is less, and maintaining the dimension when they are equal. For example, resizing may include performing dimension-by-dimension processing of the output matrix to match the dimensions of the current block.

[0105] In some implementations, technique 900 may only be performed when the dimensions of the current block are greater than or equal to a threshold size, such as 128 samples. For example, technique 900 may include determining whether the dimensions of the current block are greater than or equal to a threshold size and performing steps of technique 900 in response to determining that the dimensions are greater than or equal to the threshold size. The data-driven intra-inter-prediction mode may be encoded in a compressed bitstream using various syntax elements including a flag indicating whether data-driven prediction is used and a syntax element identifying which mode is used.

[0106] Other or more specific implementations of technique 900 are possible.

[0107] FIG. 10 is a flowchart of a technique 1000 for generating a prediction block for a current block using a data-driven intra-inter-prediction. The technique 1000 can be implemented, for example, as a software program that may be executed by computing devices such as transmitting station 102 or receiving station 106. The software program can include machine-readable instructions that may be stored in a memory such as the memory 204 or the secondary storage 214, and that, when executed by a processor, such as the processor 202, may cause the computing device to perform the technique 1000. The software program may be stored in a non-transitory computer readable medium and / or may take the form of a computer program product. A processor may be configured to execute instructionsstored in the memory to perform technique 1000. The technique 1000 may be implemented in whole or in part in the intra / inter prediction stage 402 of the encoder 400 of FIG. 4 and / or the intra / inter prediction stage 508 of the decoder 500 of FIG. 5. When implemented by an encoder, coding means encoding, as described with respect to FIG. 4; and when implemented by a decoder, coding means decoding, as described with respect to FIG. 5. The technique 1000 can be implemented using specialized hardware or firmware. Multiple processors, memories, or both, may be used. Technique 1000 may be used to encode and / or decode a compressed bitstream. The compressed bitstream may be stored on a non-transitory computer readable medium and / or may take the form of a computer media product.

[0108] At 1002, features are extracted from first source regions adjacent to a current block. For example, pixels in a source region may be averaged to determine a feature for that source region. Other techniques may be utilized such as described previously.

[0109] At 1004, features are extracted from second source regions in an inter-prediction of the current block. For example, an inter-prediction may be generated for a current block, such as by using inter-prediction techniques as previously described. Once the interprediction is generated, features may be extracted from pixels in the inter-prediction. For example, pixels in a source region in the inter-prediction may be averaged to determine a feature for that source region. Other techniques may be utilized such as described previously.

[0110] At 1006, an intermediate prediction block is generated by applying weights to the extracted features. For example, a matrix multiplication operation may be used between a matrix of extracted feature values (e.g., including the first plurality of features and the second plurality of features) and a prediction matrix of respective sets of weights for intermediate prediction locations corresponding to the intermediate prediction block (e.g., if an intermediate prediction is 8x8, there would be 64 intermediate prediction locations). The resulting values may be arranged in a block format (e.g., the intermediate prediction block). An intermediate prediction block may also be referred to as an output matrix herein. Step 1006 may be implemented according to or utilizing techniques such as described previously with respect to 610 to 614 or 906-908.

[0111] At 1008, a data-driven prediction block is generated based on the intermediate prediction block. For example, the intermediate prediction block may be resampled to produce a data-driven prediction block that is the same dimensions as the current block.

[0112] Other or more specific implementations of technique 1000 are possible. For example, the current block is a matrix of pixels, such as an 8x8 block of pixels, the intermediate prediction block is a matrix of intermediate predictions, such as a 4x4 block ofintermediate predictions. For example, a height or width of the matrix of pixels may be greater than a height or width of the intermediate prediction, and resampling the intermediate prediction includes up sampling the intermediate prediction block. For example, the respective set of weights and locations of the first source areas and second source areas may be defined according to a data-driven intra-inter-prediction mode. For example, an identification of the intra-inter-prediction mode may be encoded into an encoded bitstream. For example, an identification of the intra-inter-prediction mode may be decoded from an encoded bitstream.

[0113] For example, the respective set of weights may be determined by a training process performed on a training set of videos. For example, the respective set of weights may be pre-determined and made available to encoder and decoder prior to encoding or decoding.

[0114] In some implementations, a current block has a size W x H and an intermediate predictor is generated of size bwx bh, where W > bwand H > bh. In some examples, bw= bh = 8. Let T denote the DDIIP matrix (e.g., prediction matrix M as previously described) corresponding to the chosen DDIIP mode m. The dimensions of T are (bwbh) x F, where F is a feature vector size (e.g., corresponding to a number of features extracted relating to the current block). Prediction may be performed by first multiplying T with a F-dimensional feature vector v obtained from the causal intra and inter neighborhood of the block (e.g., based on pixel values in source regions respectively associated with elements in the feature vector). Therefore, the intermediate (bwbh)-dimensional prediction vector Pint may be determined as, Pint = T x v. The (bwbh)-dimensional vector Pint may next be rearranged to form a bwx bh block, which is up sampled using a standard spatial interpolator such as bilinear, bicubic or Lanczos in the horizontal and vertical directions by ratios W / bwand H / bh respectively to obtain the final prediction block of size W x H. To summarize, we can refer to the process of generating the final predictor of a WxH block P as:

[0115] P = Upsamplewxn(Pint) = Upsamplewxn(T x v)

[0116] There may be multiple DDIIP modes, say M, in a codec. For individual, or each block the mode index m in [0, M-l] may be signaled. A mode m corresponds to a DDIIP matrix Tmthat is used to generate the predictor Pmusing:

[0117] Pm= UpsarnplewxH(Pintm) = Upsamplewxn(Tmx v)

[0118] For simplicity of explanation, the techniques 600, 900, and 1000 of FIGS. 6, 9, and 10, respectively, are each depicted and described as respective series of steps or operations. However, the steps or operations in accordance with this disclosure can occur in various orders and / or concurrently. Additionally, other steps or operations not presented anddescribed herein may be used. Furthermore, not all illustrated steps or operations may be required to implement a technique in accordance with the disclosed subject matter.Implementations of techniques 600, 900, or 1000 may utilize techniques or disclosure described with respect to other techniques or otherwise described herein.

[0119] The aspects of encoding and decoding described above illustrate some examples of encoding and decoding techniques. However, it is to be understood that encoding and decoding, as those terms are used in the claims, could mean compression, decompression, transformation, or any other processing or change of data.

[0120] The word “example” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “example” is not necessarily to be construed as being preferred or advantageous over other aspects or designs. Rather, use of the word “example” is intended to present concepts in a concrete fashion. As used in this application, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless specified otherwise or clearly indicated otherwise by the context, the statement “X includes A or B” is intended to mean any of the natural inclusive permutations thereof. That is, if X includes A; X includes B; or X includes both A and B, then “X includes A or B” is satisfied under any of the foregoing instances. In addition, the articles “a” and “an” as used in this application and the appended claims should generally be construed to mean “one or more,” unless specified otherwise or clearly indicated by the context to be directed to a singular form. Moreover, use of the term “an implementation” or the term “one implementation” throughout this disclosure is not intended to mean the same embodiment or implementation unless described as such.

[0121] Implementations of the transmitting station 102 and / or the receiving station 106 (and the algorithms, methods, instructions, etc., stored thereon and / or executed thereby, including by the encoder 400 and the decoder 500) can be realized in hardware, software, or any combination thereof. The hardware can include, for example, computers, intellectual property (IP) cores, application- specific integrated circuits (ASICs), programmable logic arrays, optical processors, programmable logic controllers, microcode, microcontrollers, servers, microprocessors, digital signal processors, or any other suitable circuit. In the claims, the term “processor” should be understood as encompassing any of the foregoing hardware, either singly or in combination. The terms “signal” and “data” are used interchangeably. Further, portions of the transmitting station 102 and the receiving station 106 do not necessarily have to be implemented in the same manner.

[0122] Further, in one aspect, for example, the transmitting station 102 or the receiving station 106 can be implemented using a general purpose computer or general purpose processor with a computer program that, when executed, carries out any of the respective methods, algorithms, and / or instructions described herein. In addition, or alternatively, for example, a special purpose computer / processor can be utilized which can contain other hardware for carrying out any of the methods, algorithms, or instructions described herein.

[0123] The transmitting station 102 and the receiving station 106 can, for example, be implemented on computers in a video conferencing system. Alternatively, the transmitting station 102 can be implemented on a server, and the receiving station 106 can be implemented on a device separate from the server, such as a handheld communications device. In this instance, the transmitting station 102, using an encoder 400, can encode content into an encoded video signal and transmit the encoded video signal to the communications device. In turn, the communications device can then decode the encoded video signal using a decoder 500. Alternatively, the communications device can decode content stored locally on the communications device, for example, content that was not transmitted by the transmitting station 102. Other suitable transmitting and receiving implementation schemes are available. For example, the receiving station 106 can be a generally stationary personal computer rather than a portable communications device, and / or a device including an encoder 400 may also include a decoder 500.

[0124] Further, all or a portion of implementations of the present disclosure can take the form of a computer program product accessible from, for example, a computer-usable or non-transitory computer-readable storage medium. A computer-usable or computer-readable medium can be any device that can, for example, tangibly contain, store, communicate, or transport the program (e.g., including executable instructions) for use by or in connection with any processor. The medium can be, for example, an electronic, magnetic, optical, electromagnetic, or semiconductor device. Other suitable mediums are also available.

[0125] Further, all or a portion of implementations of the present disclosure may include a video encoding apparatus or video decoding apparatus implementing techniques, or a portion thereof, described herein.

[0126] The above-described embodiments, implementations, and aspects have been described in order to facilitate easy understanding of this disclosure and do not limit this disclosure. On the contrary, this disclosure is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims, which scope is tobe accorded the broadest interpretation as is permitted under the law so as to encompass all such modifications and equivalent arrangements.

Claims

What is claimed is:

1. A method, comprising:identifying a data-driven intra-inter-prediction mode associated with a prediction matrix;generating an inter-prediction for a current block;extracting an input vector of features from source regions associated with the current block, the source regions including at least a first source region from the inter-prediction and at least a second source region from a neighboring block spatially adjacent to the current block;multiplying the input vector by the prediction matrix to obtain an output vector; converting the output vector into an output matrix; andresizing the output matrix to match dimensions of the current block to generate a prediction block for the current block.

2. The method of claim 1, wherein identifying the data-driven intra-interprediction mode comprises:decoding the data-driven intra-inter-prediction mode from a compressed bitstream.

3. The method of claim 1, wherein identifying the data-driven intra-interprediction mode comprises:selecting the data-driven intra-inter-prediction mode based on rate-distortion optimization.

4. The method of any one of claims 1 to 3, wherein the prediction matrix is one of a plurality of pre-trained prediction matrices, each prediction matrix corresponding to at least one respective data-driven intra-inter-prediction mode.

5. The method of claim 4, wherein the plurality of pre-trained prediction matrices comprises six prediction matrices generated using machine learning techniques.

6. The method of any one of claims 1 to 5, further comprising:determining whether the dimensions of the current block are greater than or equal to a threshold size; andperforming the extracting, multiplying, converting, and resizing in response to determining that the dimensions are greater than or equal to the threshold size.

7. The method of claim 6, wherein the threshold size is 128 samples.

8. The method of any one of claims 1 to 7, wherein extracting the input vector comprises:determining a feature for each respective source region associated with the current block.

9. The method of any one of claims 1 to 8, wherein the source regions include source regions above the current block, source regions left of the current block, and source regions within an inter-prediction of the current block.

10. The method of any one of claims 1 to 9, wherein the data-driven intra-interprediction mode has a pre- determined number of source regions each encompassing a respective pre-determined region of pixels that are proportionate to a size of the current block and either adjacent to the current block or within the inter-prediction block generated for the current block.

11. The method of any one of claims 1 to 10, wherein at least one of the features in the input vector is obtained as an average of samples in a source region.

12. The method of any one of claims 1 to 11, wherein converting the output vector into the output matrix comprises:reshaping the output vector into the output matrix.

13. The method of any one of claims 1 to 12, wherein resizing the output matrix comprises:performing dimension-by-dimension processing of the output matrix to match the dimensions of the current block.

14. The method of claim 13, wherein performing the dimension-by-dimension processing comprises:applying up sampling when a target dimension is greater than a corresponding output matrix dimension;applying down sampling when the target dimension is less than the corresponding output matrix dimension; andmaintaining a dimension of the output matrix when the target dimension equals the corresponding output matrix dimension.

15. The method of any one of claims 1 to 14, further comprising:encoding the data-driven intra-inter-prediction mode in a compressed bitstream.

16. The method of claim 15, wherein encoding the data-driven intra-interprediction mode in the compressed bitstream comprises:encoding a flag indicating whether data-driven intra-inter-prediction is used for the current block; andencoding a syntax element identifying which of a plurality of available data-driven intra-inter-prediction modes is used.

17. The method of any one of claims 1 to 16, wherein the prediction matrix has dimensions PxQ or QxP, where P corresponds to a number of input features and Q is a perfect square.

18. The method of claim 17, wherein P is 12 and Q is 64.

19. The method of any one of claims 1 to 18, wherein generating an interprediction for the current block includes utilizing at least one of bi-prediction or an implicit inter prediction technique.

20. The method of any one of claims 1 to 19, wherein generating an interprediction for the current block includes constraining a reference frame to a closest reference picture and constraining a motion vector used for inter prediction to a first motion vector in a dynamic list of reference candidates generated from a spatial and temporal neighborhood.

21. A method for predicting a current block in a video frame, the method comprising:extracting a first plurality of features based on pixels in first source regions adjacent to the current block;extracting a second plurality of features based on pixels in second source regions in an inter-prediction of the current block;generating an intermediate prediction block for the current block by applying a respective set of weights to the first and second plurality of features at intermediate prediction locations in the intermediate prediction block; andgenerating a data-driven prediction block for the current block by resampling the intermediate prediction block.

22. The method of claim 21, wherein the current block is a matrix of pixels, the intermediate prediction block is a matrix of intermediate predictions, a height or width of the matrix of pixels is greater than a height or width of the intermediate prediction block, and resampling the intermediate prediction block includes up sampling the intermediate prediction block.

23. The method of any one of claims 21 to 22, wherein the respective set of weights and locations of the first source regions and second source regions are defined according to a data-driven intra-inter-prediction mode.

24. The method of claim 23, further comprising:encoding an identification of the intra-inter-prediction mode into an encoded bitstream.

25. The method of claim 23, further comprising:decoding an identification of the intra-inter-prediction mode from an encoded bitstream.

26. The method of any one of claims 21 to 25, wherein generating the intermediate prediction block includes performing a matrix multiplication operation between a first matrix including the first plurality of features and the second plurality of features and a second matrix including respective sets of weights for intermediate prediction locations corresponding to the intermediate prediction block.

27. The method of any one of claims 21 to 26, wherein the respective set of weights is determined by a training process performed on a training set of videos.

28. The method of any one of claims 21 to 27, wherein the respective set of weights is pre- determined and made available to encoder and decoder prior to encoding or decoding.

29. A device, comprising:a processor that is configured to perform the method of any one of claims 1 to 28.

30. A device, comprising:a memory; anda processor, the processor configured to execute instructions stored in the memory to perform the method of any one of claims 1 to 28.

31. A video decoding apparatus implementing the method of any one of claims 1 to 2, 4 to 14, 17 to 23, or 25 to 28.

32. A video encoding apparatus implementing the method of any one of claims 1, 3 to 24, or 26 to 28.

33. A non-transitory computer-readable storage medium, comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising operations that perform the method of any one of claims 1 to 28.

34. A computer program product comprising instructions which, when the instructions are executed by a computer, cause the computer to carry out the method of any one of claims 1 to 20.

35. A non-transitory computer readable medium comprising an encoded bitstream that was encoded using the method of any one of claims 1, 3 to 24, or 26 to 28 or is decodable using the method of any one of claims 1 to 2, 4 to 14, 17 to 23, or 25 to 28.

36. A computer media product comprising an encoded bitstream encoded using-SO-the method of any one of claims 1, 3 to 24, or 26 to 28 or that is decodable using the method of any one of claims 1 to 2, 4 to 14, 17 to 23, or 25 to 28.

Images