Intra block copy with long-term referencing

Abstract

A first local search area that includes previously decoded sub-blocks for predicting first sub-blocks of a first super-block using an intra-block copy (IntraBC) mode is identified. A patch from the first local search area is stored in a cache. The patch includes a reference block that is used for predicting at least one of the first sub-blocks. A second local search area for predicting second sub-blocks of a second super-block is identified. A determination is made whether to predict one of the second sub-blocks using a reference block from the second local search area or the patch from the cache. Determining whether to predict may include comparing prediction efficiencies between using the reference block from the second local search area and the patch from the cache, and predicting the one of the second sub-blocks using the patch from the cache based on the comparing.

Description

Atty. Doc. No. GOGL-2295-A-WO PATENTINTRA BLOCK COPY WITH LONG-TERM REFERENCINGCROSS-REFERENCE TO RELATED APPLICATION(S)

[0001] This application claims the benefit of and priority to U.S. Provisional Patent Application Serial No. 63 / 755,323, filed February 07, 2025, the entire disclosure of which is incorporated herein by reference.BACKGROUND

[0002] Digital video streams may represent video using a sequence of frames or still images. Digital video can be used for various applications including, for example, video conferencing, high definition video entertainment, video advertisements, or sharing of usergenerated videos. A digital video stream can contain a large amount of data and consume a substantial amount of computing or communication resources of a computing device for processing, transmission, or storage of the video data. Various approaches have been proposed to reduce the amount of data in video streams, including encoding or decoding techniques.SUMMARY

[0003] One aspect of the disclosed implementations relates to a method, including: identifying a first local search area including previously decoded sub-blocks for predicting first sub-blocks of a first super-block using an intra-block copy (IntraBC) mode; storing, in a cache, a patch from the first local search area, wherein the patch includes a reference block that is used for predicting at least one of the first sub-blocks; identifying a second local search area for predicting second sub-blocks of a second super-block; and determining whether to predict one of the second sub-blocks using a reference block from the second local search area or the patch from the cache.

[0004] In some implementations, the method includes tracking usage frequencies of reference blocks in the first local search area during prediction of the first sub-blocks.

[0005] In some implementations of the method, storing the patch includes: storing, in the cache, the patch corresponding to a highest frequency amongst the usage frequencies.

[0006] In some implementations, the method includes applying, after completing prediction of the first sub-blocks, a decay function to reduce the usage frequencies over time.{ YB:00561517.DOCX } -1-

[0007] In some implementations of the method, the decay function reduces each usage frequency by one quarter when moving from one super-block to a next super-block.

[0008] In some implementations, the method includes identifying clusters of block vectors pointing to regions in the first local search area, wherein storing the patch includes: determining a center location for each cluster; and storing the patch centered at the center location.

[0009] In some implementations of the method, identifying the clusters includes: applying a clustering algorithm based on spatial distances between block vector target locations.

[0010] In some implementations of the method, each patch in the cache includes a 32x32 pixel region extracted from a corresponding 64x64 sub-block.

[0011] In some implementations of the method, the patch is centered at a location determined by clustering block vectors that target similar regions within the corresponding 64x64 sub-block.

[0012] In some implementations of the method, the patch has a size smaller than a superblock size.

[0013] In some implementations of the method, the patch has a size of 32x32 pixels.

[0014] In some implementations of the method, determining whether to predict includes comparing prediction efficiencies between using the reference block from the second local search area and the patch from the cache; and predicting the one of the second sub-blocks using the patch from the cache based on the comparing.

[0015] In some implementations of the method, predicting using the patch from the cache includes: performing a search within the patch to identify an offset that minimizes a prediction error for the one of the second sub-blocks.

[0016] In some implementations of the method, predicting using the patch from the cache includes: searching within multiple entries in the cache to identify a stored patch and an offset that minimize a prediction error for the one of the second sub-blocks.

[0017] In some implementations of the method, determining whether to predict the one of the second sub-blocks using the reference block from the second local search area or the patch from the cache includes decoding, from a compressed bitstream, an index identifying the patch in the cache and an indication of an offset within the patch.

[0018] In some implementations of the method, the cache is limited to storing a predetermined number of patches.{ YB:00561517.DOCX } -2-

[0019] In some implementations of the method, the first super-block and the second super-block each have a size of 128x128 pixels.

[0020] In some implementations of the method, each patch in the cache is associated with a respective index.

[0021] In some implementations, the method includes applying warping parameters when matching between one of the second sub-blocks and patches in the cache for prediction.

[0022] In some implementations of the method, the warping parameters include at least one of scaling, zoom, or rotation.

[0023] In some implementations, the method includes inserting artificially generated entries into the cache.

[0024] In some implementations of the method, the artificially generated entries are generated using machine learning based on pre-analysis of video content.

[0025] In some implementations of the method, the first local search area includes two super-blocks left of the first super-block.

[0026] One aspect of the disclosed implementations relates to a method, including: maintaining a cache of reference patches extracted from previously coded portions of a current frame; tracking usage frequencies of block vectors within local search areas including previously coded super-blocks; adding a patch to the cache when a super-block of the previously coded super-blocks is no longer usable in subsequent local search areas, wherein the patch is copied from the super-block; applying a decay function to usage counts of cached patches after coding each super-block; and predicting portions of a current block using either block vectors referencing a local search area, or the cached patches stored in the cache with corresponding index and offset values.

[0027] In some implementations of the method, adding the patch includes: identifying clusters by applying a clustering algorithm to positions referenced by block vectors within the super-block; identifying respective patches for the clusters, wherein the respective patches are centered at respective center points of the respective clusters; and adding, as the patch, one of the respective patches to the cache based on cluster usage count.

[0028] In some implementations of the method, the decay function includes reducing the usage counts of the cached patches by one quarter after coding each super-block.

[0029] In some implementations of the method, adding the patch to the cache includes: adding the patch as a new patch when the cache is not full; or replacing one of the cached patches having lowest usage counts with the patch having higher initial counts.{ YB:00561517.DOCX } -3-

[0030] In some implementations, the method includes incrementing the usage counts of the cached patches when used for prediction; and maintaining the usage counts across superblock boundaries subject to the decay function.

[0031] In some implementations of the method, the local search area includes two superblocks left of a current super-block.

[0032] In some implementations of the method, each of the two super-blocks includes multiple 64x64 sub-blocks, and each extracted patch includes a 32x32 pixel region.

[0033] In some implementations, the method includes signaling in a bitstream whether prediction uses the local search area or the cache; for local area prediction, signaling a prediction block vector; and for cache-based prediction, signaling a cache index and offset.

[0034] These and other aspects of the present disclosure are disclosed in the following detailed description of the embodiments, the appended claims and the accompanying figures. It will be appreciated that aspects can be implemented in any convenient form. For example, aspects may be implemented by appropriate computer programs which may be carried on appropriate carrier media which may be tangible carrier media (e.g. disks) or intangible carrier media (e.g. communications signals). Aspects may also be implemented using suitable apparatus which may take the form of programmable computers running computer programs arranged to implement the methods and / or techniques disclosed herein. Aspects can be combined such that features described in the context of one aspect may be implemented in another aspect.BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The description herein makes reference to the accompanying drawings described below, wherein like reference numerals refer to like parts throughout the several views.

[0036] FIG. 1 is a schematic of a video encoding and decoding system.

[0037] FIG. 2 is a block diagram of an example of a computing device that can implement a transmitting station or a receiving station.

[0038] FIG. 3 is a diagram of a typical video stream to be encoded and subsequently decoded.

[0039] FIG. 4 is a block diagram of an encoder according to implementations of this disclosure.

[0040] FIG. 5 is a block diagram of a decoder according to implementations of this disclosure.{ YB:00561517.DOCX } -4-

[0041] FIG. 6 is a block diagram illustrating a conventional intra-block copy (IntraBC) mode.

[0042] FIGS. 7A-7B illustrate an example of another conventional IntraBC mode.

[0043] FIG. 8 illustrates a dictionary-based IntraBC coding technique with a cache-based long-term reference structure.

[0044] FIG. 9 is a flowchart of a technique for block prediction.

[0045] FIG. 10 is a flowchart of a technique for maintaining and using a cache of reference patches for IntraBC prediction.DETAILED DESCRIPTION

[0046] As mentioned above, compression schemes related to coding video streams may include breaking images (i.e., original or source images) into blocks and generating a digital video output bitstream using one or more techniques to limit the information included in the output. A received encoded bitstream can be decoded to re-create the blocks and the source images from the limited information. Encoding a video stream, or a portion thereof, such as a frame or a block, can include using temporal or spatial similarities in the video stream to improve coding efficiency. For example, a current block of a video stream may be encoded based on identifying a difference (residual) between previously coded pixel values and those in the current block. In this way, only the residual and parameters used to generate the residual need be added to the encoded bitstream. The residual may be encoded using a lossy quantization step. Decoding (i.e., reconstructing) an encoded block from such a residual often results in a distortion between the original and the reconstructed block.

[0047] Encoding using temporal similarities is known as inter prediction. Inter prediction uses a motion vector that represents the temporal displacement of a previously coded block relative to the current block. The motion vector can be identified using a method of motion estimation, such as a motion search. In the motion search, a portion of a reference frame can be translated to a succession of locations to form a predictor block that can be subtracted from a portion of a current frame to form a series of residuals. The horizontal and / or vertical translations corresponding to the location having, e.g., the smallest residual can be selected as the motion vector. The motion vector can be encoded in the encoded bitstream along with an indication of the reference frame.

[0048] Encoding using spatial similarities is known as intra prediction. Using an intraprediction mode, intra prediction can attempt to predict the pixel values of a current block of a current frame of a video stream using pixels peripheral to the current block. The pixels{ YB:00561517.DOCX } -5-peripheral to the current blocks are pixels within the current frame but that are outside the current block. The pixels peripheral to the block can be pixels adjacent to the current block. Which pixels peripheral to the block are used can depend on the intra-prediction mode and / or a scan order of the blocks of a frame. For example, in a raster scan order, peripheral pixels above a current block (i.e., the block being encoded or decoded) and / or peripheral pixels to the left of the current block may be used.

[0049] IntraBC is another technique that uses spatial similarities. IntraBC utilizes a block vector (BV) to obtain a predictor for a current block from a previously coded block of pixels in the same frame, with the BV specifying an x, y offset from the current block. As described with respect to FIGS. 6 and 7A-7B, global IntraBC enables referencing blocks from anywhere in the previously decoded frame area but requires fetching data from long-term memory. This leads to high bandwidth consumption and latency that makes hardware implementations costly and inefficient, particularly for devices with constrained memory architectures While local IntraBC reduces memory bandwidth by limiting the reference area to recently decoded blocks stored in a local cache (e.g., on-chip storage with limited space but faster access times), it fails to capture useful reference blocks once they leave the limited cache scope, resulting in missed opportunities for compression efficiency. Thus, hardware manufacturers prefer to eliminate global IntraBC entirely due to the high bandwidth and latency costs of accessing off-chip memory for hardware implementations with constrained memory architectures.

[0050] Implementations according to this disclosure solve these problems by caching frequently referenced blocks based on their usage patterns during local IntraBC coding. In some implementations, this caching can be implemented using a dictionary data structure that stores and indexes frequently used reference blocks for later prediction, where a dictionary refers to an indexed collection of pixel patches that can be accessed using an identifier rather than spatial coordinates.

[0051] When coding a current block using local IntraBC (which allows referencing blocks within the N (e.g., N=2) most recently decoded super-blocks to the left of the current super-block), block vector usage patterns are tracked to identify frequently referenced regions. A super-block refers to a largest possible coding block with a configurable size (e.g., 128x128, 256x256). In coded hardware implementations, patches centered around these frequently referenced regions are preserved in an on-chip cache after they would normally leave the local cache scope. These cached patches can be smaller than a super-block size (e.g., 32x32 or 64x64) and can serve as prediction references for subsequent blocks.{ YB:00561517.DOCX } -6-

[0052] A decay function gradually reduces the importance of older entries, allowing new frequently-used patterns to replace dictionary entries that are no longer actively referenced. Additionally, clustering techniques group adjacent patches with similar motion vectors to improve reference accuracy. This approach provides the coding efficiency benefits of having access to reference blocks beyond the local cache while maintaining low memory bandwidth through a size-limited dictionary cache.

[0053] In an alternative implementation, rather than maintaining a dictionary cache, the global IntraBC search range may be limited to one row of super-blocks above the current super-block. In this approach, memory usage is constrained to a row buffer that stores the immediately preceding row, reducing memory bandwidth while maintaining access to spatially proximate reference blocks.

[0054] Further details of techniques for intra-block copy with long-term referencing are described herein with initial reference to a system in which they can be implemented. FIG. 1 is a schematic of a video encoding and decoding system 100. A transmitting station 102 can be, for example, a computer having an internal configuration of hardware such as that described in FIG. 2. However, other implementations of the transmitting station 102 are possible. For example, the processing of the transmitting station 102 can be distributed among multiple devices.

[0055] A network 104 can connect the transmitting station 102 and a receiving station 106 for encoding and decoding of the video stream. Specifically, the video stream can be encoded in the transmitting station 102, and the encoded video stream can be decoded in the receiving station 106. The network 104 can be, for example, the Internet. The network 104 can also be a local area network (LAN), wide area network (WAN), virtual private network (VPN), cellular telephone network, or any other means of transferring the video stream from the transmitting station 102 to, in this example, the receiving station 106.

[0056] The receiving station 106, in one example, can be a computer having an internal configuration of hardware such as that described in FIG. 2. However, other suitable implementations of the receiving station 106 are possible. For example, the processing of the receiving station 106 can be distributed among multiple devices.

[0057] Other implementations of the video encoding and decoding system 100 are possible. For example, an implementation can omit the network 104. In another implementation, a video stream can be encoded and then stored for transmission at a later time to the receiving station 106 or any other device having memory. In one implementation, the receiving station 106 receives (e.g., via the network 104, a computer bus, and / or some{ YB:00561517.DOCX } -7-communication pathway) the encoded video stream and stores the video stream for later decoding. In an example implementation, a real-time transport protocol (RTP) is used for transmission of the encoded video over the network 104. In another implementation, a transport protocol other than RTP may be used (e.g., a Hypertext Transfer Protocol-based (HTTP-based) video streaming protocol).

[0058] When used in a video conferencing system, for example, the transmitting station 102 and / or the receiving station 106 may include the ability to both encode and decode a video stream as described below. For example, the receiving station 106 could be a video conference participant who receives an encoded video bitstream from a video conference server (e.g., the transmitting station 102) to decode and view and further encodes and transmits his or her own video bitstream to the video conference server for decoding and viewing by other participants.

[0059] FIG. 2 is a block diagram of an example of a computing device 200 that can implement a transmitting station or a receiving station. For example, the computing device 200 can implement one or both of the transmitting station 102 and the receiving station 106 of FIG. 1. The computing device 200 can be in the form of a computing system including multiple computing devices, or in the form of one computing device, for example, a mobile phone, a tablet computer, a laptop computer, a notebook computer, a desktop computer, and the like.

[0060] A processor 202 in the computing device 200 can be a conventional central processing unit. Alternatively, the processor 202 can be another type of device, or multiple devices, capable of manipulating or processing information now existing or hereafter developed. For example, although the disclosed implementations can be practiced with one processor as shown (e.g., the processor 202), advantages in speed and efficiency can be achieved by using more than one processor.

[0061] A memory 204 in computing device 200 can be a read only memory (ROM) device or a random access memory (RAM) device in an implementation. However, other suitable types of storage device can be used as the memory 204. The memory 204 can include code and data 206 that is accessed by the processor 202 using a bus 212. The memory 204 can further include an operating system 208 and application programs 210, the application programs 210 including at least one program that permits the processor 202 to perform the techniques described herein. For example, the application programs 210 can include applications 1 through N, which further include a video coding application that performs the techniques described herein. The computing device 200 can also include a secondary storage{ YB:00561517.DOCX } -8-214, which can, for example, be a memory card used with a mobile computing device.Because the video communication sessions may contain a significant amount of information, they can be stored in whole or in part in the secondary storage 214 and loaded into the memory 204 as needed for processing.

[0062] The computing device 200 can also include one or more output devices, such as a display 218. The display 218 may be, in one example, a touch sensitive display that combines a display with a touch sensitive element that is operable to sense touch inputs. The display 218 can be coupled to the processor 202 via the bus 212. Other output devices that permit a user to program or otherwise use the computing device 200 can be provided in addition to or as an alternative to the display 218. When the output device is or includes a display, the display can be implemented in various ways, including by a liquid crystal display (LCD), a cathode-ray tube (CRT) display, or a light emitting diode (LED) display, such as an organic LED (OLED) display.

[0063] The computing device 200 can also include or be in communication with an image-sensing device 220, for example, a camera, or any other image-sensing device 220 now existing or hereafter developed that can sense an image such as the image of a user operating the computing device 200. The image-sensing device 220 can be positioned such that it is directed toward the user operating the computing device 200. In an example, the position and optical axis of the image-sensing device 220 can be configured such that the field of vision includes an area that is directly adjacent to the display 218 and from which the display 218 is visible.

[0064] The computing device 200 can also include or be in communication with a soundsensing device 222, for example, a microphone, or any other sound-sensing device now existing or hereafter developed that can sense sounds near the computing device 200. The sound-sensing device 222 can be positioned such that it is directed toward the user operating the computing device 200 and can be configured to receive sounds, for example, speech or other utterances, made by the user while the user operates the computing device 200.

[0065] Although FIG. 2 depicts the processor 202 and the memory 204 of the computing device 200 as being integrated into one unit, other configurations can be utilized. The operations of the processor 202 can be distributed across multiple machines (wherein individual machines can have one or more processors) that can be coupled directly or across a local area or other network. The memory 204 can be distributed across multiple machines such as a network-based memory or memory in multiple machines performing the operations of the computing device 200. Although depicted here as one bus, the bus 212 of the{ YB:00561517.DOCX } -9-computing device 200 can be composed of multiple buses. Further, the secondary storage 214 can be directly coupled to the other components of the computing device 200 or can be accessed via a network and can comprise an integrated unit such as a memory card or multiple units such as multiple memory cards. The computing device 200 can thus be implemented in a wide variety of configurations.

[0066] FIG. 3 is a diagram of an example of a video stream 300 to be encoded and subsequently decoded. The video stream 300 includes a video sequence 302. At the next level, the video sequence 302 includes a number of adjacent frames 304. While three frames are depicted as the adjacent frames 304, the video sequence 302 can include any number of adjacent frames 304. The adjacent frames 304 can then be further subdivided into individual frames, for example, a frame 306. At the next level, the frame 306 can be divided into a series of planes or segments 308. The segments 308 can be subsets of frames that permit parallel processing, for example. The segments 308 can also be subsets of frames that can separate the video data into separate colors. For example, a frame 306 of color video data can include a luminance plane and two chrominance planes. The segments 308 may be sampled at different resolutions.

[0067] Whether or not the frame 306 is divided into segments 308, the frame 306 may be further subdivided into blocks 310, which can contain data corresponding to, for example, 16x16 pixels in the frame 306. The blocks 310 can also be arranged to include data from one or more segments 308 of pixel data. The blocks 310 can also be of any other suitable size such as 4x4 pixels, 8x8 pixels, 16x8 pixels, 8x16 pixels, 16x16 pixels, or larger. Unless otherwise noted, the terms block and macroblock are used interchangeably herein.

[0068] FIG. 4 is a block diagram of an encoder 400 according to implementations of this disclosure. The encoder 400 can be implemented, as described above, in the transmitting station 102, such as by providing a computer software program stored in memory, for example, the memory 204. The computer software program can include machine instructions that, when executed by a processor such as the processor 202, cause the transmitting station 102 to encode video data in the manner described in FIG. 4. The encoder 400 can also be implemented as specialized hardware included in, for example, the transmitting station 102. In one particularly desirable implementation, the encoder 400 is a hardware encoder.

[0069] The encoder 400 has the following stages to perform the various functions in a forward path (shown by the solid connection lines) to produce an encoded or compressed bitstream 420 using the video stream 300 as input: an intra / inter prediction stage 402, a transform stage 404, a quantization stage 406, and an entropy encoding stage 408. The{ YB:00561517.DOCX } -10-encoder 400 may also include a reconstruction path (shown by the dotted connection lines) to reconstruct a frame for encoding of future blocks. In FIG. 4, the encoder 400 has the following stages to perform the various functions in the reconstruction path: a dequantization stage 410, an inverse transform stage 412, a reconstruction stage 414, and a loop filtering stage 416. Other structural variations of the encoder 400 can be used to encode the video stream 300.

[0070] When the video stream 300 is presented for encoding, respective adjacent frames 304, such as the frame 306, can be processed in units of blocks. At the intra / inter prediction stage 402, respective blocks can be encoded using intra-frame prediction (also called intraprediction) or inter- frame prediction (also called inter-prediction). In any case, a prediction block can be formed. In the case of intra-prediction, a prediction block may be formed from samples in the current frame that have been previously encoded and reconstructed. In the case of inter-prediction, a prediction block may be formed from samples in one or more previously constructed reference frames.

[0071] Next, the prediction block can be subtracted from the current block at the intra / inter prediction stage 402 to produce a residual block (also called a residual). The transform stage 404 transforms the residual into transform coefficients in, for example, the frequency domain using block-based transforms. The quantization stage 406 converts the transform coefficients into discrete quantum values, which are referred to as quantized transform coefficients, using a quantizer value or a quantization level. For example, the transform coefficients may be divided by the quantizer value and truncated.

[0072] The quantized transform coefficients are then entropy encoded by the entropy encoding stage 408. The entropy-encoded coefficients, together with other information used to decode the block (which may include, for example, syntax elements such as used to indicate the type of prediction used, transform type, motion vectors, a quantizer value, or the like), are then output to the compressed bitstream 420. The compressed bitstream 420 can be formatted using various techniques, such as variable length coding (VLC) or arithmetic coding. The compressed bitstream 420 can also be referred to as an encoded video stream or encoded video bitstream, and the terms will be used interchangeably herein.

[0073] The reconstruction path (shown by the dotted connection lines) can be used to ensure that the encoder 400 and a decoder 500 (described below with respect to FIG. 5) use the same reference frames to decode the compressed bitstream 420. The reconstruction path performs functions that are similar to functions that take place during the decoding process (described below with respect to FIG. 5), including dequantizing the quantized transform{ YB:00561517.DOCX } -11-coefficients at the dequantization stage 410 and inverse transforming the dequantized transform coefficients at the inverse transform stage 412 to produce a derivative residual block (also called a derivative residual). At the reconstruction stage 414, the prediction block that was predicted at the intra / inter prediction stage 402 can be added to the derivative residual to create a reconstructed block. The loop filtering stage 416 can be applied to the reconstructed block to reduce distortion such as blocking artifacts.

[0074] Other variations of the encoder 400 can be used to encode the compressed bitstream 420. In some implementations, a non-transform based encoder can quantize the residual signal directly without the transform stage 404 for certain blocks or frames. In some implementations, an encoder can have the quantization stage 406 and the dequantization stage 410 combined in a common stage.

[0075] FIG. 5 is a block diagram of a decoder 500 according to implementations of this disclosure. The decoder 500 can be implemented in the receiving station 106, for example, by providing a computer software program stored in the memory 204. The computer software program can include machine instructions that, when executed by a processor such as the processor 202, cause the receiving station 106 to decode video data in the manner described in FIG. 5. The decoder 500 can also be implemented in hardware included in, for example, the transmitting station 102 or the receiving station 106.

[0076] The decoder 500, similar to the reconstruction path of the encoder 400 discussed above, includes in one example the following stages to perform various functions to produce an output video stream 516 from the compressed bitstream 420: an entropy decoding stage 502, a dequantization stage 504, an inverse transform stage 506, an intra / inter prediction stage 508, a reconstruction stage 510, a loop filtering stage 512, and a deblocking filtering stage 514. Other structural variations of the decoder 500 can be used to decode the compressed bitstream 420.

[0077] When the compressed bitstream 420 is presented for decoding, the data elements within the compressed bitstream 420 can be decoded by the entropy decoding stage 502 to produce a set of quantized transform coefficients. The dequantization stage 504 dequantizes the quantized transform coefficients (e.g., by multiplying the quantized transform coefficients by the quantizer value), and the inverse transform stage 506 inverse transforms the dequantized transform coefficients to produce a derivative residual that can be identical to that created by the inverse transform stage 412 in the encoder 400. Using header information decoded from the compressed bitstream 420, the decoder 500 can use the intra / inter{ YB:00561517.DOCX } -12-prediction stage 508 to create the same prediction block as was created in the encoder 400 (e.g., at the intra / inter prediction stage 402).

[0078] At the reconstruction stage 510, the prediction block can be added to the derivative residual to create a reconstructed block. The loop filtering stage 512 can be applied to the reconstructed block to reduce blocking artifacts. Other filtering can be applied to the reconstructed block. In this example, the deblocking filtering stage 514 is applied to the reconstructed block to reduce blocking distortion, and the result is output as the output video stream 516. The output video stream 516 can also be referred to as a decoded video stream, and the terms will be used interchangeably herein. Other variations of the decoder 500 can be used to decode the compressed bitstream 420. In some implementations, the decoder 500 can produce the output video stream 516 without the deblocking filtering stage 514.

[0079] FIG. 6 is a block diagram illustrating a conventional IntraBC mode. FIG. 6 illustrates a current frame 600 being coded and a current block 602 within the current frame 600. It is also noted that the IntraBC mode is limited to intra-coded frames.

[0080] In this IntraBC mode, an encoder (e.g., the encoder 400 of FIG. 4) searches at least a subset of a reconstructed area 603 of the current frame 600 to identify a reference block (e.g., a reference block 604) that best matches the current block 602. The reference block 604 is used as a prediction block for the current block 602. A difference (i.e., residual) between the current block 602 and the reference block 604 is encoded in a compressed bitstream (e.g., the compressed bitstream 420 of FIG. 4) along with a BV, such as a block vector 606. As such, IntraBC can be compared to “motion compensation”, but within the same frame (e.g., the current frame 600), where a BV is used instead of a motion vector. The encoder encodes a flag (e.g., an IntraBC flag) indicating that the current block 602 is encoded using the IntraBC mode and also encodes the BV.

[0081] When decoding the current block 602, a decoder, such as the decoder 500 of FIG.5, decodes the IntraBC flag to determine whether the current block 602 is to be decoded using the IntraBC mode. If so, then the decoder decodes the block vector 606 from the compressed bitstream to identify the reference block 604 therewith obtaining the prediction block for the current block 602. A residual block can then be decoded from the compressed bitstream to add to the prediction block therewith reconstructing the current block 602.

[0082] A problem with the conventional IntraBC mode described with respect to FIG. 6 is that accessing reference blocks from previously decoded regions requires high memory bandwidth due to fetching data from off-chip memory. This high memory bandwidth{ YB:00561517.DOCX } -13-consumption leads to increased latency that makes hardware implementations costly and inefficient, particularly for devices with constrained memory architectures.

[0083] FIGS. 7A-7B illustrate an example of another conventional IntraBC mode. FIG.7A illustrates global and local reference areas for this IntraBC mode. Frame 702 includes super-blocks which include sub-blocks and an uncoded / not decoded area 704. As depicted, super-blocks are 128x128 and sub-blocks are 64x64, but any size blocks may be used. The legend of FIG. 7A indicates how super-blocks 712, sub-blocks 714, global reference area 722, local reference area / current block 724, and unavailable areas 726 are depicted.

[0084] In frame 702, there are multiple current blocks because intra coding can be performed concurrently on a row-by-row basis once sufficient blocks from prior row(s) have been coded / decoded to perform intra prediction. On a row basis, the prior two super-blocks that have been coded / decoded are identified as unavailable due to the delay in storing coded / decoded blocks in non-cache memory. Depending on the implementation, the number of prior blocks that are unavailable may vary.

[0085] The sub-blocks that are identified as a current block are also available as a local reference area. For example, intra coding may be performed on a 4x4 block basis and previously coded 4x4 blocks in the sub-block may be available for prediction using intra BC. Blocks denoted as within the global reference area 722 may also be used for prediction using Intra BC. Blocks that are within an unavailable area 726 or are in the have not yet been coded / decoded area 704 are not available for prediction using intra BC.

[0086] FIG. 7B is a block diagram illustrating an extended local reference area for an IntraBC mode. The illustration of FIG. 7B may be understood as an extension of the IntraBC mode illustrated in FIG. 7A. FIG. 7B includes illustrations of an extended local reference area for a 256x256 super-block and a 128x128 super-block. Specifically, super-blocks 752 illustrate an application of an extended local reference area for 256x256 super-blocks and super-blocks 756 illustrate an application of an extended local reference area for 128x128 super-blocks. In both illustrations, sub-blocks are illustrated as 64x64.

[0087] The sub-blocks within the extended local reference area are identified based on respective super-block indices 754, 758 and a pre- determined mapping between the index of the current block (as identified according to the identification of the current block 760) and reference sub-block indices. For example, the pre- determined mapping may take the form of a reference table known to both the encoder and decoder, such as the tables below:{ YB:00561517.DOCX } -14-TABLE ITABLE II

[0088] In the foregoing example tables, Table I refers to an example pre-determined mapping for 128x128 super-blocks and Table II refers to an example pre-determined mapping for 256x256 super-blocks.{ YB:00561517.DOCX } -15-

[0089] Variations on the illustrated super-block sizes, sub-block sizes, super-block indices, and pre-determined mappings are possible depending on the implementation. For example, in some implementations, the pre-determined mappings map a current block index to a different number of reference blocks (e.g., other than 8).

[0090] Performing IntraBC with extended reference area would include identifying available local reference sub-blocks for a current sub-block based on a pre- determined mapping between a current sub-block index and reference sub-block indices. Examples of identified available local reference sub-blocks are illustrated in FIG. 7B as local reference area 762. FIG. 7B also illustrates unavailable areas 764. Examples of implementing a predetermined mapping using a reference table are illustrated above. When a reference table is used, it may include rows mapping each sub-block index of a current super-block to reference sub-block indices corresponding to the current super-block and at least one prior super-block in raster order. In some implementations, there are eight reference sub-block indices corresponding to each sub-block index. In some implementations, the current sub-block is in a current super-block and the pre-determined mapping is based on a size of the current superblock.

[0091] As shown with respect to FIG. 7B, an implementation may have different predetermined mappings for 128x128 and 256x256 super-blocks. As shown with respect to FIG.7B, an implementation may have current sub-block and available local reference sub-blocks that are 64x64. In such implementations utilizing 128x128 super-blocks, the current superblock and two previously coded super-blocks in the vicinity of the current super-block each include at least one of the available local reference sub-blocks. Different block sizes are possible depending on the implementation.

[0092] The pixel values of the current sub-block are then predicted based on a block vector and pixel values of the identified available local reference sub-blocks. For example, with respect to FIG. 7B, pixel values in a current block 760 may be predicted from pixel values in the local reference area 762 that are identified using a block vector. The block vector may have single-pixel precision, sub-pixel precision, or a combination thereof. When sub-pixel precision is utilized, for example, the block vector refers to a sub-pixel position in one of the identified available local reference sub-blocks. A precision syntax element may be coded or decoded indicating whether to utilize a 2-pixel, 1 -pixel, or G-pixel precision followed by a block vector that is coded / decoded based on the precision syntax element value. Different interpolation filters, such as bicubic or bilinear may be utilized to generate the interpolated pixels.{ YB:00561517.DOCX } -16-

[0093] Performing IntraBC with extended reference area would include identifying available local reference sub-blocks for a current sub-block based on a pre- determined mapping between a current sub-block index and reference sub-block indices. Examples of identified available local reference sub-blocks are illustrated in FIG. 7B as local reference area 762. Examples of implementing a pre-determined mapping using a reference table are illustrated above. When a reference table is used, it may include rows mapping each subblock index of a current super-block to reference sub-block indices corresponding to the current super-block and at least one prior super-block in raster order.

[0094] A problem with the IntraBC mode described with respect to FIGS. 7A-7B is that while local IntraBC reduces bandwidth by restricting references to recently decoded blocks, accessing reference blocks outside the local cache scope still requires costly off-chip memory access. This approach thus continues to face bandwidth and latency challenges for hardware implementations with constrained memory architectures when good prediction candidates exist beyond the local cache area.

[0095] FIG. 8 illustrates a dictionary-based IntraBC coding technique with a cache-based long-term reference structure. FIG. 8 includes a portion of a frame 800 partitioned into rows (e.g., a row 802) of super-blocks, including super-blocks 804, 806, 808, and 810, where super-block 804 is currently being coded. An encoder and a decoder process super-blocks in a raster scan order. Even though the term "frame" is used, the teachings apply equally to the processing of rows of super-blocks of tiles.

[0096] FIG. 8 illustrates that the super-block 804 is currently being coded. A window 812 (a sliding window) defines a local search area containing previously decoded sub-blocks available for predicting portions of the super-block 804 using IntraBC mode. The local search area typically includes two super-blocks to the left of the current super-block, though implementations may vary in the size and arrangement of the search area. Sub-blocks 814, 815, and 816 within super-block 804 demonstrate different prediction sources, indicated by block vectors 817, 818, and 820, respectively. The sub-blocks 814 and 815 use references from the local search area (e.g., the window 812), while the sub-block 816 uses a patch from a cache 826 (further described below).

[0097] When encoding a current sub-block, the encoder may employ a structured search process. First, the encoder may search within the local area using block matching techniques such as sum of absolute differences (SAD) or sum of squared differences (SSD).Additionally, the encoder searches patches stored in the cache 826, selecting the best match based on rate-distortion optimization (RDO). In some implementations, the search may also{ YB:00561517.DOCX } -17-incorporate sub-pixel refinement, where interpolation filters (e.g., bilinear or bicubic) are used to increase matching accuracy. Additionally, hierarchical motion estimation techniques may be employed to accelerate the search in larger super-blocks. The encoder may signal (via a compressed bitstream, such as the compressed bitstream 420 of FIG. 4) to a decoder: (1) whether IntraBC mode is used, (2) whether prediction uses the local search area or cache, (3) a block vector for local area prediction, or (4) a cache index and offset for cache-based prediction. This information enables accurate reconstruction at the decoder.

[0098] Cache 826 transitions to updated cache 826' after coding super-block 804 is completed. That is, the cache 826 is updated to be the updated cache 826'. The update process involves several steps.

[0099] First, a decay function reduces counts of existing cache entries, if not used. As illustrated, a count of an entry 828A decreases from 5 to 4 due to non-usage; and a count of an entry 828B is increased from 4 to 5, reflecting its use during coding. The decay function may implement various strategies including, but not limited to, linear reduction by a fixed fraction (e.g., one-quarter per super-block), exponential decay over time, or threshold-based removal after periods of non-use.

[0100] Second, a clustering analysis is used to identify new cache entries. BVs that point to previously coded super-blocks that will no longer be included within local search areas for subsequent super-blocks are used in the clustering, while block vectors referencing superblocks that remain available for subsequent predictions are not included in the clustering process. As such, in this example, BVs that point to the super-block 808, which exits (e.g., is no longer within) the local search area, are used in the clustering. A K- means clustering algorithm groups BVs based on their spatial distances, identifying frequently referenced regions where multiple BVs point to adjacent areas. For each identified cluster, the algorithm determines a center point based on the spatial distribution of the BVs in that cluster, and a patch is centered at this point. As illustrated in FIG. 8, the clustering algorithm resulted in the identification of patches 822 and 824.

[0101] In some implementations, the center point for a dictionary entry may be dynamically adjusted. For example, for each use of a local IntraBC block vector, the block vector may be added to a moving average, and the final moving average may be used as a reference vector to determine the center point for the patch to be stored in the cache.

[0102] Each cluster's usage count is determined by the number of BVs that reference that region. BVs that reference the super-block 810, which remains available for the next superblock's prediction, are not used in the clustering. These BVs targeting the super-block 810 are{ YB:00561517.DOCX } -18-retained for future clustering analysis when that super-block exits the local search area. A new entry 828C represents a patch selected from super-block 808 based on clustering results.

[0103] Implementations may support various patch configurations including, but not limited to, fixed 32x32 patches from 64x64 sub-blocks, multiple patch sizes (16x16, 32x32, 64x64), dynamic size selection based on content characteristics. Additionally, various cache organizations may be available including, but not limited to, fixed number of entries (e.g., 16 entries of 32x32 patches), mixed entry sizes, and priority-based organization by usage frequency. Additionally, different clustering approaches may be used, including: hierarchical clustering, density-based clustering, or custom distance metrics.

[0104] In some implementations, patches may be aligned with predefined partitions within a super-block in the local search area. For example, a 128x128 super-block may be partitioned into 16 non-overlapping 32x32 regions that align with potential patch boundaries. When the clustering algorithm identifies frequently referenced areas within the super-block, the resulting patches may be selected to coincide with these predefined partitions rather than being arbitrarily centered.

[0105] The cache implementation may employ various replacement strategies when adding new entries. In some implementations, a least recently used approach may remove entries that have not been referenced for extended periods. In some implementations, entries with the lowest usage counts may be replaced when new patches demonstrate higher prediction utility. For example, a candidate patch from the local search area may be selected for addition to the cache, and the candidate patch may have an initial usage count. If the cache is full, the candidate patch may replace a patch in the cache if the initial usage count of the candidate patch is higher than the current usage count of the patch in the cache having the lowest usage count. In some implementations, the replacement decision may consider multiple factors including usage frequency, prediction quality, and pattern repetition characteristics. In some implementations, when multiple patches have equal usage counts, the system may retain the older entry that is already stored in the cache, thereby providing a stable tie-breaking rule that favors established dictionary entries over newer candidates with equivalent usage.

[0106] FIG. 9 is a flowchart of a technique 900 for block prediction. The technique 900 can be implemented, for example, as a software program that may be executed by computing devices such as transmitting station 102 or receiving station 106. The software program can include machine-readable instructions that may be stored in a non-transitory computer-readable storage medium, such as the memory 204 or the secondary storage 214, and that,{ YB:00561517.DOCX } -19-when executed by a processor, such as processor 202, may cause the computing device to perform the technique 900. The technique 900 may be implemented in whole or in part in the intra / inter prediction stage 402 of the encoder 400 of FIG. 4 and / or the intra / inter prediction stage 508 of the decoder 500 of FIG. 5. When implemented by an encoder, coding means encoding, as described with respect to FIG. 4; and when implemented by a decoder, coding means decoding, as described with respect to FIG. 5. The technique 900 can be implemented using specialized hardware or firmware. Multiple processors, memories, or both, may be used.

[0107] At 902, the technique 900 includes identifying a first local search area comprising previously decoded sub-blocks for predicting first sub-blocks of a first super-block using IntraBC mode. For example, the encoder 400 or decoder 500 may identify a local search area that includes two super-blocks to the left of a current super-block being coded. In some implementations, each super-block may have a size of 128x128 pixels.

[0108] During the prediction process, the technique 900 may track usage frequencies of reference blocks in the first local search area during prediction of the first sub-blocks.Clusters of block vectors pointing to regions in the first local search area can be identified by applying a clustering algorithm (e.g., a K-means clustering algorithm) based on spatial distances between block vector target locations. In some implementations, the clustering process may determine a center location for each cluster to guide patch selection.

[0109] At 904, the technique 900 includes storing, in a cache, a patch from the first local search area. The patch includes a reference block that is used for predicting at least one of the first sub-blocks. For example, the encoder 400 may store, in the cache, the patch corresponding to a highest frequency amongst the usage frequencies. In some implementations, each patch in the cache may be a 32x32 pixel region extracted from a corresponding 64x64 sub-block, where the patch is centered at a location determined by clustering block vectors that target similar regions within the corresponding 64x64 sub-block.

[0110] The cache may be limited to storing a predetermined number of patches, where each patch is associated with a respective index. After completing prediction of the first subblocks, the technique 900 may apply a decay function to reduce the usage frequencies over time. In some implementations, the decay function reduces each usage frequency by one quarter when moving from one super-block to a next super-block.

[0111] At 906, the technique 900 includes identifying a second local search area for predicting second sub-blocks of a second super-block. For example, as the encoding or{ YB:00561517.DOCX } -20-decoding process moves to a new super-block, the encoder 400 or decoder 500 may update the local search area to include newly available reference blocks.

[0112] At 908, the technique 900 includes determining whether to predict one of the second sub-blocks using a reference block from the second local search area or the patch from the cache. This determination may involve comparing prediction efficiencies between using the reference block from the second local search area and the patch from the cache.

[0113] When predicting using the patch from the cache, the technique 900 may perform a search within the patch to identify an offset that minimizes a prediction error for the one of the second sub-blocks. Additionally, the technique 900 may search within multiple entries (e.g., patches) in the cache to identify a stored patch and an offset that minimize a prediction error.

[0114] In some implementations, determining whether to predict the second sub-blocks may include decoding, from a compressed bitstream, an index identifying the patch in the cache and an indication of an offset within the patch. In some implementations, the technique 900 may also apply warping parameters when matching between a current block and patches in the cache for prediction, where the warping parameters may include scaling, zoom, or rotation.

[0115] In some implementations, the cache may include artificially generated entries that are not extracted from previously coded portions of the frame. These entries may be generated using machine learning models trained on video content characteristics. For example, a pre-analysis stage may identify common patterns, textures, or recurring visual elements in the video content, and a machine learning model may generate representative patches based on this analysis. The artificially generated entries may be signaled to the decoder, or both the encoder and decoder may independently generate matching entries using the same generation process, thereby maintaining synchronization without explicit signaling. As such, artificially generated entries may be inserted into the cache.

[0116] FIG. 10 is a flowchart of a technique 1000 for maintaining and using a cache of reference patches for IntraBC prediction. The technique 1000 can be implemented, for example, as a software program that may be executed by computing devices such as transmitting station 102 or receiving station 106. The software program can include machine-readable instructions that may be stored in a non-transitory computer-readable storage medium, such as the memory 204 or the secondary storage 214, and that, when executed by a processor, such as processor 202, may cause the computing device to perform the technique 1000. The technique 1000 may be implemented in whole or in part in{ YB:00561517.DOCX } -21-the intra / inter prediction stage 402 of the encoder 400 of FIG. 4 and / or the intra / inter prediction stage 508 of the decoder 500 of FIG. 5. When implemented by an encoder, coding means encoding, as described with respect to FIG. 4; and when implemented by a decoder, coding means decoding, as described with respect to FIG. 5. The technique 1000 can be implemented using specialized hardware or firmware. Multiple processors, memories, or both, may be used.

[0117] At 1002, the technique 1000 includes maintaining a cache of reference patches extracted from previously coded portions of a current frame. For example, the encoder 400 or decoder 500 may maintain a dictionary cache of frequently used reference patches that can be accessed during prediction of subsequent blocks. At 1004, the technique 1000 includes tracking usage frequencies of block vectors within local search areas comprising previously coded super-blocks. In some implementations, the local search area may include at least two super-blocks left of a current super-block, where each of the two super-blocks includes multiple 64x64 sub-blocks, and each extracted patch is a 32x32 pixel region.

[0118] At 1006, the technique 1000 includes adding a patch to the cache when a superblock of the previously coded super-blocks is no longer usable in subsequent local search areas, wherein the patch is copied from the super-block. The process of adding patches may involve identifying clusters by applying a clustering algorithm to positions referenced by block vectors within the super-block, identifying respective patches centered at respective center points of the respective clusters, and adding patches based on cluster usage count (e.g., cluster density). That is, a patch corresponding to a cluster referenced by a higher number of block vectors is prioritized for addition to the cache.

[0119] When adding patches to the cache, the technique 1000 may either add the patch as a new patch when the cache is not full or replace cached patches having lowest usage counts with new patches having higher initial counts. The technique 1000 may increment usage counts of cached patches when they are used for prediction and maintain these counts across super-block boundaries.

[0120] In some implementations, rather than automatically registering patches based on IntraBC usage, a flag may be signaled in the bitstream for each coding unit to indicate whether that block should be registered to the dictionary. This approach provides explicit control over dictionary contents regardless of the prediction mode used for that block. In some implementations, the flag may be limited to blocks encoded using specific prediction modes, such as excluding blocks encoded using DC prediction or directional prediction modes, thereby reducing signaling overhead.{ YB:00561517.DOCX } -22-

[0121] At 1008, the technique 1000 includes applying a decay function to usage counts of cached patches after coding each super-block. In some implementations, the decay function may reduce the usage counts of cached patches by one quarter after coding each super-block. At 1010, the technique 1000 includes predicting portions of a current block using either block vectors referencing a local search area, or the cached patches stored in the cache with corresponding index and offset values. The system may signal in a bitstream whether prediction uses the local search area or the cache.

[0122] When using local area prediction, the system may signal a prediction block vector in the bitstream. Alternatively, for cache-based prediction, the system may signal a cache index and offset. This signaling enables the decoder to properly reconstruct the predicted block using either the local search area or cached patches.

[0123] In some implementations, the cache may be organized to align with predefined partitions within super-blocks, facilitating more efficient storage and retrieval of patches. The system may also support various patch configurations, including multiple patch sizes and dynamic size selection based on content characteristics.

[0124] For simplicity of explanation, the techniques 900 and 1000 of FIGS. 9 and 10, respectively, are each depicted and described as a respective series of steps or operations. However, the steps or operations in accordance with this disclosure can occur in various orders and / or concurrently. Additionally, other steps or operations not presented and described herein may be used. Furthermore, not all illustrated steps or operations may be required to implement a technique in accordance with the disclosed subject matter.

[0125] The aspects of encoding and decoding described above illustrate some examples of encoding and decoding techniques. However, it is to be understood that encoding and decoding, as those terms are used in the claims, could mean compression, decompression, transformation, or any other processing or change of data.

[0126] The word “example” is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as “example” is not necessarily to be construed as being preferred or advantageous over other aspects or designs. Rather, use of the word “example” is intended to present concepts in a concrete fashion. As used in this application, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless specified otherwise or clearly indicated otherwise by the context, the statement “X includes A or B” is intended to mean any of the natural inclusive permutations thereof. That is, if X includes A; X includes B; or X includes both A and B, then “X includes A or B” is satisfied under any of the foregoing instances. In addition, the articles “a” and “an” as used{ YB:00561517.DOCX } -23-in this application and the appended claims should generally be construed to mean “one or more,” unless specified otherwise or clearly indicated by the context to be directed to a singular form. Moreover, use of the term “an implementation” or the term “one implementation” throughout this disclosure is not intended to mean the same embodiment or implementation unless described as such.

[0127] Implementations of the transmitting station 102 and / or the receiving station 106 (and the algorithms, methods, instructions, etc., stored thereon and / or executed thereby, including by the encoder 400 and the decoder 500) can be realized in hardware, software, or any combination thereof. The hardware can include, for example, computers, intellectual property (IP) cores, application- specific integrated circuits (ASICs), programmable logic arrays, optical processors, programmable logic controllers, microcode, microcontrollers, servers, microprocessors, digital signal processors, or any other suitable circuit. In the claims, the term “processor” should be understood as encompassing any of the foregoing hardware, either singly or in combination. The terms “signal” and “data” are used interchangeably. Further, portions of the transmitting station 102 and the receiving station 106 do not necessarily have to be implemented in the same manner.

[0128] Further, in one aspect, for example, the transmitting station 102 or the receiving station 106 can be implemented using a general purpose computer or general purpose processor with a computer program that, when executed, carries out any of the respective methods, algorithms, and / or instructions described herein. In addition, or alternatively, for example, a special purpose computer / processor can be utilized which can contain other hardware for carrying out any of the methods, algorithms, or instructions described herein.

[0129] The transmitting station 102 and the receiving station 106 can, for example, be implemented on computers in a video conferencing system. Alternatively, the transmitting station 102 can be implemented on a server, and the receiving station 106 can be implemented on a device separate from the server, such as a handheld communications device. In this instance, the transmitting station 102, using an encoder 400, can encode content into an encoded video signal and transmit the encoded video signal to the communications device. In turn, the communications device can then decode the encoded video signal using a decoder 500. Alternatively, the communications device can decode content stored locally on the communications device, for example, content that was not transmitted by the transmitting station 102. Other suitable transmitting and receiving implementation schemes are available. For example, the receiving station 106 can be a{ YB:00561517.DOCX } -24-generally stationary personal computer rather than a portable communications device, and / or a device including an encoder 400 may also include a decoder 500.

[0130] Further, all or a portion of implementations of the present disclosure can take the form of a computer program product accessible from, for example, a computer-usable or computer-readable medium. A computer-usable or computer-readable medium can be any device that can, for example, tangibly contain, store, communicate, or transport the program for use by or in connection with any processor (that is, the computer-readable medium can be a non-transitory computer-readable storage medium). The medium can be, for example, an electronic, magnetic, optical, electromagnetic, or semiconductor device. Other suitable mediums are also available.

[0131] The above-described embodiments, implementations, and aspects have been described in order to facilitate easy understanding of this disclosure and do not limit this disclosure. On the contrary, this disclosure is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims, which scope is to be accorded the broadest interpretation as is permitted under the law so as to encompass all such modifications and equivalent arrangements.{ YB:00561517.DOCX } -25-

Claims

What is claimed is:

1. A method, comprising:identifying a first local search area comprising previously decoded sub-blocks for predicting first sub-blocks of a first super-block using an intra-block copy (IntraBC) mode;storing, in a cache, a patch from the first local search area, wherein the patch includes a reference block that is used for predicting at least one of the first sub-blocks;identifying a second local search area for predicting second sub-blocks of a second super-block; anddetermining whether to predict one of the second sub-blocks using a reference block from the second local search area or the patch from the cache.

2. The method of claim 1, further comprising:tracking usage frequencies of reference blocks in the first local search area during prediction of the first sub-blocks.

3. The method of claim 2, wherein storing the patch comprises:storing, in the cache, the patch corresponding to a highest frequency amongst the usage frequencies.

4. The method of claim 2 or claim 3, further comprising:applying, after completing prediction of the first sub-blocks, a decay function to reduce the usage frequencies over time.

5. The method of claim 4, wherein the decay function reduces each usage frequency by one quarter when moving from one super-block to a next super-block.

6. The method of any one of claim 1 to claim 5, further comprising: identifying clusters of block vectors pointing to regions in the first local search area, wherein storing the patch comprises:determining a center location for each cluster; andstoring the patch centered at the center location.

7. The method of claim 6, wherein identifying the clusters comprises:{ YB:00561517.DOCX } -26-applying a clustering algorithm based on spatial distances between block vector target locations.

8. The method of claim 6, wherein each patch in the cache comprises a 32x32 pixel region extracted from a corresponding 64x64 sub-block.

9. The method of claim 8, wherein the patch is centered at a location determined by clustering block vectors that target similar regions within the corresponding 64x64 subblock.

10. The method of claim 6, wherein the patch has a size smaller than a super-block size.

11. The method of claim 10, wherein the patch has a size of 32x32 pixels.

12. The method of any one of claim 1 to claim 11, wherein determining whether to predict comprises:comparing prediction efficiencies between using the reference block from the second local search area and the patch from the cache; andpredicting the one of the second sub-blocks using the patch from the cache based on the comparing.

13. The method of claim 12, wherein predicting using the patch from the cache comprises:performing a search within the patch to identify an offset that minimizes a prediction error for the one of the second sub-blocks.

14. The method of claim 12, wherein predicting using the patch from the cache comprises:searching within multiple entries in the cache to identify a stored patch and an offset that minimize a prediction error for the one of the second sub-blocks.{ YB:00561517.DOCX } -27-15. The method of any one of claim 1 to claim 14, wherein determining whether to predict the one of the second sub-blocks using the reference block from the second local search area or the patch from the cache comprises:decoding, from a compressed bitstream, an index identifying the patch in the cache and an indication of an offset within the patch.

16. The method of any one of claim 1 to claim 15, wherein the cache is limited to storing a predetermined number of patches.

17. The method of any one of claim 1 to claim 16, wherein the first super-block and the second super-block each have a size of 128x128 pixels.

18. The method of any one of claim 1 to claim 17, wherein each patch in the cache is associated with a respective index.

19. The method of any one of claim 1 to claim 18, further comprising: applying warping parameters when matching between one of the second sub-blocks and patches in the cache for prediction.

20. The method of claim 19, wherein the warping parameters include at least one of scaling, zoom, or rotation.

21. The method of any one of claim 1 to claim 20, further comprising: inserting artificially generated entries into the cache.

22. The method of claim 21, wherein the artificially generated entries are generated using machine learning based on pre-analysis of video content.

23. The method of any one of claim 1 to claim 22, wherein the first local search area comprises two super-blocks left of the first super-block.

24. A method, comprising:maintaining a cache of reference patches extracted from previously coded portions of a current frame;{ YB:00561517.DOCX } -28-tracking usage frequencies of block vectors within local search areas comprising previously coded super-blocks;adding a patch to the cache when a super-block of the previously coded super-blocks is no longer usable in subsequent local search areas, wherein the patch is copied from the super-block;applying a decay function to usage counts of cached patches after coding each superblock; andpredicting portions of a current block using either block vectors referencing a local search area, or the cached patches stored in the cache with corresponding index and offset values.

25. The method of claim 24, wherein adding the patch comprises: identifying clusters by applying a clustering algorithm to positions referenced by block vectors within the super-block;identifying respective patches for the clusters, wherein the respective patches are centered at respective center points of the respective clusters; andadding, as the patch, one of the respective patches to the cache based on cluster usage count.

26. The method of any one of claim 24 to claim 25, wherein the decay function comprises reducing the usage counts of the cached patches by one quarter after coding each super-block.

27. The method of any one of claim 24 to claim 26, wherein adding the patch to the cache comprises:adding the patch as a new patch when the cache is not full; orreplacing one of the cached patches having lowest usage counts with the patch having higher initial counts.

28. The method of any one of claim 24 to claim 27, further comprising: incrementing the usage counts of the cached patches when used for prediction; and maintaining the usage counts across super-block boundaries subject to the decay function.{ YB:00561517.DOCX } -29-29. The method of any one of claim 24 to claim 28, wherein the local search area comprises two super-blocks left of a current super-block.

30. The method of claim 29, wherein each of the two super-blocks comprises multiple 64x64 sub-blocks, and each extracted patch comprises a 32x32 pixel region.

31. The method of any one of claim 24 to claim 30, further comprising: signaling in a bitstream whether prediction uses the local search area or the cache; for local area prediction, signaling a prediction block vector; andfor cache-based prediction, signaling a cache index and offset.

32. A device, comprising:a processor that is configured to perform the method of any one of claims 1 to 31.

33. A device, comprising:a memory; anda processor, the processor configured to execute instructions stored in the memory to perform the method of any one of claims 1 to 31.

34. A non-transitory computer-readable storage medium, comprising executable instructions that, when executed by a processor, facilitate performance of operations, comprising operations that perform the method of any one of claims 1 to 31.{ YB:00561517.DOCX } -SO-

Images