Tokenized video compression with reference frames

Abstract

A method comprises receiving a group of pictures (GoP); selecting one or more reference frames from the GoP, wherein remaining frames comprise non-reference frames; partitioning the reference frames and the non-reference frames to generate a patchified reference representation and a patchified non-reference representation; encoding the patchified reference representation into an embedded discrete reference latent feature and an embedded continuous reference latent feature; encoding the patchified non-reference representation into an embedded discrete non-reference latent feature and an embedded continuous non-reference latent feature; mapping the embedded discrete reference latent feature to a selected subset of the discrete reference tokens to generate discrete reference token indices; mapping the embedded discrete non-reference latent feature to a selected subset of the discrete non-reference tokens to generate discrete non-reference token indices; and generating a discrete reference indices string based on the discrete reference token indices and the discrete non-reference token indices to transmit toward a decoder.

Description

Atty. Docket No. 4502-85401 (6000748PCT02)Tokenized Video Compression with Reference FramesCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to U. S. Provisional Application No. 63 / 781,815 filed on April 1, 2025, which is hereby incorporated by reference.TECHNICAL FIELD

[0002] The present disclosure describes techniques for learned video compression. More specifically, this disclosure is related to video compression by tokenized video generation.BACKGROUND

[0003] Artificial intelligence generated content (AIGC) uses a wide range of image generative models, including generative adversarial networks (GAN) and diffusion models. The goal is to enable fast and accessible high-quality content creation. Various methods have been developed to allow for efficient manipulation of the generated content using different types of inputs, such as using text descriptions and / or spatial / spatiotemporal compositions, like sketches or segmentations.

[0004] Large-scale pretrained vision-language models (VLM) have reached a milestone in text-to-image generation for AIGC. By training a very large model using very large datasets of captioned images from the internet, a multi-modal language-image pre-training representation, like contrastive language-image pre-training (CLIP) can be successfully learned through self-supervised contrastive learning. The joint embedding space of text and image is robust to image distribution shift.SUMMARY

[0005] The disclosed embodiments provide a general framework that jointly uses discrete and continuous video tokenizers as well as text tokenizers, together with adaptive token prediction and token selection, for video compression. In some embodiments, the framework selects reference frames in each group-of-pictures processing unit and combines with spatial -temporal referenced attention to improve efficiency of tokenized video compression. The proposed methods can achieve high compression rate with high reconstruction quality, where the bitrate and the reconstruction quality can be flexibly adjusted according to different compression needs.Atty. Docket No. 4502-85401 (6000748PCT02)

[0006] A first aspect relates to a method implemented by an encoder, comprising receiving a first group of pictures (GoP) comprising a plurality of frames; selecting one or more reference frames from the plurality of frames in the first GoP, wherein remaining frames from the plurality of frames in the first GoP comprise non-reference frames; partitioning the reference frames and the non-reference frames into spatial blocks to generate a patchified reference representation and a patchified non-reference representation; encoding the patchified reference representation into an embedded discrete reference latent feature and an embedded continuous reference latent feature; encoding the patchified non-reference representation into an embedded discrete non-reference latent feature and an embedded continuous non-reference latent feature; selecting a first subset of discrete reference tokens from a pre-trained discrete video tokenizer based on the embedded discrete reference latent feature to obtain a first selected subset of the discrete reference tokens; selecting a second subset of discrete non-reference tokens based on the embedded discrete non-reference latent feature to obtain a second selected subset of the discrete non-reference tokens; mapping the embedded discrete reference latent feature to the first selected subset of the discrete reference tokens to generate discrete reference token indices; mapping the embedded discrete non-reference latent feature to the second selected subset of the discrete non-reference tokens to generate discrete non-reference token indices; generating a discrete reference indices string based on the discrete reference token indices and the discrete non-reference token indices; and transmitting the discrete reference indices string toward a decoder.

[0007] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the patchified non-reference representation is encoded into the embedded discrete non-reference latent feature and the embedded continuous non-reference latent feature using a spatial-temporal referenced attention mechanism that utilizes the one or more reference frames to encode, wherein the spatial-temporal referenced attention mechanism comprises combining the spatial blocks from the one or more reference frames with temporal blocks from the non-reference frames at corresponding spatial locations; and processing combined spatial blocks using a three-dimensional embedding network to compute latent features that capture temporal movements of spatial visual content in the one or more reference frames.

[0008] Optionally, in any of the preceding aspects, another implementation of the aspect provides applying a reference mask to the discrete reference token indices to compute masked discrete reference token indices; applying a non-reference mask to the discrete non-reference tokenAtty. Docket No. 4502-85401 (6000748PCT02)indices to compute masked discrete non-reference token indices; and determining the discrete reference indices string based on the masked discrete reference token indices and the masked discrete non-reference token indices.

[0009] Optionally, in any of the preceding aspects, another implementation of the aspect provides applying the reference mask and the non-reference mask comprises masking out elements according to a pre-defined masking ratio, and wherein masked-out elements are set to a special symbol and are not transmitted.

[0010] Optionally, in any of the preceding aspects, another implementation of the aspect provides receiving a second GoP comprising a second plurality of frames; selecting one or more reference frames from the second plurality of frames in the second GoP; determining second discrete reference token indices for the second GoP; determining second discrete non-reference token indices for the second GoP; determining difference token indices based on differences between the discrete reference token indices of the first GoP and the second discrete reference token indices of the second GoP; and encoding the difference token indices into a difference indices string for transmission toward the decoder.

[0011] Optionally, in any of the preceding aspects, another implementation of the aspect provides selecting a reference feature mask based on the embedded continuous reference latent feature; determining a tokenized reference latent using a continuous video tokenizer based on the embedded continuous reference latent feature and the reference feature mask; determining a tokenized non-reference latent using the continuous video tokenizer based on the embedded continuous non-reference latent feature and a non-reference feature mask; encoding the tokenized reference latent and the tokenized non-reference latent into a continuous latent string; and transmitting the continuous latent string toward the decoder.

[0012] Optionally, in any of the preceding aspects, another implementation of the aspect provides decoding the continuous latent string to compute a recovered tokenized reference latent and a recovered tokenized non-reference latent; predicting a predicted tokenized reference latent and a predicted tokenized non-reference latent based on the recovered tokenized reference latent, the recovered tokenized non-reference latent, and predicted discrete token indices; determining a residual based on differences between the tokenized reference latent and the predicted tokenized reference latent and between the tokenized non-reference latent and the predicted tokenized nonAtty. Docket No. 4502-85401 (6000748PCT02)reference latent; encoding the residual into a residual string; and transmitting the residual string toward the decoder.

[0013] Optionally, in any of the preceding aspects, another implementation of the aspect provides generating a text description of content in the first GoP using a vision-language model; determining a text token mask based on the text description; determining a tokenized textual latent using a text tokenizer based on the text description and the text token mask; encoding the tokenized textual latent into a textual string; and transmitting the textual string toward the decoder.

[0014] Optionally, in any of the preceding aspects, another implementation of the aspect provides that the discrete video tokenizer comprises a first set of reference video tokens to capture spatial visual content and a second set of non-reference video tokens to capture temporal movements of visual content.

[0015] A second aspect relates to an encoder comprising a memory or storage means configured to store instructions; and one or more processors or processing means coupled to the memory or the storage means and configured to execute the instructions to cause the encoder to perform the method in any of the disclosed embodiments.

[0016] A third aspect relates to a method comprising receiving a discrete reference indices string, a reference token mask, and a non-reference token mask; decoding the discrete reference indices string to compute masked discrete reference token indices and masked discrete non-reference token indices; refilling masked-out elements in the masked discrete reference token indices and the masked discrete non-reference token indices using a discrete token prediction model to predict discrete reference token indices and discrete non-reference token indices; reconstructing a group of pictures (GoP) comprising reference frames and non-reference frames based on predicted discrete reference token indices and predicted discrete non-reference token indices; and outputting the reconstructed GoP.

[0017] Optionally, in any of the preceding aspects, another implementation of the aspect provides determining the predicted discrete reference token indices using spatial attention; and determining the predicted discrete non-reference token indices using a spatial-temporal attention mechanism based on the predicted discrete reference token indices.

[0018] Optionally, in any of the preceding aspects, another implementation of the aspect provides receiving a difference indices string; decoding the difference indices string to compute difference token indices; and determining the masked discrete reference token indices and theAtty. Docket No. 4502-85401 (6000748PCT02)masked discrete non-reference token indices based on the difference indices string, the reference token mask, and the non-reference token mask.

[0019] Optionally, in any of the preceding aspects, another implementation of the aspect provides receiving a continuous latent string; decoding the continuous latent string to compute a recovered tokenized reference latent and a recovered tokenized non-reference latent; predicting a predicted tokenized reference latent and a predicted tokenized non-reference latent based on the recovered tokenized reference latent, the recovered tokenized non-reference latent, and the predicted discrete reference token indices using a continuous token prediction model; and reconstructing the GoP based on the predicted tokenized reference latent, the predicted tokenized non-reference latent, the predicted discrete reference token indices, and the predicted discrete non-reference token indices.

[0020] Optionally, in any of the preceding aspects, another implementation of the aspect provides converting the recovered tokenized reference latent and the recovered tokenized non-reference latent into serialized tokens; and determining the predicted tokenized reference latent and the predicted tokenized non-reference latent based on the serialized tokens.

[0021] Optionally, in any of the preceding aspects, another implementation of the aspect provides receiving a residual string; decoding the residual string to compute a decoded residual; recovering a recovered tokenized reference latent and a recovered tokenized non-reference latent based on the predicted tokenized reference latent, the predicted tokenized non-reference latent, and the decoded residual; and reconstructing the GoP based on the recovered tokenized reference latent and the recovered tokenized non-reference latent.

[0022] Optionally, in any of the preceding aspects, another implementation of the aspect provides receiving a textual string and a text token mask; decoding the textual string to compute a tokenized textual latent; determining an embedded textual latent feature using a text embedding module based on the tokenized textual latent; and reconstructing the GoP based on the predicted discrete reference token indices, the predicted discrete non-reference token indices, and the embedded textual latent feature.

[0023] Optionally, in any of the preceding aspects, another implementation of the aspect provides that reconstructing the GoP comprises using a basis decoder model paired with a discrete tokenizer to reconstruct basis output reference frames and basis output non-reference frames basedAtty. Docket No. 4502-85401 (6000748PCT02)on the predicted discrete reference token indices and the predicted discrete non-reference token indices.

[0024] Optionally, in any of the preceding aspects, another implementation of the aspect provides reconstructing the GoP further comprises converting a recovered tokenized reference latent and a recovered tokenized non-reference latent into conditioning information; and applying the conditioning information to interact with input and output of different layers in the basis decoder model to refine the reconstructed GoP

[0025] A fourth aspect relates to a decoder comprising a memory or storage means configured to store instructions; and one or more processors or processing means coupled to the memory or the storage means and configured to execute the instructions to cause the decoder to perform the method in any of the disclosed embodiments.

[0026] A fifth aspect relates to a computer program product comprising computer-executable instructions stored on a non-transitory computer-readable storage medium, the computerexecutable instructions when executed by one or more processors of an encoder, cause the apparatus to perform the method in any of the disclosed embodiments.

[0027] A sixth aspect relates to a computer program product comprising computer-executable instructions stored on a non-transitory computer-readable storage medium, the computerexecutable instructions when executed by one or more processors of a decoder, cause the apparatus to perform the method in any of the disclosed embodiments.

[0028] A seventh aspect relates to a non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause an encoder to perform operations in any of the disclosed embodiments.

[0029] An eighth aspect relates to a non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause a decoder to perform operations in any of the disclosed embodiments.

[0030] For the purpose of clarity, any one of the foregoing embodiments may be combined with any one or more of the other foregoing embodiments to create a new embodiment within the scope of the present disclosure.

[0031] These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims.Atty. Docket No. 4502-85401 (6000748PCT02)BRIEF DESCRIPTION OF THE DRAWINGS

[0032] For a more complete understanding of this disclosure, reference is now made to the following brief description, taken in connection with the accompanying drawings and detailed description, wherein like reference numerals represent like parts.

[0033] FIG. 1 is a diagram illustrating a general framework for tokenized visual representation (TVR) according to an embodiment of the present disclosure.

[0034] FIG. 2Ais a diagram illustrating processing pipeline of previous neural networks (NN)-based learned video compression (LVC) methods on a sender side according to an embodiment of the present disclosure.

[0035] FIG. 2B is a diagram illustrating processing pipeline of previous NN-based LVC methods on a receiver side according to an embodiment of the present disclosure.

[0036] FIGS. 3A-3B illustrate an encoding framework according to an embodiment of the present disclosure.

[0037] FIGS. 4A-4C illustrate embodiments of the spatial-temporal referenced attention according to an embodiment of the present disclosure.

[0038] FIG. 5 illustrates a decoding framework according to an embodiment of the present disclosure.

[0039] FIGS. 6A-6B illustrate an encoding framework using a text tokenizer according to an embodiment of the present disclosure.

[0040] FIG. 7 illustrates a decoding framework according to an embodiment of the present disclosure.

[0041] FIG. 8 illustrates a flowchart according to an embodiment of the present disclosure.

[0042] FIG. 9 illustrates a flowchart according to an embodiment of the present disclosure.

[0043] FIG. 10 is a diagram illustrating an apparatus according to an embodiment of the present disclosure.DETAILED DESCRIPTION

[0044] It should be understood at the outset that although an illustrative implementation of one or more embodiments are provided below, the disclosed systems and / or methods may be implemented using any number of techniques, whether currently known or in existence. The disclosure should in no way be limited to the illustrative implementations, drawings, and techniquesAtty. Docket No. 4502-85401 (6000748PCT02)illustrated below, including the exemplary designs and implementations illustrated and described herein, but may be modified within the scope of the appended claims along with their full scope of equivalents.

[0045] The disclosed embodiments provide a general framework that jointly uses discrete and continuous video tokenizers as well as text tokenizers, together with adaptive token prediction and token selection, for video compression. In some embodiments, the framework selects reference frames in each group-of-pictures processing unit and combines with spatial -temporal referenced attention to improve efficiency of tokenized video compression. The proposed methods can achieve high compression rate with high reconstruction quality, where the bitrate and the reconstruction quality can be flexibly adjusted according to different compression needs.

[0046] The disclosed tokenized video compression framework has broad practical applications in real-time and large-scale video systems. It can be used in live video conferencing, where reference frames capture static scene content and token prediction efficiently represents facial expressions and motion, enabling smooth communication over limited bandwidth. In mobile live streaming and social media platforms, adaptive token selection and optional text-guided generation allow stable streaming under fluctuating network conditions while reducing transmission costs. The framework is also well suited for cloud gaming and remote rendering, where low-latency delivery is critical and reference-based tokenization reduces bandwidth without sacrificing responsiveness. In surveillance and smart city deployments, long-duration video feeds can be stored and transmitted efficiently by encoding static backgrounds once and updating only motion-related tokens. Additionally, the disclosure can be applied to drones, robotics, and telemedicine systems, where real-time video transmission over constrained wireless links benefits from high compression efficiency and controllable reconstruction quality.

[0047] A general framework for AIGC framework can be described as follows. A prompt input y is passed through a prompt encoder. The prompt input y may be a text input that provides a description of content to be generated using Al. In some embodiments, the prompt input y may also include images related to the to-be-generated Al content. The prompt encoder is a component configured to encode the input prompts y into a format that a multi-modal embedding network can understand and process. In an embodiment, the prompt encoder is configured to generate a prompt embedding feature zyfrom the prompt input y, which represents the prompt input y encoded into an input format for the multi-modal embedding network. The prompt embedding feature zyAtty. Docket No. 4502-85401 (6000748PCT02)captures the semantic meaning and contextual information of the prompt input y, enabling the multi-modal embedding network to understand and process the input effectively. The multi-modal embedding network is a type of neural network architecture designed to merge information from multiple modalities, such as text, images, audio, or other types of data. In an embodiment, the multi-modal embedding network is configured to compute an image embedding feature zxthat models the prior of P(zy|zx). In an embodiment, the image embedding feature zxis a numerical representation of an image in a high-dimensional vector space. The image embedding feature zxis passed to a decoder. The decoder is a decoding neural network configured to compute an output image x based on the image embedding feature zxand the prompt embedding feature zy. The target is to achieve high visual perceptual quality (e.g., natural and photo realistic, low level of visible artifacts) of the generated image x, and the semantic alignment of x to the requirement described by the prompt input y.

[0048] A general framework for LIC can be described as follows. LIC is a modern approach to image compression that utilizes deep learning techniques to learn efficient representations of images. Traditional image compression techniques rely on handcrafted algorithms that transform the image data into a compressed format. However, LIC aims to improve compression efficiency by training neural networks to automatically learn the most effective compression strategies directly from the data. LIC based on neural networks (NN) has been largely studied in recent years and has shown superior performance over traditional coding methods like Joint Photographic Experts Group (JPEG), Versatile Video Coding (VVC), and High Efficiency Video Coding (HEVC). In an embodiment, on a sender side, an input image x is passed through an input encoder to generate an image embedding feature zx, which is a representation of the input image x in a numerical format. In an embodiment, the input encoder is a neural network configured to convert the raw pixel values of the input image x into a compressed and semantically meaningful numerical representation in a high-dimensional vector space. In some embodiments, the image embedding feature zxis further compressed through quantization and arithmetic coding into a data string that is efficient for storage and transmission from the sender to a receiver.

[0049] In an embodiment, on the receiver side, a decoded image embedding feature zxis recovered from the received data string sent by the sender using arithmetic decoding and dequantization. The decoded image embedding feature zxis used as input for a decoder. The decoder is configured to reconstruct an output image x based on the decoded image embedding feature zx.Atty. Docket No. 4502-85401 (6000748PCT02)The target is to minimize the restoration loss between the reconstructed output x and the original input x, and to minimize the bits to represent the image embedding feature zxfor storage and transmission.

[0050] FIG. 1 is a diagram 100 illustrating a general framework for tokenized visual representation (TVR) according to an embodiment of the present disclosure. The core component of image / video generation is the TVR, where an input tokenizer transforms input visual signals into discretized sequences of latent codes (e.g., using a vector-quantized generative adversarial network (VQGAN)) or into a continuous latent space (e.g., using a variational autoencoder (VAE)), by modelling the probability distributions of the visual space. As shown in FIG. 1, a patchify module 102 configured to divide an input image or video x into a sequence of overlapping or nonoverlapping two-dimensional (2D) or three-dimensional (3D) patches xp. The input tokenizer module 104 configured to transform each patch into a latent code or a latent feature. The resulting sequence of latent codes or latent features is concatenated along a sequence dimension to form an embedding zx(when transformed to discrete latent codes) or z̃x(when transformed to continuous latent features). A decoder 106 is configured to transform the embedding back to the pixel space to reconstruct the output x. The input tokenizer module 104, the latent space, and the decoder 106 are optimized end-to-end to balance the efficiency of representation and the reconstruction quality. Various tokenizers may be designed for either images (e.g., VQGAN) or videos (like masked generative video transformer (Mag Vi t).

[0051] Pretrained Large Language Models (LLM) has shown strong capability in natural language understanding and natural language generation. The LLM is trained over large-scale text data like spreadsheets, fictional novels, etc., and can perform a variety of linguistic tasks. By recasting visual synthesis as a sequence prediction problem, LLM-based methods generate images / videos by predicting tokens using Autoregressive (AR) models like OpenAI's DALL-E or Non-Autoregressive (Non-AR) models like Magvit.

[0052] Learned Video compression (LVC) based on NN has been actively studied in recent years and has shown superior performance over other coding methods such as HEVC and VVC. Given a video segment consisting of n frames X = x1,..., xn, in a general LVC, on the sender side, a set of embedding features Y = y1,...,yn, are generated. These embedding features are further compressed through quantization and entropy coding into a data string that is efficient for storage and transmission. On the receiver side, a set of decoded embedding features Y = y1,...,ynisAtty. Docket No. 4502-85401 (6000748PCT02)recovered from the received data string through entropy decoding and dequantization. Based on the decoded embedding features, a corresponding set of reconstructed frames X — x1,...,xnis generated.

[0053] FIG. 2Ais a diagram 200A illustrating processing pipeline of previous NN-based LVC methods on a sender side according to an embodiment of the present disclosure. As shown in FIG.2A, on the sender side, an I-Frame selection module 202 configured to select, for a Group of Pictures (GoP) consisting of n frames X = x1,...,xn, a number of m (1 < m < ri) I-frames X1= x^1, —,x Each 1-frame is compressed by an NN-based L1C encoder 204 using methods to generate a set of compressed I-frame features Y1— y[, —,y[. The compressed I-frame features Y1are transmitted to the receiver side, where a corresponding NN-based LIC decoder 206 reconstructs the I-framesX = x[...,xm. Each I-frame x- can correspond to any frame in the GoP, i.e., 1 < ij < n. The remaining predictive (P) frames and bidirectional predictive (B) frames are then compressed based on x-, —,x- In general, to compress a remaining frame xt, a set of reference frames Rtis selected by a reference frame selection module 208. The set Rtcomprises the reconstructed 1-frames x^1, —,xilmand a set of k previously reconstructed frames x^, ■ ■■,xfk, where each xf may correspond to a frame occurring before or after the current frame t. Following the video coding pipeline, existing LVC methods compute a motion vector vtbased on the reference frames Rtand the current frame xtusing an NN-based motion estimation module 210. Aresidual rtis then computed by an NN-based motion compensation 212 module using the motion vector vt, the reference frames Rt, and xt. The motion vector vtand residual rtare transferred to the receiver side, usually with after additional entropy encoding and decoding process to further reduce transmission overhead. Finally, based on the reference frames Rt, the residual rtand the motion vector vt, an NN-based decoder 214 reconstructs the output xt.

[0054] FIG. 2B is a diagram 200B illustrating processing pipeline of previous NN-based LVC methods on a receiver side according to an embodiment of the present disclosure. As shown in FIG.2B, on the receiver side, an NN-based LIC decoder module 216, based on the received I-frame features Y1= yl, —.yl, compute the reconstructed I-frames X1= %■,For each remaining frame xt, a set of reference frames Rtis selected by a reference frame selection module 218. The set Rtcomprises the reconstructed I-framesx-,...,x- and a set of k previouslyAtty. Docket No. 4502-85401 (6000748PCT02)reconstructed frames x^, —,xf Based on the received motion vector vtand residual rt, together with the reference frames Rt, the NN-based decoder 220 reconstructs the output xt. The overall target of LVC is to minimize the restoration loss between the reconstructed output frames X — x1,..., xnand the original input frames X = x1,..., xn, and to minimize the amount of transmitted data, including embedding features Y1= y,for I-frames, and the residule rtand motion vector vtfor the remaining frames.

[0055] LIC with image tokenizers

[0056] Both discrete image tokenizers such as VQGAN and continuous tokenizers such as VAE has been used for LIC. For discrete tokens, the sender side encodes an input image into a latent space spanned by the discrete visual tokens, which are mapped into a sequence of token indices zlt...,zK. The token indices are integers, which can be effectively transferred to the receiver side. The receiver then uses the same discrete visual tokens to retrieve a decoded feature y, based on which a decoder reconstructs the output image x. Transferring integer token indices is very robust to heterogeneous computing platforms. By encoding token indices instead of latent features, token-based LIC avoids the sensitivity to computation mismatches between the sender and receiver. Moreover, transferring token indices gives the freedom of expanding latent feature dimension (often associated with better representation power for better reconstruction) without increasing bitrate, in comparison to transferring latent features or residues. In addition, discretetoken-based LIC increases robustness to input degradations. Realistic and rich textures can be generated using high-quality (HQ) visual tokens even for low-quality (LQ) inputs. For continuous tokens, the sender side encodes the input image into a latent feature using a continuous tokenizer (e.g., a VAE). On the receiver side, the transmitted bitstream is entropy decoded and dequantized to obtain a decoded latent feature, which is subsequently used by a diffusion model to reconstruct the output image x.

[0057] Despite the advantages of transferring token indices for the compression application, such as robustness to heterogeneous platforms, the freedom to expand latent feature dimensions without increasing bitrates, robustness of HQ visual tokens against LQ inputs, it is non-trivial to extend discrete-token-based LIC to LVC. In the case of LVC, sequence of individual frames, which leads to temporal inconsistency (often manifested as temporal jittering artifacts) and inefficiency, as they fail to exploit the substantial temporal redundancy inherent in video data that could otherwise be used to reduce bitrates. Conversely, video tokenizers often employ a very largeAtty. Docket No. 4502-85401 (6000748PCT02)number of spatiotemporal tokens — on the order of tens or hundreds of millions — to accurately represent complex spatiotemporal visual content. This results in a dramatic increase in the number of bits required to represent the token indices, thereby defeating the primary purpose of discretetoken-based LVC. Similarly, for LVC with continuous tokenizers, the dimensionality of the continuous latent feature is usually very large to reliably encode the complex spatial-temporal visual content. Consequently, the resulting bitrates after quantization and entropy encoding are excessively high, making efficient transmission for practical LVC applications difficult.

[0058] The present disclosure proposes a general framework for LVC that leverages a powerful TVR for LVC. Video tokenizers capture rich and complex spatial -temporal information and enable high-quality video reconstruction. However, reliable reconstruction typically requires many visual tokens, which conflicts with the goal of maintaining low bitrates. To address this trade-off, the proposed framework introduces three key mechanisms.

[0059] First, an adjustive token selection mechanism is used to select only the most relevant tokens so that the required bits to represent each used token can be reduced. Second, reference frames are selected in each GoP processing unit which can be effectively reconstructed using a small number of tokens capturing mainly spatial visual content, and then through spatial-temporal referenced attention, the remaining frames can be efficiently reconstructed using a small number of tokens capturing mainly the temporal movements of the spatial visual content from the reference frames. This reference frame-based tokenization through spatial-temporal referenced attention decomposes the vast highly complicated spatial -temporal visual space along spatial and temporal dimensions, largely reducing the difficulty of learning a large number of spatial-temporal tokens sufficient enough to represent the complex entangled spatial-temporal visual space. For instance, it requires a lot of tokens to describe a moving train and a moving car, which not only need to capture various spatial appearance of the train and the car coupled with different types of movements, but also need to differentiate the moving train and the moving car. In contrast, it is relatively easier to describe and separate a train and a car with tokens focusing on spatial visual appearance and then describe their movements with tokens focusing on temporal motion conditioned on the spatial visual appearance. Third, an efficient spatial-temporal token prediction mechanism is introduced to further improve token utilization and reduce bitrate requirements for LVC.Atty. Docket No. 4502-85401 (6000748PCT02)

[0060] FIGS. 3A-3B illustrates an encoding framework 300 (a.k.a., system) according to an embodiment of the present disclosure. An encoder may be implemented by the encoding device including one or more processors and a memory storing instructions executable by the one or more processors. The encoder may include a plurality of functional modules configured to process video segments. The modules configured to process video segments by selecting one or more reference frames and one or more non-reference frames, patchifying the reference frames and the nonreference frames, embedding the patchified reference frames and the patchified non-reference frames into discrete and continuous latent features, selecting tokens based on the embedded latent features, generating discrete token indices for the reference frames and the non-reference frames, masking selected token indices, encoding the token indices, predicting masked tokens using the reference frames, computing differences between token indices, and encoding the computed differences. Although illustrated and described as separate modules for clarity, the modules may be combined, separated, or share functionality. Each module may be implemented in hardware, software, firmware, or any combination thereof. An encoder can be interpreted as any system that performs encoding, regardless of implementation.

[0061] On the sender side, the system 300 is provided with a GoP xtof length T > 1. xtis a general 4D tensor of shape T X w x h X c, where w, h, c are the width, height, and number of channels. For example, c = 3 for color videos, c = 1 for spectral videos, or c = 4 for RGB-D (color and depth) videos. The system 300 further provides a first reference selection module 302 configured to select one or multiple reference framesfrom xt, and the remaining non-reference frames are denoted as%r. In an embodiment, the reference frames can be selected by uniform sampling, e.g., one reference frame for every 4 or 8 frames, or by predefined rules, e.g., the first or middle frame in xtis the reference frame, or by representation saliency, e.g., the reference frames are the most representative frames in xt. The system 300 further provides a first patchify module 304 configured to compute a patchified xtr pand xrpbased on x and xr, respectively. In an embodiment, the patchify module 304 processes x frame by frame. That is, for each frame in Xf, it is partitioned along the width and height dimensions into nwand nhblocks, each block having size w / nwx h / nhx c. Note that the original frame can be first padded to ensure integer block size. The patchify module 304 usually processes xrdifferently from x. For example, frames in xrare processed together as a whole unit, which is partitioned along the width, height, and temporal dimensions intow, nh, and nTblocks, each block having size Tnr / nTx w / nwxAtty. Docket No. 4502-85401 (6000748PCT02)h / nhx c, where Tnris the length of xr. The system 300 further provides a first discrete reference embedding module EMBrd306 configured to encode the patchified reference x^pinto an embedded discrete reference latent feature e^dVarious neural networks can be used as the discrete reference embedding module EMBrd. In a preferred embodiment, a Visual Transformer (ViT) is used. At the same time, the system 300 further provides a first continuous reference embedding module EMBrl308 configured to encode the patchified xpinto an embedded continuous reference latent feature ef. Various neural networks can be used as the continuous embedding module EMBrl. In a preferred embodiment, a ViT is also used as EMBrl. In an embodiment, for each block in the patchified x^p, a discrete latent feature tensor with cddimensions is computed by the first discrete reference embedding module EMBrd306. In an embodiment, for each block in the patchified x^p, a continous latent feature tensor with cldimensions is computed by the first continuous reference embedding module EMBrl308. In an embodiment, all the discrete latent feature tensors of all the blocks in the patchified x^pform the embedded discrete reference latent feature efd, and all the continuous and all the continuous latent feature tensors of all the blocks in the patchified xtr pform the embedded continuous reference latent feature ejf In an embodiment, the neural network structure of EMBrdand EMBrldoes not have temporal sequential processing. For example, for each frame, a two-dimensional (2D) embedding neural network computes a feature tensor from each w / nwx h / nhx c block, which is serialized and processed by transformer encoders to compute a cd-dim or cz-dim latent feature tensor.

[0062] The system 300 further provides a first discrete non-reference embedding module EMBnrd310 configured to encode the patchified non-reference xrpinto an embedded discrete non-reference latent feature etrd, by using the patchified xtr pas references. At the same time, the system 300 further provides a first continuous non-reference embedding module EMBnrl312 configured to encode the patchified non-reference xrpinto an embedded continuous non-reference latent feature etrZ, by using the patchified x^pas references. The first discrete non-reference embedding module EMBnrd310 and the first continuous non-reference embedding module EMBnrl312 are neural networks utilizing a spatial -temporal referenced attentionAtty. Docket No. 4502-85401 (6000748PCT02)mechanism with temporal sequential processing. For illustration, FIGS. 4A-4C present preferred embodiments of the spatial-temporal referenced attention mechanism.

[0063] In FIGS. 4A-4C, each column is a patchified reference frame in xtr por a patchified non-reference frame in xrp. The arrows denote the information flow between serialized blocks through neural network operations like convolution, linear connection, self-attention, etc. The blocks in reference frames participated in the latent calculation in non-reference frames with a deterioration rate, i.e., a reference frame influences the close-by non-reference frames more than far-away frames. For simplicity, information flow among non-reference frames is omitted from the FIGS. 4A-4C. In the preferred embodiment, the non-reference frames are treated as three-dimensional (3D) tensors. That is, the input to EMBnrdor EMBnrlis a combination of xtr pand xrp, where from each reference frame inx^p, aw / nwx h / nhx c block with the same spatial location in all reference frames is combined with the Tnr / nTx w / nwx h / nhx c block in xrpat the same spatial location, e.g., by appending the w / nwx h / nhx c blocks along the temporal dimension and at the original temporal location of the reference frames. A 3D embedding network then is used to compute a feature tensor from this combined input. Depending on how reference frames are selected, e.g., whether there is one reference frame at the beginning of the 3D tensor as shown in FIG. 4A, one reference frame at the middle of the 3D tensor as shown in FIG. 4B, or two reference frames at the beginning and end of the 3D tensor as shown in FIG. 4C, the corresponding 3D embedding network calculates e^dor e1so that the non-reference latent features capture the temporal movements of the spatial visual content in the reference frames.

[0064] Referring back to FIG. 3A, the system 300 is also provided with another GoPxt+1of length T > 1, where xt+1and xthave the same shape. In an embodiment, xt+1is the next consecutive video segment of xtalong the timeline. The system 300 further provides a second reference frame selection module 314 configured to select, same as xt, the one or multiple reference frames Xj+1from xt+1, and the remaining non-reference frames are denoted as x^. The system 300 further provides a second patchify module 316 configured to compute a patchified x^ and x^pbased on x+1and x +15respectively, in the same way patchified xtr pand xrpare computed based on x^ and xr. That is, the patchified x^ and xtr phave the same shape, and the patchified x^pand x”rphave the same shape. The system 300 further provides a second discrete reference embedding module EMBrd318 configured to encode the patchified reference xtr^1intoAtty. Docket No. 4502-85401 (6000748PCT02)an embedded discrete reference latent feature, and a second continuous reference embedding module EMBrl320 configured to encode the patchified x^ into an embedded continuous reference latent feature e^+i. The system 300 further provides a second discrete non-reference embedding module EMBnrd322 configured to encode the patchified non-reference x into an embedded discrete non-reference latent featureby using the patchified x^ as references. The system 300 further provides a second continuous non-reference embedding module EMBnrl324 configured to encode the patchified non-reference xp^f into an embedded continuous non-reference latent featureby using the patchified x^ as references.

[0065] The system 300 further provides a discrete reference token selection module 326 configured to select, based on the embedded discrete reference latent features exdand e£da subset of discrete reference tokensTknxdfrom a list of reference video tokens Tknrdin a pre-trained discrete video tokenizer (such as Nvidia’s cosmos discrete video tokenizer), and compute a reference token maskmLr„, to mask out the unselected tokens in Tknrd. The system 300 further provides a discrete reference tokenization module 328 configured to compute a discrete reference token indices zxdbased on exdand mxdXtand compute a discrete reference token indices zxd„ based on exd„and. In an embodiment, the pre-learned reference video tokens Tknrdcomprises N of tokens, each is a latent feature vector of cddimension. Each latent feature vector in exdor ed+icorresponds to each block in xtrpor x^, which is mapped to a token tknrdi E T / cnLXdb-Q4i-d1 or tknr*dt+1A < E i — 1,...,nwx nhx nT, where tknxdi or tknrd< is closest to the latent feature vector of the block in exdor exdmeasured by a distance metric (such as LI or L2 norm). That is, the entire zxdor z£d+ihas nwx nhx nTintegers correspond to the indices of the mapped selected tokens. Since the number of selected reference tokens is much less than the number of original pre-learned reference tokens, the bit consumption of encoding indices of the selected reference tokens is largely reduced. There are many ways the discrete reference token selection module 326 selects the subset of discrete tokens. For example, a pretrained neural network model can be used, which predicts the most relevant tokens in Tknrdto represent the video content of xtr vand x^ based on exdand exd+i.

[0066] The system 300 further provides a discrete non-reference token selection module 330 configured to select, based on the embedded discrete non-reference latent features e”trdand e!^.Atty. Docket No. 4502-85401 (6000748PCT02)a subset of discrete non-reference tokens Tkri^ frOm a list of non-reference video tokens Tknnrdin a pre-trained discrete video tokenizer (such as Nvidia’s cosmos discrete video tokenizer), and computes a non-reference token maskto mask out the unselected tokens in Tknnrd. The video tokenizer can simply use the same list of video tokens for both Tknrdand Tknnrd, or can have different sets of tokens for Tknrdand Tknnrd. For example, the reference Tknrdmay be specialized to capture spatial visual content and can be trained with a large variety of both images and video frames, while the non-reference Tknnrdmay be specialized to capture temporal movements of the different visual content and trained with video frames. The system 300 further provides a discrete non-reference tokenization module 332 configured to compute a discrete non-reference token indices z^dbased on e”trdand, and compute a discrete non-reference token indices z”^ based onand In an embodiment, the pre-learned non-reference video tokens Tknnrdcomprises of N of tokens, each is a latent feature vector of cddimension. Each latent feature vector in e^dor corresponds to each block in xrpor x^p, which is mapped to a token tkn^f G, or tkn^d< G Tkn^x^,, i = 1,...,nwx nhx nT, where tknror tkn^dis closest to the latent feature vector of the block in edOr e^t+dl measured by a distance metric (such as LI or L2 norm). That is, the entire z^dor z”^ has nwx nhx nTintegers corresponding to the indices of the mapped selected tokens. Since the number of selected non-reference tokens is much less than the number of original pre-learned non-reference tokens, the bit consumption of encoding indices of the selected non-reference token is largely reduced. There are many ways the discrete non-reference token selection module 328 configured to select the subset of discrete tokens, e.g., by using a pre-trained neural network model to predict the most relevant tokens in Tknnrdto represent the video content of xrpand x^pbased on e£trdand

[0067] As shown in FIG. 3B, the system 300 further provides a reference masking module 334 configured to apply masks to the discrete reference token indices zdto compute a masked discrete reference token indicesz^d’mask^ and apply masks to the discrete reference token indices z^dto compute a masked discrete reference token indicesz^d^-askUsually, a pre-defined masking ratio is given a user, and elements in z*dor zd+1are masked out according to the ratio. Masked-out elements are set to a special symbol agreed between sender and receiver to avoid transmission.Atty. Docket No. 4502-85401 (6000748PCT02)The reference masking module 334 can randomly mask out elements in zxdor zxd+ior use some selection mechanism to mask out less frequent indices in zxdorzydrThe system 300 further provides a non-reference masking module 336 that applies masks to the discrete non-reference token indices zx^dto compute a masked discrete non-reference token indicesz^d,maskan(j applies masks to the discrete non-reference token indices z"^ to compute a masked discrete non-reference token indicesz^d’maskUsually, a pre-defined masking ratio is given by user, and elements in zx^dor z”^ are masked out according to the ratio. Masked-out elements are set to a special symbol agreed between sender and receiver to avoid transmission.

[0068] The system 300 further provides a compute discrete difference module 338 configured to compute a different token indices SXf+ibased on the masked discrete reference token indiceszy>and zy^, and the masked discrete non-reterence token indiceszYtand z^T^as we'lasthe reference and non-reference token masks mJdXt+1and mx^dt+1. The target is to compute o“t+1to utilize the redundancy in zXt, zXt^, zXtand zx’ so that less transmission is required.

[0069] After that, the masked discrete reference token indicesz^d’maskand zx^ask, the reference token mask mxdXt+1, the masked discrete non-reference token indicesz^d’maskanc[ nrd.mask non-reference token mask mx Xt, „, and the different token indices Sd, „ are fed into an indices encoding module 340 to compute a discrete reference indices string sXf. Also, SXfis further fed into a difference encoding module 342 to compute a difference indices string sXt. The discrete reference indices string sXt, the difference indices string sdt, the reference token maskand the non-reference token mask mxtrdMare all transmitted to the receiver side. The indices encoding module 340 and the difference encoding module 342 usually employ lossless integer entropy coding methods to further reduce the bit consumption of representing the token indices zy^, zy', zY, and zy^, and the different token indices„. Due to the spatial-temporal redundancy in the visual content of video sequences, the token indices between consecutive GoPs are highly predictable and redundant, and therefore sXtand sXtusually consume much less bits than a string directly encoded from the original non-masked token indices.Atty. Docket No. 4502-85401 (6000748PCT02)

[0070] Furthermore, on the sender side, the masked discrete reference token indicesz^’maskj rd, mask 1 j r. i • j- nr d, mask < nr d, mask r < and z,,, as well as the masked non-reference token indices z„ and z_ are red into a discrete token prediction module 344 to compute a predicted discrete reference token indices zxr^ and z£f as well as a predicted discrete non-reference token indices z”trdand z™^, where the masked-out elements inz„^,5zxt+'l, ’ z„ ’,5and zx,,t+l are refilled byJtoken prediction.

[0071] On the other hand, the embedded continuous reference latent features extland e£+1are fed into a continuous reference token selection module 346 to compute a reference feature mask which is used by a continuous reference tokenization module 348 to compute a tokenized reference latent z£ based on the continuous reference latent featuresand to compute a tokenized reference latent zxJ based on the continuous reference latent features ej^+ i. In an embodiment, the continuous reference tokenization module 348 uses a pre-trained continuous video tokenizer (like Nvidia’s Cosmos continuous video tokenizer) to compute zxJ and z£* and the feature maskcan mask out a set of feature channels that are unimportant to represent the visual content of the current extland ex'+ |so that the corresponding channels in the tokenized reference latent z^ and zl),, are masked out. The embedded continuous non-reference latent features e1andare fed into a continuous non-reference token selection module 350 to compute a non-reference feature mask, which is used by a continuous non-reference tokenization module 352 to compute a tokenized non-reference latent zXtribased on the continuous non-reference latent features e1and to compute a tokenized non-reference latent zj^ based on the continuous non-reference latent features. In an embodiment, the continuous non-reference tokenization module 352 uses a pre-trained continuous video tokenizer (like Nvidia’s Cosmos continuous video tokenizer) to compute z^1and zj^, and the feature maskcan mask out a set of feature channels that are unimportant to represent the visual content of the current e”triand so that the corresponding channels in the tokenized non-reference latent zXfHand z”r,^ are masked out. In an embodiment, the pre-trained continuous video tokenizer can be the same for computing the reference extland non-reference e1(also the reference extl+iand non-reference or can be different. For example, the reference extland exJ+imay be specializedAtty. Docket No. 4502-85401 (6000748PCT02)to capture spatial visual content and can be trained with a large variety of both images and video frames, while the non-referenceex^1and may be specialized to capture temporal movements of visual content and trained with video frames.

[0072] In an embodiment, the tokenized reference latent zxtland the tokenized non-reference latent zXfr(are processed by a latent encoding module 354 to generate a continuous latent string sXt, which is sent to the receiver. The latent encoding module 354 usually comprises quantization and entropy encoding processes so that the generated continuous latent string sxlconsumes less bits than the tokenized reference latent zxand the tokenized non-reference latent z”rI. In an embodiment, in the sender, sXltis further processed by a latent decoding module 356 to compute a recovered tokenized reference latent zxland a recovered tokenized non-reference latent zx^1. The latent decoding module 356 usually comprises corresponding entropy decoding and dequantization processes which is used also in the receiver to recover zxtland zx^1from the received sxIn an embodiment, a continuous token prediction module 358 computes a predicted tokenized reference latent zxland a predicted tokenized non-reference latent z”^, based on the recovered tokenized reference latent zxtl, the recovered tokenized non-reference latent zXfr(, and the predicted discrete reference token indices zxdand zxd+i, and the predicted discrete non-reference token indices zx^dand z”^. The predicted referenced and non-referenced tokenized latent zxland z”^ aim to recover the referenced and non-referenced tokenized latent z£* and zx^±, respectively, which carry important visual details to reconstruct the input Xj+1and x^, respectively. In an embodiment, a compute continuous difference module 360 further computes a residual rxbased on the predicted zx\, „ and zxas well as the original zx\, „ and zx5. In an embodiment, the residual is further processed by a residual encoding module 362 to generate a residual string Sxt+1, which is transmitted to the receiver. The residual encoding module 362 usually comprises quantization and entropy encoding processes to reduce the bit consumption of the residual string Art+i-

[0073] FIG. 5 illustrates a decoding framework 500 (a.k.a., system) according to an embodiment of the present disclosure. A decoder may be implemented by a decoding device including one or more processors and a memory storing instructions executable by the one or more processors. The decoder may include a plurality of functional modules configured to decodeAtty. Docket No. 4502-85401 (6000748PCT02)difference indices, reconstruct masked discrete reference token indices and masked discrete nonreference token indices, predict discrete token indices, recover continuous latent features, and reconstruct video frames based on data received from the encoder. Although illustrated and described as separate modules for clarity, the modules may be combined, separated, or share functionality. Each module may be implemented in hardware, software, firmware, or any combination thereof. A decoder can be interpreted as any system that performs decoding, regardless of implementation.

[0074] As shown in FIG. 5, on the receiver side, the system 500 provides a difference decoding module 502 configured to compute, using the received difference indices stringthe different token indices 5Xt+1- In anembodiment, using the received discrete indices string sdt, the reference token mask rnXfjXf., the non-reference token mask mx xand the different token indices Sdtan indices decoding module 504 computes the masked discrete reference token indicesZxd’maskand z ' as well as the masked discrete non-reference token indicesz„, and z„, The difference decoding module 502 and the indices decoding module 504 usually employ lossless integer entropy decoding methods corresponding to the lossless integer entropy encoding methods in the difference decoding module 502 and the indices decoding module 504, respectively, so that rd, mask nr d, mask rd, mask. nr d, maskc<T4 41zxt z t ’, z t+ ’l, ’ and zrtt -I- 1 are fullyJrecovered. In an embodiment, ’ the masked i- 4 rd, mask rd, mask nr d, mask i nr d, mask.. 1 i-, • discrete zrt, z ', zr’, and zrtare ted into a discrete token prediction module 506 to compute the predicted discrete reference token indices zxdand zx^ and the predicted discrete non-reference token indices z^dand z”^.

[0075] At the same time, using the received continuous latent string sXlt, a latent decoding module 508 (same as the one in sender) computes the recovered tokenized reference latent zxland the recovered tokenized non-reference latent z"rJ. In an embodiment, based on zxd, zxd, z!d,Candacontinuous token prediction module 510 same as the sender side computes the predicted tokenized reference latent zxtl+iand the predicted tokenized non-reference latent ^xt+r- Using the received residual stringa residual decoding module 512 computes a decoded residual rX(.+1. The residual decoding module 512 usually comprises entropy decoding and dequantization processes corresponding to the residual encoding module 362 in the sender. In an embodiment, a continuous token recover module 514 computes the recovered tokenizedAtty. Docket No. 4502-85401 (6000748PCT02)reference latent zxf,„ and the recovered tokenized non-reference latent z(r,1based on the predicted tokenized reference latent z£*+i, the predicted tokenized non-reference latent zx^±, and the decoded residual rX(+i. Finally, the recovered tokenized referencelatent and z£*+i, the recovered tokenized non-reference latent zx^1and z”^, the predicted discrete reference token indices zxdand zxd+i, and the predicted discrete non-reference token indices zx^dandzj^, are all fed into a reconstruction module 516 to compute a reconstructed GoPs xtand xt+1, corresponding to the input xtand xt+1. In an embodiment, the system 500 then goes into the next encoding / decoding cycle to process GoPs xt+1and xt+2. In an embodiment, the reconstruction module 516 comprises a neural network that is the decoder part of a conditional generative neural network, such as the decoder part of a conditional VAE. In an embodiment, the reconstruction module 516 comprises of a basis decoder model that is paired with the discrete tokenizer and is pre-trained end-to-end with the discrete tokenizer to reconstruct basis output GoPs x^ and x[+1based on zxdandand to reconstruct basis output GoPs xrandbased on zx^dand In anembodiment, a submodule converts zx +iand z^ into conditions to interact with the input and output of different layers in the basis decoder model, such as the LoRA mechanism for computing and x^+1, and converts zx^±and z^1into conditions to interact with the input and output of different layers in the basis decoder model, such as the LoRA mechanism for computing xrand. xtrand xrgives the reconstructed xt. x+1and x^ gives the reconstructed xt+1. Usually, the basis decoder model is further fine-tuned when training the submodules for condition generation in this disclosure.

[0076] FIGS. 6A-6B illustrates an encoding framework 600 (a.k.a., system) using a text tokenizer according to an embodiment of the present disclosure. In an embodiment, when text tokenizer is employed at the sender side of embodiment 2, and in addition to the operations described with reference to FIGS. 3A-3B, the input GoPs xtand xt+1are also fed into a text generation module 602 to compute a text description YXf >x. In an embodiment, the text generation module 602 usually uses a pre-trained VLM (such as CLIP ) to generate the text description LXtiXj+ito describe the content ofxtandt+1, such as the semantics, objects, object relations, motions, etc. In an embodiment, a text token selection module 604 computes a text token mask mXtiXbased on YXf > X(.+1. Both the text token mask mXf xand the text description YXf Xf+1are further fed into a text tokenizer 606 to compute a tokenized textual latent zXf;x. In anAtty. Docket No. 4502-85401 (6000748PCT02)embodiment, the text tokenizer 606 usually uses the text encoder of an LLM (such as OpenAI's DALL-E) or a VLM (such as CLIP) to compute the tokenized textual latent zx xwhich usually consists of a sequence of integer indices corresponding to codeword indices of a pre-trained textual codebook. The text token mask mXt xaims at masking out irrelevant codewords in this textual codebook so that less bits can be used to represent the remaining codeword indices. In an embodiment, the pre-trained textual codebook can be a semantic dictionary where each codeword is a natural semantic word in the dictionary or can be a pre-learned codebook where each codeword comprises a combination of semantic words. In an embodiment, a text token encoder module 608 computes a bit-efficient textual string sXf xfrom the tokenized textual latent zXf X■ The textual string sxis transmitted to the receiver side. In an embodiment, a text token encoder module 608 usually uses lossless integer entropy encoding methods to reduce the bit consumption of the tokenized textual latent zx^t’xt-n

[0077] FIG. 7 illustrates a decoding framework 700 (a.k.a., system) according to an embodiment of the present disclosure. In an embodiment, in addition to the operations described with reference to FIG. 5, on the receiver side of the embodiment 2, using the received textual stringst,xt+1and text token maska text token decoding module 702 recovers the tokenized textual latent In an embodiment, the text token decoding module 702 usually employs lossless integer entropy decoding methods corresponding to the integer entropy encoding methods in the text token encoding module to fully recover z^Xt+ i. In an embodiment, a text embedding module 704 computes an embedded textual latent featurezXt>x, which is fed into a reconstruction module 706 in addition to other inputs of the reconstruction module as in FIGS.3 A-3B to reconstruct the output GoPs xtand xt+1. The text embedding module 704 is usually the text encoder of an LLM or a VLM. The reconstruction module 706 usually comprises a neural network that is the decoder part of a conditional generative neural network, such as the decoder part of a conditional VAE. The reconstruction module 706 contains an additional submodule that converts zXt x(in addition to z£*, z^+1, z^1, and z^J into conditions to interact with the input and output of different layers in the basis decoder model, such as the LoRA mechanism. The basis decoder model is usually fine-tuned when training the submodules for condition generation in this disclosure.

[0078] Individual Inference ModulesAtty. Docket No. 4502-85401 (6000748PCT02)

[0079] This section describes the details of the discrete token prediction module, the continuous token prediction module, the compute discrete difference module, the compute continuous difference module, and the continuous token recover module.

[0080] In an embodiment, the discrete token prediction module usually has a transformer encoder-decoder structure. In one embodiment, the masked discrete reference token indices z^d,maskare first fed into a transformer encoder-decoder network to compute the predicted discrete reference token indices z*d, where mainly spatial attention is used. Then, z^dis further fed into another transformer encoder-decoder network together withz^d’maskto compute the predicted discrete reference token indices z^d, with both spatial and temporal attention mechanisms. After that, zdand z^dare fed into another transformer encoder-decoder network to compute the predicted discrete reference token indices z£^+iand the predicted discrete non-reference token indices zj^ with both spatial and temporal attentions. In another embodiment, the predicted zdand z^dare fed into a transformer encoder-decoder network to compute the predicted discrete reference token indices zj^ first, and then z^dzdand z™tdare fed into another transformer encoder-decoder network to compute the predicted discrete non-reference token indices z”^. This disclosure does not put restrictions on the order z^, z^d, z^d+l, andz"^ are computed, or the network structures used for such prediction.

[0081] In an embodiment, the continuous token prediction module has a similar transformer encoder-decoder structure, but usually with an additional serialization process where the recovered tokenized reference latent z^tland the recovered tokenized non-reference latent z”triare turned into serialized tokens and then fed into the transformer encoder-decoder module to compute the predicted tokenized reference latent z^+iand the predicted tokenized non-reference latent zj^. The predicted discrete reference token indices zdand z*d, and the predicted discrete non-reference token indices z"trdand z"^ are used to guide the token generation through crossattention, where a cross-attention module is inserted into each decoder block of the transformer decoder so that the guidance computed from z^d, z^d+i, z^dand z^d^ are fed into the crossattention module as conditions. In the preferred embodiment, z£* and zdare used by a transformer encoder-decoder module to compute zj^+1, and then z£, z”triand zj^ are used by a transformer encoder-decoder module to compute This disclosure does not put restrictionsAtty. Docket No. 4502-85401 (6000748PCT02)on the order z£* andare computed, or the specific network structures used for such prediction.

[0082] In an embodiment, the compute continuous difference module computes the different token indices 6Xt+1to reduce the bits of transferring the masked discrete reference and non- reference token indices, z ' ■> z ’, and zYt. In the preferredembodiment,5 6XAf-I-! consists of the difference ^t+1 between zxA^f ’maskand zx^t+’m1ask,5and the i- r * r rd u *nrd,mask i *nrd.maskTr i, • ^rd.mask *rd.mask • difference between z and zr^. If an element in z ’ — z ’ is zero (the same index), it does not need to be transmitted and is marked as a special symbol known by both sender and receiver in 5ya+i. The indices of non-zero elements are kept unchanged in 6X^+1, whose representation are determined by the reference token mask. Similarly, if an elementinzx' — zXtis zero (the same index), it does not need to be transmitted and is marked as a special symbol known by both sender and receiver in Sx^. The indices of non-zero elements are kept unchanged in 8™^, whose representation are determined by the reference token mask °ther embodiments, 8Xt+1can consist of the difference between any pair of z^,mask,zY„, z ', and z~.. This disclosure does not put any restrictions on how of, is computed.

[0083] In an embodiment, the compute continuous difference module computes the residualrxt+1- Inthe preferred embodiment, rXt+1consists of the difference between zxtl+1and the predicted z£+1, as well as the difference between zx^ and the predicted zx^t.

[0084] In an embodiment, the continuous token recover module computes the recovered tokenized reference latent zf,, and the recovered tokenized non-reference latent zybased on the predicted tokenized reference latent zxtl+i, the predicted tokenized non-reference latent ^xth’anc* hedecoded residual rXt+1. Since the residual encoding and residual decoding modules normally encodes and decodes the residual in a lossy way, the recovered tokenized reference and non-reference latent zx, „ and z”ron the receiver side are lossy version the recovered tokenized reference and non-reference latent zxxt+, „l and zytT+l on the sender side.

[0085] This disclosure does not put any restrictions on the model structures of or how to obtain the pre-trained models for the previously mentioned VLM, LLM, discrete video tokenizers,Atty. Docket No. 4502-85401 (6000748PCT02)continuous video tokenizer, text tokenizer, etc. This disclosure also does not put any restrictions on the model structure of or how to obtain the neural network models for any model -based modules in the framework. The specific methods used by other non-model related modules such as the compute discrete difference module and compute continuous difference module and the specific format of the computed different token indices 8^t+iand residual r^t+1are not restricted either. Furthermore, this disclosure does not put any restrictions on the method and order to train different network modules in the framework, which modules are pre-trained which modules are finetuned or trained from scratch, the exact loss functions to train these modules, etc.

[0086] FIG. 8 is a method 800 implemented by an encoding device according to an embodiment of the disclosure. In an embodiment, the encoding device is a computer, a smart phone, a smart tablet, or other device configured to process and display video content. The encoding device may execute one or more software modules, hardware accelerators, or combinations thereof to perform the tokenization, token selection, token prediction, and encoding operations described herein. In certain embodiments, the encoding framework operates in real time during gameplay or video rendering to compress visual data for transmission or storage.

[0087] In block 802, the method 800 comprises receiving a first group of pictures (GoP) comprising a plurality of frames.

[0088] In block 804, the method further comprises selecting one or more reference frames from the plurality of frames in the first GoP, wherein remaining frames from the plurality of frames in the first GoP comprise non-reference frames.

[0089] In block 806, the method further comprises partitioning the reference frames and the non-reference frames into spatial blocks to generate a patchified reference representation and a patchified non-reference representation.

[0090] In block 808, the method further comprises encoding the patchified reference representation into an embedded discrete reference latent feature and an embedded continuous reference latent feature.

[0091] In block 810, the method further comprises encoding the patchified non-reference representation into an embedded discrete non-reference latent feature and an embedded continuous non-reference latent feature using a spatial-temporal referenced attention mechanism that utilizes the one or more reference frames.Atty. Docket No. 4502-85401 (6000748PCT02)

[0092] In block 812, the method further comprises selecting a first subset of discrete reference tokens from a pre-trained discrete video tokenizer based on the embedded discrete reference latent feature to obtain a first selected subset of the discrete reference tokens.

[0093] In block 814, the method further comprises selecting a second subset of discrete nonreference tokens based on the embedded discrete non-reference latent feature to obtain a second selected subset of the discrete non-reference tokens.

[0094] In block 816, the method further comprises mapping the embedded discrete reference latent feature to the first selected subset of the discrete reference tokens to generate discrete reference token indices.

[0095] In block 818, the method further comprises mapping the embedded discrete non-reference latent feature to the second selected subset of the discrete non-reference tokens to generate discrete non-reference token indices.

[0096] In block 820, the method further comprises generating a discrete reference indices string based on the discrete reference token indices and the discrete non-reference token indices.

[0097] In block 822, the method further comprises transmitting the discrete reference indices string toward a decoder over a network for display or further processing.

[0098] FIG. 9 is a method 900 implemented by a decoding device according to an embodiment of the disclosure. In an embodiment, the decoding method may be implemented by the decoding device configured to reconstruct and display video content. The decoding device may comprise one or more processors, memory, and communication interfaces for receiving encoded video bitstreams and performing the decoding operations described herein. Examples of such decoding devices include, but are not limited to, a computer, a smart phone, a smart tablet, a smart television, a gaming console, or any other electronic device capable of decoding and displaying video content.

[0099] In block 902, the method comprises receiving a discrete reference indices string, a reference token mask, and a non-reference token mask.

[0100] In block 904, the method further comprises decoding the discrete reference indices string to compute masked discrete reference token indices and masked discrete non-reference token indices.

[0101] In block 906, the method further comprises refilling masked-out elements in the masked discrete reference token indices and the masked discrete non-reference token indices using aAtty. Docket No. 4502-85401 (6000748PCT02)discrete token prediction model to predict discrete reference token indices and discrete nonreference token indices.

[0102] In block 908, the method further comprises reconstructing a group of pictures (GoP) comprising reference frames and non-reference frames based on predicted discrete reference token indices and predicted discrete non-reference token indices.

[0103] In block 910, the method further comprises outputting the reconstructed GoP.

[0104] FIG. 10 is a schematic diagram of a computing device 1000 (e.g., a personal computer, smart phone, smart tablet, handheld gaming device, etc.) according to an embodiment of the disclosure. The computing device 1000 is suitable for implementing the disclosed embodiments as described herein. The computing device 1000 comprises ingress ports / ingress means 1010 (a.k.a., upstream ports) and receiver units (Rx) / receiving means 1020 for receiving data; a processor, logic unit, or central processing unit (CPU) / processing means 1030 to process the data; transmitter units (Tx) / transmitting means 1040 and egress ports / egress means 1050 (a.k.a., downstream ports) for transmitting the data; and a memory / memory means 1060 for storing the data. In an embodiment, the receiver units (Rx) / receiving means 1020 comprise a discrete circuit, integrated circuit, chip set, package, hardware module, electronic device, or other structure capable of receiving signals. In an embodiment, the transmitter units (Tx) / transmitting means 1040 comprise a discrete circuit, integrated circuit, chip set, package, hardware module, electronic device, or other structure capable of transmitting signals. The computing device 1000 may also comprise optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the ingress ports / ingress means 1010, the receiver units / receiving means 1020, the transmitter units / transmitting means 1040, and the egress ports / egress means 1050 for egress or ingress of optical or electrical signals.

[0105] The processor / processing means 1030 is implemented by hardware and software. The processor / processing means 1030 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), field-programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and digital signal processors (DSPs). The processor / processing means 1030 is in communication with the ingress ports / ingress means 1010, receiver units / receiving means 1020, transmitter units / transmitting means 1040, egress ports / egress means 1050, and memory / memory means 1060. The processor / processing means 1030 comprises a tokenized video generation module 1070. The tokenized video generation module 1070 is able to implement the methods disclosed herein. The inclusion of the hybrid-style video generation module 1070 therefore provides aAtty. Docket No. 4502-85401 (6000748PCT02)substantial improvement to the functionality of the computing device 1000 and effects a transformation of the computing device 1000 to a different state. Alternatively, the tokenized video generation 1070 is implemented as instructions stored in the memory / memory means 1060 and executed by the processor / processing means 1030.

[0106] The computing device 1000 may also include input and / or output (I / O) devices or I / O means 1080 for communicating data to and from a user. The I / O devices or I / O means 1080 may include output devices such as a display for displaying video data, speakers for outputting audio data, etc. The I / O devices or I / O means 1080 may also include input devices, such as a keyboard, mouse, trackball, etc., and / or corresponding interfaces for interacting with such output devices. The memory / memory means 1060 comprises one or more disks, tape drives, and solid-state drives and may be used as an over-flow data storage device, to store programs when such programs are selected for execution, and to store instructions and data that are read during program execution. The memory / memory means 1060 may be volatile and / or non-volatile and may be read-only memory (ROM), random-access memory (RAM), ternary content-addressable memory (TCAM), and / or static RAM (SRAM).

[0107] Embodiments of the present disclosure provide at least the following technical advantages.

[0108] This disclosed framework offers high compression rate with high-quality video generation at the same time by exploiting the power of both discrete video tokenizers and continuous video tokenizers. The video tokenizers have more representation power in describing the spatial-temporal visual content than image tokenizers, giving high-quality to the reconstructed videos. The token selection and token prediction mechanism reduce the required bits to represent the tokens and the required tokens to be encoded for transmission, giving effective bitrate reduction.

[0109] The disclosed framework improves rate-distortion performance by the reference-frame-based tokenization mechanism. Reference frames that can be effectively reconstructed using a small number of spatial-content-focusing tokens are first selected. Then through spatial -temporal referenced attention the remaining non-reference frames are efficiently reconstructed using a small number of temporal-movement-focusing tokens.

[0110] The discrete video tokenizer employs a pre-learned spatial-temporal visual codebook that is robust to input degradations and optimized for perceptual reconstruction quality. The continuous video tokenizer extracts fine-grained detail information from the input video to ensureAtty. Docket No. 4502-85401 (6000748PCT02)high-fidelity reconstruction. By combining discrete and continuous tokenizers, the system achieves a balanced tradeoff between perceptual quality and fidelity.

[0111] The framework further supports flexible, task-oriented quality control. Depending on application requirements, the system can dynamically adjust the contribution of continuous tokenized latents relative to discrete tokens to emphasize either perceptual quality or fidelity during reconstruction.

[0112] When textual descriptions are incorporated, a vision-language model (VLM) is employed to jointly model the distribution of video tokens and corresponding text descriptions. Trained on large-scale image-text and / or video-text datasets, the VLM enriches the feature space with complementary visual and linguistic information, enabling more expressive video representations and improved compression efficiency compared to visual-only methods.

[0113] The system also enables guided compression through text-based control commands. Such commands may take default forms (e.g., prioritizing fidelity or perceptual quality) or specify customized compression objectives, such as emphasizing a particular object or region for enhanced reconstruction.

[0114] While several embodiments have been provided in the present disclosure, it may be understood that the disclosed systems and methods might be embodied in many other specific forms without departing from the spirit or scope of the present disclosure. The present examples are to be considered as illustrative and not restrictive, and the intention is not to be limited to the details given herein. For example, the various elements or components may be combined or integrated in another system or certain features may be omitted, or not implemented.

[0115] In addition, techniques, systems, subsystems, and methods described and illustrated in the various embodiments as discrete or separate may be combined or integrated with other systems, components, techniques, or methods without departing from the scope of the present disclosure. Other examples of changes, substitutions, and alterations are ascertainable by one skilled in the art and may be made without departing from the spirit and scope disclosed herein.

Claims

Atty. Docket No. 4502-85401 (6000748PCT02)CLAIMSWhat is claimed is:

1. A method implemented by an encoder, comprising:receiving a first group of pictures (GoP) comprising a plurality of frames;selecting one or more reference frames from the plurality of frames in the first GoP, wherein remaining frames from the plurality of frames in the first GoP comprise non-reference frames;partitioning the reference frames and the non-reference frames into spatial blocks to generate a patchified reference representation and a patchified non-reference representation; encoding the patchified reference representation into an embedded discrete reference latent feature and an embedded continuous reference latent feature;encoding the patchified non-reference representation into an embedded discrete non-reference latent feature and an embedded continuous non-reference latent feature;selecting a first subset of discrete reference tokens from a pre-trained discrete video tokenizer based on the embedded discrete reference latent feature to obtain a first selected subset of the discrete reference tokens;selecting a second subset of discrete non-reference tokens based on the embedded discrete non-reference latent feature to obtain a second selected subset of the discrete non-reference tokens;mapping the embedded discrete reference latent feature to the first selected subset of the discrete reference tokens to generate discrete reference token indices;mapping the embedded discrete non-reference latent feature to the second selected subset of the discrete non-reference tokens to generate discrete non-reference token indices;generating a discrete reference indices string based on the discrete reference token indices and the discrete non-reference token indices; andtransmitting the discrete reference indices string toward a decoder.

2. The method of claim 1, wherein the patchified non-reference representation is encoded into the embedded discrete non-reference latent feature and the embedded continuous non-reference latent feature using a spatial-temporal referenced attention mechanism that utilizes the one or more reference frames to encode, and wherein the spatial-temporal referenced attention mechanism comprises:Atty. Docket No. 4502-85401 (6000748PCT02)combining the spatial blocks from the one or more reference frames with temporal blocks from the non-reference frames at corresponding spatial locations; andprocessing combined spatial blocks using a three-dimensional embedding network to compute latent features that capture temporal movements of spatial visual content in the one or more reference frames.

3. The method of any of claims 1-2, further comprising:applying a reference mask to the discrete reference token indices to compute masked discrete reference token indices;applying a non-reference mask to the discrete non-reference token indices to compute masked discrete non-reference token indices; anddetermining the discrete reference indices string based on the masked discrete reference token indices and the masked discrete non-reference token indices.

4. The method of any of claims 1-3, wherein applying the reference mask and the non-reference mask comprises masking out elements according to a predefined masking ratio, and wherein masked-out elements are set to a symbol and are not transmitted.

5. The method of any of claims 1-4, further comprising:receiving a second GoP comprising a second plurality of frames;selecting one or more reference frames from the second plurality of frames in the second GoP;determining second discrete reference token indices for the second GoP; determining second discrete non-reference token indices for the second GoP; determining difference token indices based on differences between the discrete reference token indices of the first GoP and the second discrete reference token indices of the second GoP; andencoding the difference token indices into a difference indices string for transmission toward the decoder.Atty. Docket No. 4502-85401 (6000748PCT02)6. The method of any of claims 1-5, further comprising:selecting a reference feature mask based on the embedded continuous reference latent feature;determining a tokenized reference latent using a continuous video tokenizer based on the embedded continuous reference latent feature and the reference feature mask;determining a tokenized non-reference latent using the continuous video tokenizer based on the embedded continuous non-reference latent feature and a non-reference feature mask; encoding the tokenized reference latent and the tokenized non-reference latent into a continuous latent string; andtransmitting the continuous latent string toward the decoder.

7. The method of any of claims 1-6, further comprising:decoding the continuous latent string to compute a recovered tokenized reference latent and a recovered tokenized non-reference latent;predicting a predicted tokenized reference latent and a predicted tokenized non-reference latent based on the recovered tokenized reference latent, the recovered tokenized non-reference latent, and predicted discrete token indices;determining a residual based on differences between the tokenized reference latent and the predicted tokenized reference latent and between the tokenized non-reference latent and the predicted tokenized non-reference latent;encoding the residual into a residual string; andtransmitting the residual string toward the decoder.

8. The method of any of claims 1-7, further comprising:generating a text description of content in the first GoP using a vision-language model; determining a text token mask based on the text description;determining a tokenized textual latent using a text tokenizer based on the text description and the text token mask;encoding the tokenized textual latent into a textual string; andtransmitting the textual string toward the decoder.Atty. Docket No. 4502-85401 (6000748PCT02)9. The method of any of claims 1-8, wherein the discrete video tokenizer comprises a first set of reference video tokens to capture spatial visual content and a second set of non-reference video tokens to capture temporal movements of visual content.

10. An encoder, comprising:a memory configured to store instructions; andone or more processors coupled to the memory and configured to execute the instructions to cause the encoder to perform a method according to any of claims 1-9.

11. A method implemented by a decoder, comprising:receiving a discrete reference indices string, a reference token mask, and a non-reference token mask;decoding the discrete reference indices string to compute masked discrete reference token indices and masked discrete non-reference token indices;refilling masked-out elements in the masked discrete reference token indices and the masked discrete non-reference token indices using a discrete token prediction model to predict discrete reference token indices and discrete non-reference token indices;reconstructing a group of pictures (GoP) comprising reference frames and non-reference frames based on predicted discrete reference token indices and predicted discrete non-reference token indices; andoutputting the reconstructed GoP.

12. The method of claim 11, further comprising:determining the predicted discrete reference token indices using spatial attention; and determining the predicted discrete non-reference token indices using a spatial -temporal attention mechanism based on the predicted discrete reference token indices.

13. The method of any of claims 11-12, further comprising:receiving a difference indices string;decoding the difference indices string to compute difference token indices; andAtty. Docket No. 4502-85401 (6000748PCT02)computing the masked discrete reference token indices and the masked discrete nonreference token indices based on the difference indices string, the reference token mask, and the non-reference token mask.

14. The method of any of claims 11-13, further comprising:receiving a continuous latent string;decoding the continuous latent string to compute a recovered tokenized reference latent and a recovered tokenized non-reference latent;predicting a predicted tokenized reference latent and a predicted tokenized non-reference latent based on the recovered tokenized reference latent, the recovered tokenized non-reference latent, and the predicted discrete reference token indices using a continuous token prediction model; andreconstructing the GoP based on the predicted tokenized reference latent, the predicted tokenized non-reference latent, the predicted discrete reference token indices, and the predicted discrete non-reference token indices.

15. The method of any of claims 11-14, further comprising:converting the recovered tokenized reference latent and the recovered tokenized non-reference latent into serialized tokens; andcomputing the predicted tokenized reference latent and the predicted tokenized non-reference latent based on the serialized tokens.

16. The method of any of claims 11-15, further comprising:receiving a residual string;decoding the residual string to compute a decoded residual;recovering a recovered tokenized reference latent and a recovered tokenized non-reference latent based on the predicted tokenized reference latent, the predicted tokenized non-reference latent, and the decoded residual; andreconstructing the GoP based on the recovered tokenized reference latent and the recovered tokenized non-reference latent.Atty. Docket No. 4502-85401 (6000748PCT02)17. The method of any of claims 11-16, further comprising:receiving a textual string and a text token mask;decoding the textual string to compute a tokenized textual latent;determining an embedded textual latent feature using a text embedding module based on the tokenized textual latent; andreconstructing the GoP based on the predicted discrete reference token indices, the predicted discrete non-reference token indices, and the embedded textual latent feature.

18. The method of any of claims 11-17, wherein reconstructing the GoP comprises using a basis decoder model paired with a discrete tokenizer to reconstruct basis output reference frames and basis output non-reference frames based on the predicted discrete reference token indices and the predicted discrete non-reference token indices.

19. The method of any of claims 11-18, wherein reconstructing the GoP further comprises:converting a recovered tokenized reference latent and a recovered tokenized non-reference latent into conditioning information; andapplying the conditioning information to interact with input and output of different layers in the basis decoder model to refine the reconstructed GoP.

20. A decoder, comprising:a memory configured to store instructions; andone or more processors coupled to the memory and configured to execute the instructions to cause the decoder to perform a method according to any of claims 11-19.

21. A computer program product comprising computer-executable instructions stored on a non-transitory computer-readable storage medium, the computer-executable instructions when executed by one or more processor of an encoder, cause the encoder to perform a method according to any of claims 1-9.

22. A computer program product comprising computer-executable instructions stored on a non-transitory computer-readable storage medium, the computer-executable instructions whenAtty. Docket No. 4502-85401 (6000748PCT02)executed by one or more processor of a decoder, cause the decoder to perform a method according to any of claims 11-19.

23. An encoder, comprising:a storage means configured to store instructions; andone or more processing means coupled to the storage means and configured to execute the instructions to cause the encoder to perform a method according to any of claims 1-9.

24. A decoder, comprising:a storage means configured to store instructions; andone or more processing means coupled to the storage means and configured to execute the instructions to cause the decoder to perform a method according to any of claims 11-19.

25. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause an encoder to perform operations according to any of claims 1-9.

26. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause an encoder to perform operations according to any of claims 11-19.

Images