Modulated video token compression via causal compression module with positional information injection

Abstract

Described is a computer apparatus (900) configured to: obtain a plurality of latent frames (103) of the video; and compress (104) a plurality of tokens of the latent frames (103) to generate a plurality of compressed tokens (105) for input into a vision language model (VLM) (109); wherein the compression (104) comprises combining tokens along a temporal dimension and along a spatial dimension. In this way, the compression (104) may be greater than if a single dimension is used, which may reduce the workload of the VLM (109).

Images