Generating video representations using semantic attention

Abstract

Methods and systems for generating responses to queries about videos. For example, a final set of video tokens representing a video can be generated by processing a set of video tokens representing a video and a query relating to the video and the final set of video tokens and the query can be processed using a downstream neural network to generate a response to the query.

Description

[0001] GENERATING VIDEO REPRESENTATIONS USING SEMANTIC ATTENTION CROSS REFERENCE TO RELATED APPLICATION

[0002] This application claims priority to U.S. Provisional Application No. 63 / 721.399 filed on November 15. 2024. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application.

[0003] BACKGROUND

[0004] This specification relates to processing videos using machine learning models. As one example, neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to another layer in the network, e.g., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of weights.

[0005] SUMMARY

[0006] This specification describes a system implemented as computer programs on one or more computers that generates a representation of a video as a set of video tokens and then processes the representation to generate an output for a downstream task.

[0007] A video token, as used in this specification, is a portion of a video, i.e., that includes some or all of the pixels in one or more of the video frames of the video. For example, a video token can be a single video frame. As another example, a video token can be a patch or a region from a single video frame. As yet another example, a video token can be a spatio-temporal patch that includes regions from multiple different video frames within the video.

[0008] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.

[0009] Long video understanding presents challenges due to the inherent high computational complexity of processing long videos and the redundant temporal information depicted within the long video. An effective representation for long videos must process such redundancy efficiently while preserving essential contents for downstream tasks. However, existing techniques struggle with generating effective representations of long videos that are not excessively computationally expensive to compute or to store in memory.

[0010] This specification, on the other hand, describes a technique for generating a unified representation for long videos. To reduce computational complexity, long videos are decomposed into tokens representing distinct types of semantic entities, e.g., scenes, objects, and / or actions, allowing downstream models to operate on tokens representing a small number of entities rather than a large number of frames or pixels. That is, the video is processed to extract salient portions of the video for downstream processing. To further address redundancy, this specification uses an attention learning module that balances uery relevance with token diversity to generate a final number of tokens that is relatively small, eliminating redundancy, while still capturing the essential elements of the video required for downstream tasks. The resulting representations can be effectively used for any of a variety of downstream tasks, e.g., video question answering, temporal grounding tasks, action recognition tasks, and so on. Moreover, because the processing of the video to generate the final set of video tokens does not require any task-specific components, the same pipeline can be efficiently re-used for multiple different video processing tasks, with different relevant subsets of tokens being extracted for different queries (and for different tasks) through the attention learning module.

[0011] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below.

[0012] Other features, aspects, and advantages of the subject matter wall become apparent from the description, the drawings, and the claims. BRIEF DESCRIPTION OF DRAWINGS

[0013] FIG. 1A shows an example video processing system.

[0014] FIG. IB shows an example of the processing of an input video.

[0015] FIG. 2 is a flow diagram of an example process for generating a response to a query relating to a video.

[0016] FIG. 3 is a flow diagram of an example process for generating video tokens representing a video.

[0017] FIG. 4 is a flow diagram of an example process for generating a final set of video tokens representing a video.

[0018] FIG. 5 is a flow diagram of an example process for processing the final set of video tokens to generate a response to the query.

[0019] FIG. 6 shows another example of processing an input video.

[0020] Like reference numbers and designations in the various drawings indicate like elements.

[0021] DETAILED DESCRIPTION FIG. 1 A shows an example video processing system 100. The video processing system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

[0022] The system 100 generates a representation of a video as a set of video tokens and then processes the representation to generate an output for a downstream task.

[0023] A video token, as used in this specification, is a portion of a video, i.e., that includes some or all of the pixels in one or more of the video frames of the video. For example, a video token can be a single video frame. As another example, a video token can be a patch or a region from a single video frame. As yet another example, a video token can be a spatio-temporal patch that includes regions from multiple different video frames within the video.

[0024] More specifically, the system 100 receives a plurality of video frames 102 from a video 104.

[0025] The system also receives a query 106 relating to the video 104, e.g., a query' for the downstream task. The query 104 can be in any appropriate format, e.g.. can be a natural language query, can be an image or video query, or can be a multi-modal query that includes text and an image or a video. The system 100 processes the plurality of video frames 102 to generate a set of video tokens 110 characterizing the plurality of video frames.

[0026] As will be described below, the set of video tokens 110 generally includes multiple different types of tokens representing different types of semantic entities depicted within the video frames. As one example, the video tokens can include two or more of: scene tokens 112, object tokens 114, or action tokens 116.

[0027] A scene token 112 characterizes background context depicted in the plurality of video frames. That is, a scene token is a portion of the video frames that has been determined to depict background context, i.e., content that remains relatively constant throughout at least a portion of the video.

[0028] An object token 114 characterizes static properties of objects depicted in the first plurality of video frames. That is, an object token is a portion of the video frames that has been determined to depict a property of an object that remains static, i.e., does not change, across the time window spanned by the plurality of video frames. For example, an object token 114 can represent the appearance of the object.

[0029] An action token 116 characterizes moving elements depicted in the first plurality of video frames. That is, an action token is a portion of the video frames that has been determined to depict a moving element that moves during the time window7spanned by the plurality of video frames. For example, action tokens 116 can represent the motion of objects or other moving elements that is caused by motion of the object, motion of the camera, or both.

[0030] Generating these video tokens 110 will be described in more detail below7.

[0031] The system 100 then performs semantic attention to filter, using the query 106, a candidate set of video tokens 120 that includes the set of video tokens 110 characterizing the plurality of video frames to generate a final set of video tokens 130. For example, the candidate set of video tokens 120 can be filtered based upon a measure of a candidate video token’s relevancy to the query7106.

[0032] The system 100 processes at least the query7106 and the set of candidate tokens 129 that includes the plurality of video tokens 110 to generate a final set of video tokens 130 representing the video 104.

[0033] In some cases, the system 100 processes the entire video 104 in a single iteration. In these cases, the plurality of video frames 102 includes all of the video frames in the video 104 and the candidate set of video tokens 120 includes only the set of video tokens 110 characterizing the plurality of video frames 102. In other cases, the system 100 processes a respective segment of the video 104 at each of multiple iterations. In these cases, the candidate tokens 120 for a given segment can include video tokens determined by processing one or more preceding segments and video tokens 110 characterizing the segment. For example, the video tokens determined by processing the one or more preceding segments can include, for each of the one or more preceding segments, the final set of video tokens determined when the preceding segment was processed by the system 100.

[0034] That is, at each of multiple iterations, the system 100 can process the video frames in a corresponding segment of the video to generate a set of video tokens characterizing the segment and then perform semantic attention to update a current set of video tokens characterizing the video by filtering a candidate set of video tokens that includes the set of video tokens characterizing the segment and the current set of video tokens. For the first iteration, i.e., the first segment in the video, the current set of video tokens can be initialized to be empty .

[0035] For example, the video can be a streaming video and the system can perform a new iteration once a new segment has been received. As another example, the video may include a large number of video frames, so that processing the entire video in a single iteration is too computationally expensive. To account for this, the system can divide the video into segments and then perform the processing described above to, after processing the last segment in the video, arrive at a single fixed sized set of tokens that represents the entire video.

[0036] The system 100 then processes the final set of video tokens 130 and the query 106 using a downstream neural network 140 to generate a response 150 to the query 106, e.g., an output for a downstream task.

[0037] For example, the downstream task can be video question answering task and the response 150 can be a natural language or structured text response to the query. In this example, the query 106 can be a natural language query or an image or video query, e.g., an image or video of an object that is the subject of the query. Alternatively, the response 150 can be an image or video response, e.g., that identifies a portion of the video that is responsive to the query 106 or that edits a portion of the video as specified by the query 106.

[0038] As another example, the downstream task can be an object or action recognition task and the response 150 can include a respective score for each of a plurality of object categories or action categories, with the score for a category representing the likelihood that the video depicts an object or an action of the corresponding category.

[0039] As yet another example, the downstream task can be a video temporal grounding task, the query 106 can specify an object, e.g., can include an image of an object or a text description of the object, and the response 150 can identify' a time or a time interval when the specified object appears in the video or otherwise is depicted as having a property that is specified by the query.

[0040] Examples of downstream neural networks are described below with reference to FIGS. 5 and 6.

[0041] FIG. IB shows an example 170 of the processing of an input video.

[0042] In the example 170. the input to the system 100 is a "‘long’7video 172 that has many video frames. Thus, processing the entire video 172 at once in order to generate a response to a query' about the video 172 would be excessively computationally expensive.

[0043] Instead, to process the video 172, the system 100 first performs uniform sampling of the video 172 to divide the video into multiple segments 174.

[0044] The system 100 then performs a semantic decomposition 180 on each segment 174 to generate a set of video tokens 190 for the video 172.

[0045] In the example 170, the system 100 decomposes a segment 174 into four scene tokens S1-S4. six object tokens 01-06, and four action tokens A1-A4.

[0046] The system 100 can then use different ones of the video tokens 190 when responding to different queries about the video 172. That is, different aspects of the video will be relevant when responding to different queries. Therefore, rather than use the entire video 172 or even all of the video tokens 190 when responding to a given query, the system instead uses a query-aware attention learning module 192 to select the video tokens 190 that are relevant to responding to the query.

[0047] For example, for a query QI “What does the vlogger family find in visitor center?,” the system 100 determines that only the scene token SI and the object token 01 are relevant. The system 100 can then process the query’ QI, the scene token SI, and the object token 01 using a downstream neural network to generate a response to the query QI, i.e., rather than processing the raw video frames from the video or the entire set of video tokens 190.

[0048] For a more complex query' Q2 “What does the vlogger family do after buy ing fishing supplies?.” the system 100 determines that more tokens are relevant: the scene tokens SI and S2, the object token 02, and the action token Al. Similarly, for a query Q3 “What foods do the vlogger family eat the most here?” the system 100 determines that multiple object tokens 01-03 are relevant in addition to the action token A3.

[0049] As another example, for a query Q4 “What is the most common form of transportation there?” the system 100 determines that all three types of tokens are relevant: the scene token S4, the object token 06, and the action tokens A2 and A4.

[0050] Once the system identifies the relevant video tokens for a given query, the system 100 can process the relevant video tokens and the given query to generate the response to the queiy\ Thus, the system avoids needing to re-process the entire long video 170 each time a new query is received or needing to process all of the video tokens or the raw video frames using the downstream neural network when generating a response to a given query.

[0051] FIG. 2 is a flow diagram of an example process 200 for generating a response to a query7relating to a video. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a video processing system, e.g., the video processing system 100 of FIG. 1A, appropriately programmed in accordance with this specification, can perform the process 200.

[0052] The system receives a video that includes a first plurality' of video frames (step 202). As described above, the first plurality of video frames can be all of the frames from the video or can be the most-recent segment of the video frames from the video. If the first plurality of video frames are the most recent segment of the video frames the video, the system can have already performed a respective iteration of steps 204-208 for each preceding segment in the video.

[0053] The system receives a query relating to the video (step 204).

[0054] The system processes the first plurality of video frames to generate a set of video tokens characterizing the first plurality of video frames (step 206). As will be described below, the set of video tokens generally includes different types of tokens representing different types of semantic entities depicted within the video frames. As one example, the video tokens can include two or more of: scene tokens, object tokens, or action tokens.

[0055] Generating these different types of tokens is described in more detail below.

[0056] The system processes at least the query' and a set of candidate tokens that includes the first plurality of video tokens to generate a final set of video tokens representing the video (step 208). As described above, the set of candidate tokens can include only the first plurality of video tokens when the first plurality of video frames includes all of the frames from the video or when the first plurality of video frames are an initial segment of frames from the video.

[0057] When the first plurality of video frames are the most recent segment of the video, the set of candidate tokens can also include the final set of video tokens after performing step 208 for the preceding segment in the video.

[0058] As a particular example, the system can generate the final set of video tokens by filtering the candidate tokens in the set to generate a subset of candidate tokens that optimizes an objective that measures the diversity of the candidate tokens in the subset and a relevance of the candidate tokens in the subset to the query.

[0059] Generating the final set of video tokens in this manner is described in more detail below with reference to FIG. 4.

[0060] The system processes the final set of video tokens and the query using a dow nstream neural network to generate a response to the query (step 210).

[0061] Performing this processing is described in more detail below with reference to FIGS.

[0062] 5 and 6.

[0063] FIG. 3 is a flow diagram of an example process 300 for generating video tokens representing a video. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a video processing system, e.g., the video processing system 100 of FIG. 1 A, appropriately programmed in accordance with this specification, can perform the process 300.

[0064] The system receives a plurality of video frames (step 302).

[0065] The system then processes the set of video frames to generate multiple different types of video tokens.

[0066] For example, the system can process the plurality of video frames to generate a set of scene tokens characterizing background context depicted in the plurality of video frames (step 304).

[0067] As a particular example, each scene token can be a different video frame from the plurality of video frames. In this example, the system can generate the set of scene tokens by sampling a fixed number of frames from the first plurality of video frames (step 306), i.e., so that each scene token is a respective one of the sampled frames. As a particular example of this, the system can uniformly sample a fixed number of frames from the first plurality of video frames and designate each sampled frame as a respective scene token. In principle, any frame could serve as a scene token given that all frames will capture background context to some extent.

[0068] As another example, the system can process the plurality of video frames to generate a set of action tokens characterizing moving elements depicted in the plurality of video frames (step 308).

[0069] As a particular example, each action token can correspond to a respective object tracklet. An object tracklet generally includes a respective spatial region (or ‘'patch”) in multiple different video frames that have each been determined to depict the same object.

[0070] To generate the set of action tokens, the system can apply an object tracker to the first plurality of video frames to generate a set of dynamic object tracklets (step 310). The object tracker can generally be any appropriate object tracker, e.g., a statistical object tracker or a neural network system that has been trained to perform object tracking. For example, the object tracker can be a class-agnostic object tracker such as the one described in Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Radle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, ChaoYuan Wu, Ross Girshick, Piotr Dollar, and Christoph Feichtenhofer. SAM 2: Segment anything in images and videos, 2024.

[0071] The system can then generate the set of action tokens from the dynamic object tracklets (step 312), i.e., so that each action token corresponds to a respective one of the dynamic object tracklets.

[0072] Optionally, the system can filter or otherwise modify the set of tracklets generated by the object tracker as part of generating the set of action tokens. For example, the system can discard all tracklets shorter than a minimum length (in terms of number of frames). As another example, the system can split any tracklet that is longer than a maximum length into multiple, shorter tracklets that each have less than the maximum length.

[0073] In some cases, the system can set each object tracklet (optionally after filtering or modification or both) to be a respective action token.

[0074] In some other cases, the system can expand each object tracklet to generate the corresponding action token. For example, for each tracklet, the system can set the corresponding action token to be the spatial union of bounding boxes across the frames within the tracklet. This allows the action token to capture not only motion information but also the spatial movement of people and objects. As another example, the system can process the video frames to generate a set of object tokens characterizing static properties of objects depicted in the plurality of video frames (step 314).

[0075] For example, the system can identify, from the plurality of video frames, a set of key video frames (step 316).

[0076] The system can identify key video frames in any of a variety of ways. As one example, the system can randomly or uniformly sample frames from the plurality of video frames and designate each sample frame as a key frame. As another example, the system can designate each frame that has more than a threshold amount of change before the preceding frame as a key frame. The amount of change can be measured based on, e.g., an optical flow between the two frames, or a difference in object tracklets between the two frames.

[0077] For each key video frame, the system can apply a segmentation model to the key video frame to generate a set of object masks for the key video frame (step 318). The segmentation model can be any appropriate pre-trained segmentation model that can generate a segmentation output that assigns each pixel in an image to a respective object instance. For example, the segmentation model can be a class-agnostic segmentation model like the one described in Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Doll ar, and Ross B. Girshick. Segment anything. In ICCV. 2023.

[0078] The system can then generate, for each object instance to which at least a threshold number of pixels have been assigned by the model, an object mask that maintains the color values for each pixel assigned to the object instance and that (i) removes all other pixels or that (ii) sets all other pixels to a pre-determined color.

[0079] The system then generates the set of object tokens from the sets of object masks for the key video frames (step 320), e.g., by setting each object mask to be a different one of the object tokens.

[0080] Thus, as described above, each of the different types of video token captures a different aspect of the scene depicted in the video frames. Whilst generating tokens of all three described token types provides the most comprehensive representation, it will be appreciated that generating all three token types may not be required in all contexts. For example, as shown in FIG. IB, for some queries only two out of the three token types are selected to be part of the final token set. FIG. 4 is a flow diagram of an example process 400 for generating a final set of video tokens representing a video from a set of candidate video tokens. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a video processing system, e.g., the video processing system 100 of FIG. 1A, appropriately programmed in accordance with this specification, can perform the process 400.

[0081] As described above, the candidate tokens can include only the video tokens from the current plurality of video frames or can include the video tokens from the current plurality of video tokens and video tokens from one or more preceding segments.

[0082] For each candidate token, the system determines a respective relevance score that measures a relevance of the candidate token to the query (step 402).

[0083] For example, the system can map the candidate token to an embedding (token embedding) and also map the query to an embedding (query embedding). The system can perform the mapping for the query once and use the same query embedding for all of the candidate tokens. For example, the system can map the candidate tokens and the query to respective embeddings using a pre-trained multi-modal encoder neural network that has been trained to encode inputs from multiple modalities, e g., images or videos or both and text, into a shared embedding space. Examples of these include the ALIGN encoder and the BLIP-2 encoder.

[0084] The system then determines the relevance score to be a similarity between the token and query embedding, e g., a dot product, a cosine similarity or a Euclidean distance between the embeddings.

[0085] For each of a plurality of pairs of candidate tokens, the system determines a respective similarity score that measures a similarity between the candidate tokens (step 404). For example, the system can determine respective similarity scores for each possible pair of candidate tokens that includes two different ones of the candidate tokens.

[0086] For example, the system can map the first candidate token in the pair to an embedding (first embedding) and also map the second candidate token in the pair to an embedding (second embedding). The system can map each candidate token to a respective embedding and then re-use the same embedding when computing the respective similarity score for any pair that includes the candidate token.

[0087] For example, the system can generate the embeddings of the candidate tokens using the same pre-trained multi-modal encoder neural network described above. The system then determines the similarity score to be a similarity between the first and second embedding, e.g., a dot product, a cosine similarity or a Euclidean distance between the embeddings.

[0088] The system selects candidate tokens to be included in the final set of video tokens based on the respective relevance scores for the candidate tokens and the respective similarity scores for the pairs of candidate tokens (step 406).

[0089] For example, the system can select, as the final set of video tokens, a subset of candidate tokens that optimizes an objective that is based on the respective similarity scores for the candidate tokens in the subset and the respective relevance scores for the candidate tokens in the subset.

[0090] For example, the objective function FS(TS\TG, q) aims to find the optimal fixed-size subset of tokens T* that maximizes the balance between query relevance and token diversity. Here, Tcis the set of candidate tokens and q is the query.

[0091] For example, the objective can be represented as:

[0092] Ts* = argmax

[0093] <

[0094]

[0095] In more detail, the optimization involves solving a subset selection problem to find T*, maximizing the objective function FS(TS\TG, q), which is composed of two main terms: query relevance !?(■) and token diversity S(-).

[0096] The goal of the optimization is to find T*, which denotes the optimal subset of tokens that maximizes the objective function FS(TS\TC, q).

[0097] The optimization process iterates over Ts, which is the fixed-size subset of tokens being selected, where Ts must be a subset of TG (TSCTG).

[0098] The objective function FS(TS\TC, q) is maximized based on two competing criteria: query relevance and token diversity.

[0099] The query relevance term uses R(ts,q where h is a single visual token within the selected subset Ts and q denotes the query being asked by the downstream understanding task. This relevance is specifically computed as described above, i.e., using embeddings of the token and the query. The first term in FS(TS\TC, q) maximizes this relevance between the selected tokens and the query.

[0100] The token diversity term involves where and tj represent two distinct tokens within the subset Ts. S(-) calculates the cosine similarity or other appropriate similarity between respective embeddings of the tokens in the pair. The goal of the second term is to enforce token diversity by minimizing the token pairwise similarity.

[0101] Finally, a is a hyper-parameter used to balance between the query-relevance term and the token-diversity term. For example, a can be set to .2, .4, .6, .8, or 0.9.

[0102] FIG. 5 is a flow diagram of an example process 500 for processing the final set of video tokens to generate a response to the query. For convenience, the process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a video processing system, e.g., the video processing system 100 of FIG. 1 A, appropriately programmed in accordance with this specification, can perform the process 500.

[0103] The system processes each token in the final set of video tokens using a respective encoder neural network to generate an encoded representation of the video token (step 502).

[0104] For example, the system can use a different encoder neural network for each type of video token or can use the same encoder neural network for all of the video tokens. The encoder neural network for any given type of video token can generally be any appropriate vision encoder neural network, e.g., a vision Transformer neural network, a convolutional neural network, or a neural network that includes both transformer and convolutional layers.

[0105] The system generates an input to the downstream neural network from the encoded representations of the video tokens in the final set of video tokens and the query (step 504).

[0106] For example, the downstream neural network can be a multi-modal language model (MLM) neural network. In this example, the system can generate the input by embedding the query into a set of input tokens, e.g., by using a tokenizer or embedding subnetwork of the MLM neural network. The system can then project each encoded representation to have a dimensionality that matches the dimensionality of the input tokens processed by the MLM neural network, e.g., using a projection neural network. Thus, the input can be a sequence that includes the projected encoded representations and the input tokens representing the query.

[0107] As another example, the downstream neural network can be a vision head configured to perform a downstream task. In this example, the system can directly process the encoded representation as part of the input to the downstream neural network. One example of a vision head is described below with reference to FIG. 6.

[0108] The system processes the input using the downstream neural network to generate the response to the query (step 506).

[0109] FIG. 6 shows another example 600 of processing an input video 602. In the example 600, the system receives the input video 602 and performs a semantic decomposition to generate a set of video tokens that includes a set of action tokens 606, a set of object tokens 608, and a set of scene tokens 610.

[0110] The system then fdters the set of video tokens using the relevance 620 of each video token to a query 604 and the diversity' 630 of pairs of video tokens through attention learning to generate a final set of video tokens.

[0111] The system then processes each video token in the final set through a respective encoder neural network to generate a respective encoded representation 640 of each of the video tokens.

[0112] The system then processes the encoded representations 640 and the query 604 using a downstream neural network.

[0113] For example, the downstream neural network can be a vision head 650. In the example 600, the vision head includes a text encoder that processes the query, a cross-modal fusion neural network that processes the output of the text encoder and the encoded representations of the video tokens to generate a fused representation of the video tokens and the query, and a moment decoder that processes the fused representation to generate the response to the query'.

[0114] As another example, the downstream neural netw ork can be a multi-modal language model neural network (MLLM) 660. In this example, the system tokenizes the query using a text tokenizer and applies a multi-layer perceptron (MLP) or other projection neural network to generate tokens representing the query and the video tokens, and then processes the tokens using the MLLM to generate the response to the query' .

[0115] In some cases, the system maintains multiple different downstream neural networks, e.g., a MLLM and one or more vision heads, or two or more vision heads for different tasks, and then selects the appropriate downstream neural netw ork based on the task being performed. Because generating the final set of vision tokens requires only pre-trained components and does not require any task-specific training, the same final set of vision tokens can be used to perform multiple different tasks.

[0116] Prior to performing tasks using video tokens, the system or another training system trains the downstream neural network(s) and, when used, the encoder neural netw orks used to encode video tokens to generate the downstream input. When the downstream neural network is a vision head, the system can train the downstream neural netw ork from scratch or can fix the text encoder and train the remaining components from scratch. When the downstream neural network is a multi-modal language model neural network (MLLM), the system can train only the projection neural network that projects the encoded representation to generate the input to the downstream neural network or can train the projection neural network while fine-tuning the MLLM.

[0117] For example, the system can train the encoder neural networks that encode the video tokens jointly with the one or more downstream neural networks on respective objectives for each of one or more video processing tasks, e.g., for any of the tasks described above.

[0118] This specification uses the term “configured’7in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions. Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0119] The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g.. an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

[0120] A computer program, which may also be referred to or described as a program, software, a software application, an app, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0121] In this specification, the term “database” is used broadly to refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. Thus, for example, the index database can include multiple collections of data, each of which may be organized and accessed differently.

[0122] Similarly, in this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or more computers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0123] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0124] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read only memory’ or a random-access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory’ can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0125] Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory' devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory’ devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks; and CD ROM and DVD-ROM disks.

[0126] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory’ feedback, e.g.. visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0127] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

[0128] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework or a Jax framework.

[0129] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0130] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e g., a result of the user interaction, can be received at the server from the device.

[0131] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable sub-combination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a sub-combination or variation of a sub-combination. Similarly, while operations are correspond toed in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0132] This specification also provides the subject-matter of the following clauses:

[0133] Clause 1. A method performed by one or more computers, the method comprising: receiving a video comprising a first plurality of video frames;

[0134] receiving a query relating to the video;

[0135] processing the first pl urality of video frames to generate a set of video tokens characterizing the first plurality’ of video tokens;

[0136] processing at least the query and a set of candidate tokens that comprises the first plurality of video tokens to generate a final set of video tokens representing the video; and processing the final set of video tokens and the query using a dow nstream neural network to generate a response to the query.

[0137] Clause 2. The method of clause 1, wherein the set of video tokens characterizing the first plurality of video frames comprises a set of scene tokens characterizing background context depicted in the first plurality of video frames.

[0138] Clause 3. The method of clause 2, wherein processing the first plurality of video frames to generate a set of video tokens characterizing the first plurality of video tokens comprises:

[0139] generating the set of scene tokens by sampling a fixed number of frames from the first plurality of video frames.

[0140] Clause 4. The method of clause 3, wherein sampling a fixed number of frames from the first plurality of video frames comprises uniformly sampling the fixed number of frames from the first plurality of video frames.

[0141] Clause 5. The method of any preceding clause, wherein the set of video tokens characterizing the first plurality of video frames comprises a set of action tokens characterizing moving elements depicted in the first plurality of video frames. Clause 6. The method of clause 5, wherein processing the first plurality of video frames to generate a set of video tokens characterizing the first plurality of video tokens comprises:

[0142] generating the set of action tokens, comprising:

[0143] applying an object tracker to the first plurality of video frames to generate a set of dynamic object tracklets; and

[0144] generating the set of action tokens from the dynamic object tracklets.

[0145] Clause 7. The method of any preceding clause, wherein the set of video tokens characterizing the first plurality of video frames comprises a set of object tokens characterizing static properties of objects depicted in the first plurality of video frames. Clause 8. The method of clause 7, wherein processing the first plurality of video frames to generate a set of video tokens characterizing the first plurality of video tokens comprises:

[0146] generating the set of object tokens, comprising:

[0147] identifying, from the first plurality of video frames, a set of key video frames;

[0148] for each key video frame, applying a segmentation model to the key video frame to generate a set of object masks for the key video frame; and

[0149] generating the set of video tokens from the sets of object masks for the key¬ video frames.

[0150] Clause 9. The method of any preceding clause, wherein the video frame includes only the first plurality of video frames and the set of candidate tokens includes only the first plurality of video tokens.

[0151] Clause 10. The method of any one of clauses 1-8, wherein:

[0152] the video frame includes a sequence of segments that each comprise a respective plurality of video frames,

[0153] the first plurality of video frames are in a last segment in the sequence of segments, and

[0154] the candidate tokens include a current set of video tokens generated from one or more segments preceding the last segment in the sequence of segments. Clause 11. The method of any preceding clause wherein processing at least the query and a set of candidate tokens that comprises the first plurality of video tokens to generate a final set of video tokens representing the video comprises:

[0155] for each candidate token, determining a respective relevance score that measures a relevance of the candidate token to the query;

[0156] for each of a plurality of pairs of candidate tokens, determining a respective similarity score that measures a similarity between the candidate tokens; and

[0157] selecting candidate tokens to be included in the final set of video tokens based on the respective relevance scores for the candidate tokens and the respective similarity scores for the pairs of candidate tokens.

[0158] Clause 12. The method of clause 11, wherein selecting candidate tokens to be included in the final set of video tokens based on the respective relevance scores for the candidate tokens and the respective similarity scores for the pairs of candidate tokens comprises: selecting, as the final set of video tokens, a subset of candidate tokens that optimizes an objective that is based on the respective similarity scores for the candidate tokens in the subset and the respective relevance scores for the candidate tokens in the subset.

[0159] Clause 13. The method of clause 11 or clause 12, wherein determining a respective relevance score that measures a relevance of the candidate token to the query7comprises: mapping the candidate token to a first embedding;

[0160] mapping the query to a second embedding; and

[0161] determining a similarity’ between the first and second embedding.

[0162] Clause 14. The method of any one of clauses 11-13, wherein determining a respective similarity score that measures a similarity' between the candidate tokens comprises:

[0163] mapping a first candidate token in the pair to a first embedding;

[0164] mapping a second candidate token in the pair to a second embedding; and determining a similarity’ between the first and second embedding.

[0165] Clause 15. The method of any preceding clause wherein processing at least the query and a set of candidate tokens that comprises the first plurality of video tokens to generate a final set of video tokens representing the video comprises:

[0166] filtering the candidate tokens in the set to generate a subset of candidate tokens that optimizes an objective that measures a diversity of the candidate tokens in the subset and a relevance of the candidate tokens in the subset to the query'. Clause 16. The method of any preceding clause wherein processing the final set of video tokens and the query using a downstream neural network to generate a response to the query comprises:

[0167] processing each token in the final set of video tokens using a respective encoder neural network to generate an encoded representation of the video token;

[0168] generating an input to the downstream neural network from the encoded representations of the video tokens in the final set of video tokens and the query; and processing the input using the downstream neural network to generate the response to the query'.

[0169] Clause 17. The method of any preceding clause, wherein the downstream neural network is a multi-modal language model (MLM) neural network.

[0170] Clause 18. The method of clause 17, when dependent on clause 16, wherein generating an input to the downstream neural network from the encoded representations of the video tokens in the final set of video tokens and the query' comprises projecting each encoded representation to have a dimensionality’ that matches a dimensionality of tokens processed by the MLM neural network.

[0171] Clause 19. The method of any preceding clause, wherein the downstream neural netw ork is a vision head configured to perform a downstream task.

[0172] Clause 20. A system comprising:

[0173] one or more computers; and

[0174] one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the respective operations of the method of any one of clauses 1-19.

[0175] Clause 21. One or more computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the respective operations of the method of any’ one of clauses 1-19.

[0176] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes correspond toed in the accompanying figures do not necessarily’ require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0177] What is claimed is:

Claims

CLAIMS1. A method performed by one or more computers, the method comprising:receiving a video comprising a first plurality of video frames;receiving a query relating to the video;processing the first plurality of video frames to generate a set of video tokens characterizing the first plurality of video frames;processing at least the query' and a set of candidate tokens that comprises the first plurality’ of video tokens to generate a final set of video tokens representing the video; and processing the final set of video tokens and the query' using a downstream neural netyvork to generate a response to the query.

2. The method of claim 1, wherein the set of video tokens characterizing the first plurality of video frames comprises a set of scene tokens characterizing background context depicted in the first plurality of video frames.

3. The method of claim 2, wherein processing the first plurality of video frames to generate a set of video tokens characterizing the first plurality of video tokens comprises:generating the set of scene tokens by sampling a fixed number of frames from the first plurality of video frames.

4. The method of claim 3. wherein sampling a fixed number of frames from the first plurality of video frames comprises uniformly sampling the fixed number of frames from the first plurality of video frames.

5. The method of any preceding claim, wherein the set of video tokens characterizing the first plurality of video frames comprises a set of action tokens characterizing moving elements depicted in the first plurality of video frames.

6. The method of claim 5, wherein processing the first plurality of video frames to generate a set of video tokens characterizing the first plurality of video tokens comprises:generating the set of action tokens, comprising:applying an object tracker to the first plurality' of video frames to generate a set of dynamic object tracklets; andgenerating the set of action tokens from the dynamic object tracklets.

7. The method of any preceding claim, wherein the set of video tokens characterizing the first plurality of video frames comprises a set of object tokens characterizing static properties of objects depicted in the first plurality of video frames.

8. The method of claim 7, wherein processing the first plurality of video frames to generate a set of video tokens characterizing the first plurality of video tokens comprises:generating the set of object tokens, comprising:identifying, from the first plurality of video frames, a set of key video frames;for each key video frame, applying a segmentation model to the key video frame to generate a set of object masks for the key video frame; andgenerating the set of video tokens from the sets of object masks for the key video frames.

9. The method of any preceding claim, wherein the video includes only the first plurality of video frames and the set of candidate tokens includes only the first plurality of video tokens.

10. The method of any one of claims 1-8, wherein:the video includes a sequence of segments that each comprise a respective plurality of video frames,the first plurality of video frames are in a last segment in the sequence of segments, andthe candidate tokens include a current set of video tokens generated from one or more segments preceding the last segment in the sequence of segments.

11. The method of any preceding claim wherein processing at least the query and a set of candidate tokens that comprises the first plurality of video tokens to generate a final set of video tokens representing the video comprises:for each candidate token, determining a respective relevance score that measures a relevance of the candidate token to the query;for each of a plurality of pairs of candidate tokens, determining a respective similarity score that measures a similarity between the candidate tokens; andselecting candidate tokens to be included in the final set of video tokens based on the respective relevance scores for the candidate tokens and the respective similarity scores for the pairs of candidate tokens.

12. The method of claim 11, wherein selecting candidate tokens to be included in the final set of video tokens based on the respective relevance scores for the candidate tokens and the respective similarity' scores for the pairs of candidate tokens comprises:selecting, as the final set of video tokens, a subset of candidate tokens that optimizes an objective that is based on the respective similarity' scores for the candidate tokens in the subset and the respective relevance scores for the candidate tokens in the subset.

13. The method of claim 11 or claim 12, wherein determining a respective relevance score that measures a relevance of the candidate token to the query comprises:mapping the candidate token to a first embedding;mapping the query to a second embedding; anddetermining a similarity between the first and second embedding.

14. The method of any one of claims 11-13, wherein determining a respective similarity score that measures a similarity between the candidate tokens comprises:mapping a first candidate token in the pair to a first embedding;mapping a second candidate token in the pair to a second embedding; and determining a similarity’ between the first and second embedding.

15. The method of any preceding claim wherein processing at least the query and a set of candidate tokens that comprises the first plurality of video tokens to generate a final set of video tokens representing the video comprises:filtering the candidate tokens in the set to generate a subset of candidate tokens that optimizes an objective that measures a diversity of the candidate tokens in the subset and a relevance of the candidate tokens in the subset to the query’.

16. The method of any preceding claim wherein processing the final set of video tokens and the query' using a downstream neural network to generate a response to the query' comprises:processing each token in the final set of video tokens using a respective encoder neural netw ork to generate an encoded representation of the video token;generating an input to the downstream neural network from the encoded representations of the video tokens in the final set of video tokens and the query; and processing the input using the downstream neural network to generate the response to the query.

17. The method of any preceding claim, wherein the downstream neural network is a multi-modal language model (MLM) neural network.

18. The method of claim 17, when dependent on claim 16, wherein generating an input to the downstream neural network from the encoded representations of the video tokens in the final set of video tokens and the query comprises projecting each encoded representation to have a dimensionality that matches a dimensionality of tokens processed by the MLM neural netw ork.

19. The method of any preceding claim, wherein the downstream neural network is a vision head configured to perform a downstream task.

20. A system comprising:one or more computers; andone or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform the respective operations of the method of any one of claims 1-19.

21. One or more computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform the respective operations of the method of any one of claims 1-19.

Images