Generating videos using a generative model conditioned on motion conditioning data

Abstract

Methods, systems, and apparatuses, including computer programs encoded on computer storage media, for generating a video using a generative model. The method includes obtaining conditioning data that includes a conditioning video frame and motion conditioning data. The motion conditioning data can be generated from user motion prompts, such as those defining object movements or camera trajectories. In other aspects, the motion conditioning data is used as an input to a machine learning model to generate other prediction outputs, such as a video classification or an action selection output for controlling an agent.

Description

[0001] Atorney Docket No. 45288-0546WO1

[0002] GENERATING VIDEOS USING A GENERATIVE MODEL CONDITIONED ON MOTION CONDITIONING DATA BACKGROUND

[0003] This specification relates to processing data using machine learning models.

[0004] Machine learning models receive an input and generate an output, e.g., a predicted output, based on the received input. Some machine learning models are parametric models and generate the output based on the received input and on values of the parameters of the model.

[0005] Some machine learning models are deep models that employ multiple layers of models to generate an output for a received input. For example, a deep neural network is a deep machine learning model that includes an output layer and one or more hidden layers that each apply a non-linear transformation to a received input to generate an output.

[0006] SUMMARY

[0007] This specification generally describes a system implemented as computer programs on one or more computers in one or more locations that can generate a video by processing conditioning data using a generative model, where the conditioning data includes: (i) a conditioning video frame for the video, and (ii) motion conditioning data. The motion conditioning data defines a set of tracks that are each associated with a respective point in the conditioning video frame and that define a spatio-temporal trajectory of the point through (at least part of) the video. The generative model is configured through training to cause the video to have characteristics that are consistent with the conditioning data. For example, the motion conditioning data can correspond to a movement that an object in the video frame should perform in the video. The generative model can then generate a video in which the object performs the movement.

[0008] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.

[0009] The system described in this specification can process conditioning data that includes: (i) a conditioning video frame in a video, and (ii) motion conditioning data, using a generative model to generate a video that includes the provided conditioning video frame and that shows motion that is consistent with the motion conditioning data. The system provides a unified framework for performing a wide spectrum of motion-conditioned generative video modeling tasks such as controlling the movement of objects, controlling the path of a camera that defines a perspective from which the video is captured, drag-based editing, and so forth. The system can thus replace a large of number of specialized individual models that perform Atorney Docket No. 45288-0546WO1

[0010] single, narrow generative modeling tasks with a single generalist system that can perform many or all of these tasks with performance comparable to or better than specialized models.

[0011] Innovative aspects of the system described in this specification provide technical solutions to a number of technical problems that arise in training and deploying a generative model for generating videos conditioned on motion conditioning data.

[0012] First, the system can represent the motion conditioning data that is used to condition the generative model in the form of ‘"tracks” that are each associated with a point in the conditioning video frame and that define a spatio-temporal trajectory of the point through (at least part of) the video. The system can encode the tracks in the form of a spatio-temporal volume that includes a temporal dimension and a set of spatial dimensions (e.g., two orthogonal dimensions, such as height and width) matching those of the video to be generated by the generative model, and then process that spatio-temporal volume using the generative model. The tracks (and their subsequent encoding in a spatio-temporal volume) provide a technical solution to the technical problem of how to flexibly and expressively represent any type of motion that could be exhibited in the video to be generated by the generative model. Encoding the tracks in a spatio-temporal volume having a same dimensionality as the video to be generated by the generative model allows the motion conditioning data to be easily integrated into existing architectures for generative models for videos, e.g., without requiring complex intermediate operations to translate the motion conditioning data into a format consistent with the dimensionality of the video.

[0013] Second, the system can provide a user interface that enables a user to input '"motion prompts” that allow the user to control motion in the video to be generated by the generative model. The system can translate the motion prompts provided by the user by way of the user interface into the form of tracks that can be encoded into a corresponding spatio-temporal volume and then used for conditioning the generative model. The system can adaptively encode motion prompts provided by users into corresponding tracks (and spatio-temporal volumes) in a manner that allows a user to control many aspects of the motion in the video, e.g., including the movements of individual objects, the path of the camera that defines the perspective from which the video is generated, and so forth. The described user interface addresses the technical problem of providing a unified solution for many disparate motion control tasks under a single framework, in particular, by allowing users to input motion prompts which are then processed for conversion into tracks (and spatio-temporal volumes) that can be used to solve a wide variety of generative motion control tasks. Atorney Docket No. 45288-0546WO1

[0014] Third, the system provides a technical solution to the problem of training data scarcity that arises when training motion-conditioned generative video models. More specifically, the system can process a database of existing videos to generate large numbers of diverse training examples for training the generative model by an automated process. In particular, given a video, the system can apply a point tracking algorithm to generate a respective track for each point in a selected video frame that defines a spatio-temporal trajectory of the point through the video. The system can then select a subset (or all) of these tracks for inclusion in motion conditioning data in a training example that also includes the selected video frame of the video (as additional conditioning data) and the video itself (as a prediction target). The system can thus leverage the representation of the motion conditioning data in terms of tracks and the existence of accurate point tracking algorithms to automatically generate large numbers of training examples for training the generative model to perform motion-conditioned video generation.

[0015] This specification also describes methods for processing a model input that includes motion conditioning data using a machine learning model to generate a prediction output. The motion conditioning data includes one or more tracks, where each track is associated with a point in a video frame and defines a spatio-temporal trajectory of the point through a plurality of time steps in the video. This motion conditioning data can be processed by the model as a representation, for example, as a spatio-temporal volume.

[0016] These methods enable various technical applications. For example, the prediction output can define a classification of the video into one or more classes, such as in an action classification, scene classification, or object detection task. In some cases, the machine learning model can be an action selection neural network used to control an agent (e.g., an electromechanical agent, such as a robot), where the model processes the motion conditioning data (e.g., from a video of the agent's environment) to generate an action selection output. The action selection output can then be used to control the agent, e.g., as an action performed by the agent interacting with a real-world environment to perform a predefined task.

[0017] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0018] Existing machine learning models for tasks like video analysis, classification, object detection, or agent control often struggle to efficiently and accurately interpret complex motion. Models that process raw video frames must learn motion representations from scratch, which is computationally intensive and can be sensitive to noise. Other common motion representations, like optical flow, are often limited to short-term motion between Atorney Docket No. 45288-0546WO1

[0019] adjacent frames and cannot effectively represent long-range trajectories or handle occlusions. This limits the model's ability to understand nuanced actions, predict future events, or make informed decisions based on the dynamic behavior of objects in a scene.

[0020] This specification describes techniques that can address the aforementioned challenges. That is, this specification describes techniques for processing a model input that includes motion conditioning data using a machine learning model to generate a prediction output. The motion conditioning data includes one or more tracks, with each track associated with a point in a video frame and defining a spatio-temporal trajectory of that point through a plurality of time steps. These techniques can be applied to various tasks. For example, the machine learning model can be an action selection neural network that processes the model input to generate an action selection output, which is then used to select an action to be performed by an agent. In other examples, the prediction output defines a classification of the video into one or more classes, such as for an action classification task, a scene classification task, or an object detection task.

[0021] By processing motion conditioning data that includes one or more tracks defining spatio-temporal trajectories, the described techniques improve the functioning of the machine learning model by providing it with an explicit, robust, and rich representation of movement. This allows the model to directly perceive how objects are moving without having to leam motion patterns from raw pixel data, which can be more computationally efficient (e g., saving computational memory and processing resources). This track-based representation is capable of capturing complex, long-range motion and handling occlusions via visibility flags.

[0022] By generating a representation of the motion conditioning data as a spatio-temporal volume, the described techniques convert the set of trajectories into a structured, dense tensor format. This spatio-temporal volume can be generated with dimensions (e.g.. time, height, idth, and channels) that are directly compatible with existing and powerful machine learning architectures, such as 3D convolutional neural networks or video transformers, allowing the motion data to be easily integrated as an input.

[0023] By providing this explicit motion data as an input to an action selection neural network, the described techniques can enable an agent to make more informed and reactive decisions. The agent's action selection can be based not just on a static observation of the environment, but on a direct understanding of how objects within that environment are moving, which is critical for tasks requiring prediction, interaction, or navigation.

[0024] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Atorney Docket No. 45288-0546WO1

[0025] According to a first aspect there is provided a method performed by one or more computers that includes generating a video using a generative model, comprising: obtaining conditioning data specifying desired characteristics of the video to be generated by the generative model, wherein the conditioning data comprises: a conditioning video frame for a time step in the video; and motion conditioning data that comprises one or more tracks, wherein each track is associated with a respective point in the conditioning video frame and defines a spatio-temporal trajectory of the point through a plurality of time steps in the video; and processing a representation of the conditioning data using the generative model to generate the video; and outputting the video generated using the generative model.

[0026] In some implementations, the method further includes processing the conditioning data to generate the representation of the conditioning data that is processed using the generative model, comprising: generating a representation of the motion conditioning data as a spatio-temporal volume.

[0027] In some cases, the spatio-temporal volume includes a temporal dimension indexing time steps of the video and a plurality of spatial dimensions corresponding to spatial dimensions of the video.

[0028] In some cases, generating the representation of the motion conditioning data as a spatio-temporal volume comprises: initializing the spatio-temporal volume with default values; and updating the spatio-temporal volume to encode each track in the one or more tracks, comprising, for each track: determining a respective embedding for the track; and updating each location in the spatio-temporal volume that corresponds to a spatial location and time step visited by the track to include the embedding of the track.

[0029] In some cases, for each track, determining the embedding for the track comprises: assigning a unique and randomly chosen non-zero embedding to the track.

[0030] In some cases, initializing the spatio-temporal volume with default values comprises: initializing each entry in the spatio-temporal volume to have value zero.

[0031] In some implementations, the spatio-temporal volume includes a channel dimension. In some cases, the motion conditioning data defines, for each time step in each track, whether the track is off-screen or occluded at the time step; and for each track, updating each location in the spatio-temporal volume that corresponds to a spatial location and time step visited by the track to include the embedding of the track comprises, for any time step where the track is off-screen or occluded: modify ing the embedding included in the spatio-temporal volume at the spatial location and time step to reflect that the track is off-screen or occluded at the time step. Atorney Docket No. 45288-0546WO1

[0032] In some cases, modifying the embedding included in the spatio-temporal volume at the spatial location and time step comprises modifying the embedding included in the spatiotemporal volume at the spatial location and time step to be a zero-embedding with a zero in each entry.

[0033] In some cases, the motion conditioning data defines, for each time step in each track, an adherence value that defines a degree to which the video should adhere to the track at the time step; and for each track, updating each location in the spatio-temporal volume that corresponds to a spatial location and time step visited by the track to include the embedding of the track comprises: modifying the embedding included in the spatio-temporal volume at the spatial location and time step based on a corresponding adherence value for the track at the time step.

[0034] In some cases, modifying the embedding included in the spatio-temporal volume at the spatial location and time step based on the corresponding adherence value for the track at the time step comprises: modifying the embedding included in the spatio-temporal volume at the spatial location and time step to be a combination of: (i) the embedding for the track, and (ii) an embedding representing the adherence value for the track at the time step.

[0035] In some implementations, the motion conditioning data comprises a respective track for at least 10%, or at least 20%, or at least 30%, or at least 40%, or at least 50%, or at least 60%, or at least 70%, or at least 80%, or at least 90% of points in the conditioning video frame.

[0036] In some implementations, the motion conditioning data comprises a respective track for every7point in the conditioning video frame.

[0037] In some implementations, the generative model is a generative diffusion model. In some cases, the generative model has been trained on a set of training examples; and each training example comprises: (i) conditioning data comprising a conditioning video frame and motion conditioning data, and (ii) a target video that is consistent with the conditioning data.

[0038] In some cases, for a plurality of the training examples, the motion conditioning data for the training example has been generated by performing operations comprising: processing the target video of the training example using a point tracking algorithm to generate one or more tracks for inclusion in the motion conditioning data of the training example.

[0039] In some cases, for a plurality of the training examples, generating the motion condition data for the training example further comprises, for one or more tracks included in the motion conditioning data: selecting a respective adherence value for one or more time Atorney Docket No. 45288-0546WO1

[0040] steps in the track; and modifying the track by, for each time step in the track, randomly displacing a spatial location of the track at the time step by a number of pixels that is dependent upon the adherence value for the time step.

[0041] In some cases, the method further includes obtaining a motion prompt from a user by way of a user interface; and processing the motion prompt obtained from the user to generate the motion conditioning data for the generative model.

[0042] In some cases, obtaining the motion prompt from the user by way of the user interface comprises: displaying, to the user and on the user interface, an interactive user interface element that displays the conditioning video frame in the video; and receiving a user input from the user, by way of user interaction with the interactive user interface element, that defines desired movements of one or more objects shown in the conditioning video frame over a plurality of time steps in the video.

[0043] In some cases processing the motion prompt obtained from the user to generate the motion conditioning data for the generative model comprises: processing the user input defining the desired movements of the one or more objects shown in the conditioning video frame over the plurality of time steps in the video to generate a plurality of object control tracks associated with points in the conditioning video frame; where the plurality of object control tracks are included in the motion conditioning data.

[0044] In some cases, outputting the video generated using the generative model comprises: outputting only a final video frame in the video generated using the generative model in order to perform drag-based image editing of an image represented by the conditioning video frame.

[0045] In some cases, obtaining the motion prompt from the user by way of the user interface comprises: receiving a user input from the user, by way of user interaction with the interactive user interface element, that defines a trajectory of camera poses; and processing the motion prompt obtained from the user to generate the motion conditioning data for the generative model comprises: processing the trajectory of camera poses to generate motion conditioning data that causes the video generated by the generative model when conditioned on the motion conditioning data to be captured from a perspective of a camera that follows the trajectory of camera poses.

[0046] In some implementations, processing the trajectory of camera poses to generate motion conditioning data that causes the video generated by the generative model when conditioned on the motion condition data to be captured from the perspective of the camera that follows the trajectory of camera poses comprises: processing the conditioning video Atorney Docket No. 45288-0546WO1

[0047] frame using a depth estimation model to generate a point cloud representing a geometry of a scene shown in the conditioning video frame; projecting the point cloud onto each camera pose in the trajectory of camera poses to generate a plurality of camera control tracks associated with points in the conditioning video frame; where the plurality of camera control tracks are included in the motion conditioning data.

[0048] In some cases, processing the motion prompt obtained from the user to generate the motion conditioning data for the generative model comprises: processing the motion prompt obtained from the user to generate: (i) a plurality of object control tracks for controlling movement of objects in the video, and (ii) a plurality of camera control tracks for controlling a camera perspective from which the video is captured; and generating a plurality of combined tracks for inclusion in the motion conditioning data by combining the object control tracks and the camera control tracks.

[0049] In some cases, outputting the video comprises storing the video in a memory, or presenting the video on a display of a user device, or transmitting the video over a data communication network.

[0050] In some cases, the generative model has been configured through training to cause the video to have characteristics that are consistent with the conditioning data.

[0051] In some cases, the user input defines a manipulation of a geometric primitive associated with an object in the conditioning video frame, and processing the user input comprises translating the manipulation of the geometric primitive into the plurality of object control tracks.

[0052] In some cases, obtaining the conditioning data comprises: processing a source video to generate the one or more tracks comprising the motion conditioning data; and wherein the conditioning video frame is obtained from a source that is distinct from the source video, such that the video generated using the generative model included transferred motion from the source video to the conditioning video frame.

[0053] According to a second aspect there is provided a method performed by one or more computers that includes obtaining motion conditioning data that comprises one or more tracks, wherein each track is associated with a point in a video frame of a video and defines a spatio-temporal trajectory of the point through a plurality of time steps in the video; processing a model input that comprises the motion conditioning data using a machine learning model to generate a prediction output; and providing the prediction output.

[0054] In some cases of the second aspect, the method further comprises generating a representation of the motion conditioning data as a spatio-temporal volume; and where the Atorney Docket No. 45288-0546WO1

[0055] model input to the machine learning model comprises the representation of the motion conditioning data as the spatio-temporal volume.

[0056] In some cases of the second aspect, the model input to the machine learning model further comprises the video.

[0057] In some cases of the second aspect, the video captures an environment in a vicinity of an agent at a time step: and the machine learning model comprises an action selection neural network that processes the model input to generate an action selection output; and the method further comprises: selecting an action to be performed by the agent at the time step using the action selection output.

[0058] In some cases of the second aspect, the action selection output comprises a score distribution over a set of actions; and selecting the action to be performed by the agent at the time step using the action selection output comprises: selecting an action having a highest score under the score distribution as the action to be performed by the agent at the time step.

[0059] In some cases of the second aspect, the environment is a real-world environment, the agent is a mechanical agent interacting with the environment, and the action is to be performed by the agent at the time step in an attempt to accomplish a specified task.

[0060] In some cases of the second aspect, the prediction output defines a classification of the video into a class from a set of one or more classes.

[0061] In some cases of the second aspect, the machine learning model performs an action classification task, and each of the one or more classes corresponds to a respective action.

[0062] In some cases of the second aspect, the machine learning model performs a scene classification task, and each of the one or more classes corresponds to a respective ty pe of scene.

[0063] In some cases of the second aspect, the machine learning model performs an object detection task, and each of the one or more classes corresponds to a respective type of object.

[0064] In some cases of the second aspect, the video captures a real -world environment and the prediction output characterizes one or more of an action performed in the environment, a scene of the environment, and an object in the environment.

[0065] According to a third aspect there is provided the methods of the first aspect or second aspect performed by a system comprising: one or more computers; and one or more storage devices communicatively coupled to the one or more computers, where the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the respective method. Atorney Docket No. 45288-0546WO1

[0066] According to a fourth aspect, there is provided the methods of the first aspect or second aspect performed by one or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations of the respective method.

[0067] Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0068] BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1A shows a video generation system.

[0069] FIG. IB shows example conditioning data and example motion prompts.

[0070] FIG. 1C shows an example motion prompt.

[0071] FIG. ID shows an example motion prompt.

[0072] FIG. IE shows an example motion prompt.

[0073] FIG. IF shows an example motion prompt.

[0074] FIG. 1 G shows an example motion prompt.

[0075] FIG. 2 is a flow diagram of an example process for generating and outputting a video. FIG. 3 is a flow diagram of an example process for generating motion conditioning data.

[0076] FIG. 4 is a flow diagram of an example process for generating a representation of motion conditioning data.

[0077] FIG. 5 is a flow diagram of an example process for training a generative model. FIG. 6 is an example of the performance of the described techniques.

[0078] FIG. 7 is an example of the performance of the described techniques.

[0079] FIG. 8 is an example of the performance of the described techniques.

[0080] Like reference numbers and designations in the various drawings indicate like elements.

[0081] DETAILED DESCRIPTION FIG. 1 A shows a video generation system 100. The system 100 is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

[0082] The system 100 can generate a video 112 using a generative model 110 that has characteristics that are consistent with conditioning data 106. Atorney Docket No. 45288-0546WO1

[0083] More specifically, in order to generate the video 112 using the generative model 110, the system 100 obtains conditioning data 106 specifying desired characteristics of the video 112 to be generated by the generative model 110.

[0084] The conditioning data 106 can include various types of data that specify the characteristics of the video 112 (e.g., the desired content, style, or motion of the video 112). For example, the conditioning data 106 can include a text prompt describing a scene or action, one or more images, or data defining specific spatial or temporal attributes for the video 112.

[0085] As shown in FIG. 1 A, the conditioning data 106 includes, for example, a conditioning video frame 102 and motion conditioning data 104, which are examples of an image and data defining specific spatial or temporal attributes for the video 112.

[0086] In particular, the conditioning video frame 102 can be for a time step in the video 112. That is, the conditioning video frame 102 can be one of the frames that will be part of the final video 112, for example, the first frame (e.g., at time step t=0).

[0087] Additionally, the motion conditioning data 104 includes one or more tracks, and each track is associated with a respective point in the conditioning video frame 102 and defines a spatio-temporal trajectory of the point through a plurality of time steps in the video 112. That is, each track identifies a specific spatial location (e.g., a pixel coordinate, e.g., (x,y) coordinate) on the conditioning video frame 102 and then describes the path that this point should follow over time in the generated video 112. The “spatio-temporal trajectory’7is this path, specifying the point’s coordinates at different time steps throughout at least a portion of the video's duration.

[0088] So, the motion conditioning data 104 can define the desired motion of specific points, and by extension, the objects or regions those points belong to.

[0089] The system 100 can obtain the conditioning data 106 in any of a variety of ways. In some implementations, the system 100 receives the conditioning data 106 from a user by w ay of a user interface.

[0090] For example, a user can provide the system 100 a conditioning video frame 102 (e.g., a “First Frame” for the video 112) through a graphical user interface, such as a file upload dialog or by selecting an image from a gallery.

[0091] As another example, a user can provide the system 100 with a motion prompt by way of the user interface and process the motion prompt obtained from the user to generate the motion conditioning data 104 for the generative model 110. A “motion prompt” is a user Atorney Docket No. 45288-0546WO1

[0092] input that defines desired motion in the video. The system 100 can process this prompt to generate the motion conditioning data 104, which includes one or more tracks.

[0093] For example, a motion prompt can define the desired movements of one or more objects or define a trajectory of camera poses.

[0094] For the video 112 to have characteristics consistent with the conditioning data 106 means that the video 112 adheres to the constraints specified by the conditioning data 106.

[0095] For example, if the conditioning data 106 includes the conditioning video frame 102 for a specific time step, the generated video 112 can include that frame at that time step.

[0096] As another example, if the conditioning data 106 includes the motion conditioning data 104 specifying motion depicted in the generated video 112, the generated video 112 can depict that motion.

[0097] As another example, if the conditioning data 106 includes a text prompt, the generated video 112 can also be semantically consistent with the text description.

[0098] FIG. IB shows example 150 conditioning data and example 154 motion prompts. In particular, example 150 conditioning data includes an example 152A conditioning video frame that depicts an elephant with its calf, an example 152B motion conditioning data 104 shown as a plurality of tracks, and an example 152C text prompt “Two elephants in the jungle.'’ FIG. IB shows the system 100 processes the example 150 conditioning data using a generative model that in this example is a video diffusion model.

[0099] Example 154 motion prompts include the examples 156A-G that each correspond to various obtained motion prompts and respective generated video. Each example visualizes the motion prompt tracks and corresponding frames from the generated videos underneath. Each example also represents different processes the system 100 can perform to obtain motion prompts.

[0100] In some implementations, to obtain and process a motion prompt, the system 100 obtains a motion prompt from a user by way of a user interface and processes the motion prompt obtained from the user to generate the motion conditioning data 104 for the generative model 110.

[0101] In some cases, to obtain the motion prompt from the user by way of the user interface, the system 100 displays, to the user and on the user interface, an interactive user interface element that displays the conditioning video frame 102 in the video 112. Then, the system receives a user input from the user, by way of user interaction with the interactive user interface element, that defines desired movements of one or more objects shown in the conditioning video frame 102 over a plurality of time steps in the video 112. Atorney Docket No. 45288-0546WO1

[0102] For example, a user interacting with a computer interface could use a mouse to click on a displayed conditioning video frame and perform a mouse drag. The mouse drag operation can be the user input that defines the desired movement.

[0103] In addition to the above description of how the system 100 can obtain a motion prompt, the system 100 can be configured to obtain a motion prompt following one or more of the following example techniques.

[0104] The system 100 can obtain amotion prompt that includes object control. Example 156A ("Object Control") illustrates a motion prompt that defines desired movements of one or more objects show n in the conditioning video frame, in this case, the head of an owl. the system 100 processes the motion prompt to generate a plurality of object control tracks associated with points on the owl's head, which are then included in the motion conditioning data 104.

[0105] That is, in some cases, to process the motion prompt obtained from the user to generate the motion conditioning data 104, the system 100 can process the user input defining the desired movements of the one or more objects shown in the conditioning video frame 102 over a plurality of time steps in the video 112 to generate a plurality of object control tracks associated with points in the conditioning video frame 102. In these cases, the plurality of object control tracks are included in the motion conditioning data 104. This general process is what is illustrated in Example 156A.

[0106] In some cases, the system 100 is configured to output only a final video frame in the video 112 generated using the generative model. That is, the system 100 outputs the video generated using the generative model 110 by outputting only a final video frame in the video 112 generated using the generative model 110 in order to perform drag-based image editing of an image represented by the conditioning video frame 102.

[0107] FIG. 1C shows an example 160 motion prompt that includes object control and dragbased image editing of conditioning video frames. The conditioning video frames are shown in the top row and the respective resulting drag-based edits in the bottom row; The user's drag, which constitutes the motion prompt, can be visualized in both rows.

[0108] The system 100 can obtain a motion prompt that includes object control with geometric primitives. Example 156B ("Object Control with Geometric Primitives") illustrates a specific implementation of object control. The motion prompt obtained from the user is provided as a geometric primitive (e.g., a grid of points) that defines the desired movement. The system 100 processes this input to generate a plurality of object control tracks corresponding to the points in the geometric primitive. This allows a user to define the Atorney Docket No. 45288-0546WO1

[0109] motion of a larger region or object (e.g., the bear's head) without specifying individual tracks for every point within the region.

[0110] That is, in some cases, the user input defines a manipulation of a geometric primitive associated with an object in the conditioning video frame. Additionally processing the user input includes translating the manipulation of the geometric primitive into the plurality of object control tracks.

[0111] FIG. ID shows an example 162 motion prompt that includes object control with geometric primitives. FIG. ID displays how the system 100 can define geometric primitives, for example a sphere, which is manipulated by a user with a mouse to define a user input. By processing these manipulations through the user input, the system 100 can obtain tracks that exert more fine-grain control over objects, such as enabling rotations. In particular, example 162 shows the geometric primitive of a sphere over the head of a cat and the eye of a frog that the user rotates to define precise rotations of these objects as a user input.

[0112] The system 100 can obtain a motion prompt that includes camera control with depth Example 156C ("Camera Control with Depth") illustrates that the motion prompt obtained from the user defines a trajectory of camera poses. The system 100 processes this trajectory to generate motion conditioning data 104 that causes the generated video 112 to be captured from a perspective of a camera that follows the trajectory7of camera poses. The system 100 can process the conditioning video frame 102 using a depth estimation model to generate a point cloud representing the scene's geometry. The system 100 can then project the point cloud onto each camera pose in the trajectory to generate a plurality7of camera control tracks, which are included in the motion conditioning data 104.

[0113] That is, in some implementations, to process the motion prompt obtained from the user to generate the motion conditioning data for the generative model, the system 100 processes a trajectory of camera poses (e.g., received as user input from a user) to generate motion conditioning data that causes the video generated by the generative model 110 when conditioned on the motion conditioning data 104 to be captured from a perspective of a camera that follows the trajectory of camera poses.

[0114] Additionally, in some cases, the system 100 processes the conditioning video frame 102 using a depth estimation model to generate a point cloud representing a geometry of a scene shown in the conditioning video frame 102. The system 100 then projects the point cloud onto each camera pose in the trajectory of camera poses to generate a plurality of camera control tracks associated with points in the conditioning video frame 102, where the plurality of camera control tracks are included in the motion conditioning data 104. Atorney Docket No. 45288-0546WO1

[0115] FIG. IE shows an example 164 of a motion prompt that includes camera control with depth. FIG. IE shows how the system 100 processes a conditioning video frame to generate a point cloud, and then, projects the point clouds onto each camera pose included in the user input. The right side of the figure, labeled (a), shows the resulting point trajectories (top row) and the corresponding frames from the generated video (bottom row).

[0116] The system 100 can obtain amotion prompt that includes object control and camera control. Example 156D ("Object + Camera Control") illustrates that the system 100 processes a motion prompt to generate both (i) a plurality of object control tracks (e.g., for the dog's head) and (ii) a plurality of camera control tracks (e.g., for a camera pan). The system 100 therefore generates a plurality of combined tracks for inclusion in the motion conditioning data by combining the object control tracks and the camera control tracks.

[0117] That is, in some cases, to process the motion prompt obtained from the user to generate the motion conditioning data for the generative model, the system 100 processes the motion prompt obtained from the user to generate: (i) a plurality of object control tracks for controlling movement of objects in the video, and (ii) a plurality of camera control tracks for controlling a camera perspective from which the video is captured. The system 100 then generates a plurality of combined tracks for inclusion in the motion conditioning data 104 by combining the object control tracks and the camera control tracks.

[0118] FIG. IF shows an example 166 for a motion prompt that includes composition of motion prompts (e.g., object control and camera control). FIG. IF shows how the system 100 can compose multiple user inputs together, thereby attaining simultaneous object and camera control. As shown in FIG. IF, the system 100 can receive user inputs that move a dog's or horse's head while also receiving user inputs that orbits the camera from left to right. This can be done by the system 100 adding together the tracks for object control and camera control from a user input.

[0119] The system 100 can obtain a motion prompt that includes motion transfer. Example 156E ("Motion Transfer") illustrates that the system obtains the motion conditioning data 104 by processing a source video (e.g.. a video of a human). The system 100 then provides the motion conditioning data 104 (e.g., tracks defining the human's facial motion) to the generative model 110 along with a conditioning video frame 102 from a different source (e.g., an image of a monkey). Then the system 100 generates an output video 112 that transfers the motion defined by the tracks to the subject of the conditioning video frame 102.

[0120] That is. in some cases, to obtain the conditioning data, the system 100 processes a source video to generate the one or more tracks of the motion conditioning data 104. Atorney Docket No. 45288-0546WO1

[0121] Additionally, the conditioning video frame 102 is one obtained from a source that is distinct from the source video, and the generated video 112 includes transferred motion from the source video to the conditioning video frame 102.

[0122] FIG. G shows an example 168 motion prompt that includes motion transfer. FIG. G shows how the system 100 performs motion transfer by extracting motion tracks from a source video and applying them to an condition video frame, which can be useful for motions that are difficult to design. In particular FIG. G shows three sets of motion transfer examples. The first, top example shows motion tracks extracted from a "Source" video of a human face being “Transferred” to a conditioning video frame of a monkey. The second, middle example shows motion from a “Source” video of a rotating globe being “Transferred” to a panda's face. The third, bottom example shows a dense set of motion tracks from a “Source” video being “Transferred” to an image of a bird’s eye view of trees.

[0123] The system 100 can obtain a motion prompt that includes model probing. As illustrated in examples 156F ("Model Probing") and 156G ("More Model Probing"), the motion prompt is sometimes the result of the system 100 displaying, to the user and on the user interface, an interactive user interface element that displays the conditioning video frame. The system 100 then receives a user input from the user, by way of user interaction with the interactive user interface element, which is visualized by the mouse cursor (hand or black cursor). This user interaction defines desired movements of one or more objects shown in the conditioning video frame. As shown in the examples, this interaction can generate a motion prompt for just a few pixels (e.g., a single point track, as in 156G) or for patches (e.g., a grid of tracks, as show n when dragging in 156F).

[0124] Examples 156F ("Model Probing") and 156G ("More Model Probing") illustrate how the system can generate videos depicting an "interaction" with the content of the conditioning video frame 102. In these examples, the system 100 obtains a motion prompt from a user by way of a user interface through simple mouse motions (although other input devices can of course be used). The system 100 processes the motion prompt (the user's mouse motion) and expands it to generate the motion conditioning data. For instance, as shown in example 156F, a mouse drag can be processed to generate a plurality of object control tracks, e.g., a grid of tracks centered on the cursor, which are then included in the motion conditioning data. The generative model 110 then processes this conditioning data to generate the output video 112, shown in the bottom row of the examples, which depicts the interaction corresponding to the user's input (e.g.. interacting with the sand or the woman's hair). Atorney Docket No. 45288-0546WO1

[0125] Returning to the description of FIG. 1A, in some implementations, the system 100 can obtain the conditioning data 106 by loading the conditioning data 106 from system 100 maintained memory.

[0126] In some implementations, the system 100 can obtain the conditioning data 106 from another system. For example, a video processing system could apply a point tracking algorithm to a pre-existing video to extract the spatio-temporal trajectories of all pixels (or tracks), and then transmit this data (as the motion conditioning data 104) along with the first frame (as the conditioning video frame 102) to the system 100.

[0127] In some cases, when the system 100 obtains the conditioning data 106 from another system, the system 100 can use the conditioning data 106 to train the generative model 110. For example, to generate a training example that includes (i) conditioning data that includes a conditioning video frame and motion conditioning data, and (ii) a target video that is consistent with the conditioning data, the system 100 can apply a point tracking algorithm (as described above) to a target video to generate the conditioning data for the training example.

[0128] After the system 100 obtains the conditioning data 106, the system 100 processes a representation 108 of the conditioning data 106 using the generative model 110 to generate the video 112.

[0129] In some cases, the system 100 processes the conditioning data 106 to generate the representation 108 of the conditioning data 104 that the generative model 110 processes. For example, the system 100 can generate a representation 108 of the motion conditioning data 104 as a spatio-temporal volume.

[0130] In some cases, the spatio-temporal volume includes a temporal dimension indexing time steps of the video 112 and a plurality of spatial dimensions corresponding to spatial dimensions of the video. For example, if the video 112 to be generated has 16 time steps, each video frame has a height of 64 pixels, and a width of 64 pixels, the spatio-temporal volume can have dimensions corresponding to time (16), height (64), and width (64).

[0131] In some implementations, the spatio-temporal volume can also include a channel dimension. So, as an example, the above described spatio-temporal volume with time (16). height (64), and width (64) dimensions can also include a dimension C corresponding to the number of channels. For example, each location in the spatio-temporal volume can be associated with a corresponding vector that has a length C corresponding to the number of channels.

[0132] The channel dimension of the spatio-temporal volume can include channels corresponding to respective components of pixel data. For example, where there are three Atorney Docket No. 45288-0546WO1

[0133] channels, each channel can correspond to a respective color component of the pixel data (e.g., red, green, blue for color video, or hue, saturation, and value, and so on).

[0134] In some implementations, to generate the representation 108 of the motion conditioning data 104 as spatio-temporal volume, the system 100 first initializes the spatiotemporal volume with default values (e.g., by filling all entries with zero). After initialization, the system 100 updates the spatio-temporal volume to encode each track from the one or more tracks. The system 100 can perform the update on a per-track basis by determining a respective embedding for the track (e g., a unique, non-zero vector), and then updating each location in the spatio-temporal volume that corresponds to a spatial location (e.g., x, y coordinates) and time step (t) visited by that specific track to include the embedding of the track.

[0135] In some implementations, the system 100 also accounts for track visibility. The motion conditioning data 104 can define, for each time step in each track, whether the track is off-screen (e.g., when the spatial location of track at the time step is outside the spatial locations spanned by the spatial dimensions of the video) or occluded at that time step. In such cases, when the system 100 updates the spatio-temporal volume, the process of the system 100 including the embedding is modified such that for any time step where the track is designated as off-screen or occluded, the system 100 modifies the embedding included in the spatio-temporal volume at the corresponding spatial location and time step to reflect that the track is off-screen or occluded. For example, the system’s 100 modification can involve modifying the embedding to be a zero-embedding with a zero in each entry (or another value for the entry that indicates the track being off-screen or occluded).

[0136] For example, generating a representation of the motion conditioning data as a spatiotemporal volume can comprise, for each track in the motion conditioning data, assigning a respective embedding for the track to each location in the spatio-temporal volume that corresponds to a spatial location and time step visited by the track and at which the track remains visible,

[0137] In some implementations, the system 100 can also control how strictly the generated video 112 must follow a given track at any specific moment. That is, the motion conditioning data 104 can define, for each time step in each track, an adherence value that defines a degree to which the video should adhere to track at the time step (e.g., an extent to which the generated video is influenced by the track at the time step). Then, when the system 100 updates each location in the spatio-temporal volume that corresponds to a spatial location and time step visited by the track to include the embedding of the track for each track, the system Atorney Docket No. 45288-0546WO1

[0138] 100 modifies the embedding included in the spatio-temporal volume at the spatial location and time step based on a corresponding adherence value for the track at the time step.

[0139] The system 100 can set the adherence values in any appropriate way, e.g., by a user via a user interface or by the system 100. In some implementations, the system 100 can be configured with a predefined set of adherence values.

[0140] As an example, the system 100 can modify the embedding included in the spatiotemporal volume at the spatial location and time step based on the corresponding adherence value for the track at the time step by modifying the embedding included in the spatiotemporal volume at the spatial location and time step to be a combination of: (i) the embedding for the track, and (ii) an embedding representing the adherence value for the track at the time step.

[0141] The system 100 can generate the embedding representing the adherence value for the track at the time step in any appropriate way, e.g., by mapping each adherence value in a predefined set of adherence values to a predefined vector embedding.

[0142] The system can combine the: (i) the embedding for the track, and (ii) an embedding representing the adherence value for the track at the time step in any appropriate way. For example, the system 100 can perform an element- wise summation.

[0143] The generative model 110 can have any appropriate architecture in any appropriate configuration that can process a representation 108 of the condition data 104 and generate a video 112, including fully connected layers, convolutional layers, recurrent layers, attentionbased layers, and so on, as is appropriate.

[0144] In some implementations, the generative model 110 is a generative diffusion model. For example, FIG. IB illustrates the use of a “Video Diffusion Model”. Such models can be, for example, built using a U-Net architecture, which can be adapted for video generation. For instance, a “Space-Time U-Net” (STUNet) architecture which generates the entire temporal duration of the video at once. These architectures can employ convolutional layers (e.g., factorized space-time convolutions) and attention-based layers (e.g., temporal attention or cross-attention mechanisms) to process the spatial and temporal information contained in the representation 108 and synthesize a coherent video 112. For example, the generative model 110 can be the Lumiere space-time diffusion model for video generaton (as described in arXiv:2401.12945).

[0145] Generally, prior to the system 100 using the generative model 110, the system 100 or another system trains the generative model 110. That is, the system 100 trains the generative Atorney Docket No. 45288-0546WO1

[0146] model 110 to learn how to generate videos that have characteristics consistent with respective conditioning data.

[0147] For example, the system 100 can train the generative model 110 using a set of training examples, where each training example can include (i) conditioning data that includes a conditioning video frame and motion conditioning data, and (ii) a target video that is consistent with the conditioning data.

[0148] During training, the system 100 can use the generative model 110 to process the conditioning data from an example to generate an output video, evaluate an objective (e.g., a loss function) by comparing the output video to the target video, and update the generative model’s trainable parameters to optimize this objective.

[0149] In some cases, the system 100 can generate this training data by processing the target video of the training example using a point tracking algorithm (e.g., the BootsTAP point tracking method, as described in arXiv:2402.00847) to generate the one or more tracks for the motion conditioning data. For example, given a target video from a training set (e.g., a video of a car driving), the system 100 can apply a point tracking algorithm to that video to generate one or more tracks that define the car's movement. This set of tracks (as the motion conditioning data 104) and the first frame of the video (as the conditioning video frame 102) are then used by the system 100 as the conditioning data for that training example.

[0150] Further details of training the generative model 110 are described below.

[0151] After the system 100 generates the video 112, The system 100 outputs the video 112 generated using the generative model 110.

[0152] In some implementations, the system 100 outputs the complete video 112, which includes the full plurality of time steps of the video 112. This output can be performed in several ways. For example, the system 100 can output the video to store the video in memory (e.g., system 100 memory or a storage device). As another example, the system 100 can output the video 112 to present the video 112 on a display of a user device (e.g., present the video on a laptop computer or a smartphone device). As another example, the system 100 can output the video to transmit the video 112 over a data communication network (e.g., the Internet to a client device).

[0153] In some cases, the system 100 is configured to output only a final video frame in the video 112 generated using the generative model. For example, the system 100 can be configured to perform a drag-based image editing of a conditioning video frame 102 where the motion conditioning data 104 includes a motion prompt that defines the drag-based image editing. In this case, the full video 112 generated by the generative model 110 serves as an Atorney Docket No. 45288-0546WO1

[0154] intermediate computation to determine the final position and appearance of the edited object, but the system 100 outputs only the resulting still image (the final frame) as the video 112.

[0155] While FIG. 1 A illustrates a video generation system 100 that uses a generative model 110 to generate a video 112, the ut i 1 i ty of the motion conditioning data 104 and its representation 108 (e.g., as a spatio-temporal volume) is not limited to video generation.

[0156] This specification also describes systems and methods where this track-based motion representation is used as a model input to other types of machine learning models to generate various prediction outputs. For example, the representation 108 of the motion tracks can be processed by a machine learning model, optionally along with the video itself, to perform tasks such as video classification (e.g., action, scene, or object classification) or to generate an action selection output for controlling an agent in an environment. These further embodiments are described in greater detail below.

[0157] FIG. 2 is a flow diagram of an example process 200 for generating and outputting a video. For convenience, the process 200 will be described as being performed by a system of one or more computers located in one or more locations. For example, a video generation system, e.g., the video generation system 100 of FIG. 1A, appropriately programmed in accordance with this specification, can perform the process 200.

[0158] The system generates a video using a generative model (step 202).

[0159] As described above the generative model (e.g., neural network) can have any of a variety of architectures and configurations, e.g.. an architecture that includes fully connected layers, convolutional layers, recurrent layers, attention-based layers, and so on, as is appropriate such that the system can use it process a representation of the conditioning data to generate the video.

[0160] In some implementations, the generative model is a generative diffusion model. For example, the generative model can have a Space-Time U-Net (STUNet) architecture, e.g., as used in the Lumiere model (as described in arXiv:2401.12945). As another example, the generative model can be a large pre-trained text-to-video diffusion model, that is adapted with an architecture, e.g., ControlNet (as described in arXiv:2302.05543), to add conditional controls. These diffusion models can generate a video by starting from a noise sample and gradually denoising it over a series of steps to reach a video. A model like the STUNet architecture, for instance, can be configured to generate the entire temporal duration of the video at once through a single pass by processing it in multiple space-time scales. Atorney Docket No. 45288-0546WO1

[0161] Generally, prior to the system using the generative model to generate a video, the system or another system trains the generative model.

[0162] In some implementations, the system configures the generative model through training to cause the video to have characteristics that are consistent with the conditioning data. That is, the system trains the generative model such that when the generative model is given new condition data (condition data not used during the training of the generative model) the generative model will generate a video that adheres to the constraints defined by the conditioning data (e g., starting the video at a particular video frame, constraining the motion present in the video, the video being semantically consistent with a text prompt, and so on).

[0163] Further details of training the generative model are described below with reference to FIG. 5.

[0164] In order for the system to generate a video using a generative model, the system performs steps 204A and 204B.

[0165] The system obtains conditioning data specifying desired characteristics of the video to be generated by the generative model (step 204A).

[0166] As described above, the conditioning data can include various types of data specifying the desired content, sty le, or motion of the video. For example, the conditioning data can include a text prompt describing a scene, a conditioning video frame for a time step in the video, and motion conditioning data that includes one or more tracks defining spatiotemporal trajectories for points in the conditioning video frame.

[0167] In particular, the conditioning data of step 204A includes a conditioning video frame for a time step in the video. The conditioning data also includes motion conditioning data that includes one or more tracks, where each track is associated with a respective point in the conditioning video frame and defines a spatio-temporal trajectory of the point through a plurality of time steps in the video.

[0168] In some cases, the motion conditioning data includes a respective track for at least 10%, or at least 20%, or at least 30%, or at least 40%, or at least 50%, or at least 60%, or at least 70%, or at least 80%, or at least 90% of points in the conditioning video frame.

[0169] In some other cases, the motion conditioning data comprises a respective track for every' point in the conditioning video frame.

[0170] As described above, the system can obtain the conditioning data in any of a variety of wavs. Atorney Docket No. 45288-0546WO1

[0171] For example, the system can receive the conditioning data from a user by way of a user interface. This can include receiving a user input that includes a text prompt (e.g.. via a text entry box), or receiving a user selection of a conditioning video frame (e.g., by way of a file upload dialog or selection from a gallery).

[0172] As another example, the system can obtain motion conditioning data by loading the data from a storage device (e.g., system 100 maintained memory) or by receiving it from another system, such as a video processing system that applies a point tracking algorithm to a video to extract the tracks.

[0173] In some cases, as described above, the system generates the motion conditioning data. For example, the system can obtain a motion prompt from a user (e.g., a user input that defines desired motion in the video, such as the desired movements of one or more objects or a desired trajectory' of camera poses) and generate the motion conditioning data included in the conditioning data.

[0174] Further details of generating motion conditioning data are described below with reference to FIG. 3.

[0175] The system processes a representation of the conditioning data using the generative model to generate the video (step 204B).

[0176] In some implementations, the system processes the conditioning data, which includes the system generating a representation of the motion conditioning data as a spatio-temporal volume, to generate the representation of the conditioning data that is processed using the generative model.

[0177] In some cases, the spatio-temporal volume includes a temporal dimension indexing time steps of the video and a plurality of spatial dimensions corresponding to sparial dimensions of the video.

[0178] In some cases, the spatio-temporal volume includes a channel dimension.

[0179] For example, the spatio-temporal volume can include a TxHxWxC dimensional space-time volume, where T represents the temporal dimension, H represents the height, W represents the width, and C represents the channel dimension. In this notation, T corresponds to the temporal dimension indexing the time steps of the video, while H and W correspond to the plurality of spatial dimensions of the video.

[0180] In some cases, the system updates the spatio-temporal volume to encode each track in the one or more tracks of the motion conditioning data by, e.g., determining a respective embedding for each track and updating the spatio-temporal volume using the respective embedding. Atorney Docket No. 45288-0546WO1

[0181] Further details of generating a representation of motion conditioning data are described below with reference to FIG. 4.

[0182] As an example of the system processing the a representation of the conditioning data using the generative model to generate the video, where the generative model is a generative diffusion model (e.g., a STUNet or an adapted model, e.g., an adapted pre-trained text-to-image diffusion model), the system processes the representation of the conditioning data to guide the video generation. In a STUNet architecture, the representation (including the spatio-temporal volume) can be an input that the model processes through its space-time U-Net to generate the full video at once. In an adapted model architecture (e.g. ControlNet, as described in arXiv:2302.05543), the representation can be processed by additional, trainable layers, and the outputs of these layers are used to inject the motion conditioning into a pretrained model's generative process, e.g., by influencing its attention or convolutional blocks. In either case, the generative model processes the representation (which in this case encodes motion tracks) to generate a video that is consistent with the specified spatio-temporal trajectories.

[0183] The system outputs the video generated using the generative model (step 206).

[0184] As described above, in some cases, when the system outputs the video, the system stores the video in a memory’, or presents the video on a display of a user device, or transmits the video over a data communication network.

[0185] For example, the system can store the video as a file (e.g.. an .mp4 file) on a local hard drive or a cloud-based storage service. As another example, the system 100 can present the video by rendering it for playback within the user interface of an application on a user device. As another example, the system 100 can transmit the video over a data communication network by, e.g., sending it to a client device for display or uploading it to a video-sharing platform.

[0186] Also, as described above, in some cases, the system outputs only a final video frame in the video generated using the generative model in order to perform drag-based image editing of an image represented by the conditioning video frame.

[0187] For example, in a drag-based image editing application, a user provides amotion prompt by "dragging" a point on the conditioning video frame to a new’ location. The system processes this input as a track and generates the full video, which depicts the object moving along this track. The system then outputs only the final frame of this generated video as the resulting edited image. The system can then present the output to the user in the user interface Atorney Docket No. 45288-0546WO1

[0188] as the static result of their edit, or it can be saved as a new still image file (e.g., a JPEG or PNG) in memory.

[0189] The example 200 process is an example of a more general method of the system, where the system uses motion conditioning data as a part of a machine learning model input and processes this input using the machine learning model to generate a prediction output. The prediction output can be, for example, a classification of the video into one or more classes, such as for an action classification task, a scene classification task, or an object detection task.

[0190] The machine learning model can be trained to perform a prediction task (i.e., process a machine learning model input to generate a corresponding prediction output) using a set of training examples, where each training example can include (i) conditioning data that includes respective motion conditioning data, and (ii) a respective target prediction output. During training, the system can process the conditioning data from each example to generate a corresponding prediction output, evaluate an objective (e.g., a loss function, e.g., a mean squared error) by comparing the prediction output generated by the machine learning model with the target prediction output, and update the machine learning model’s trainable parameters to optimize this objective. Training examples can be obtained for a particular task, such as video classification, action classification, scene classification, or object detection, from existing datasets, such as the UCF101, Kinetics (see e g., arXiv:705.06950), and YouTube-8M (see https: / / research.google.com / youtube8m / ) datasets, or otherwise generated by, for example, human annotators.

[0191] In some cases, the machine learning model can be an action selection neural network for controlling an agent. The system can then use the model to process motion conditioning data (e.g., derived from a video of the agent's environment) to generate an action selection output, which the system then uses to select an action for the agent to perform.

[0192] Further details on these systems and methods follow'.

[0193] In some implementations, the system generates and provides a prediction output. More specifically, the system obtains motion conditioning data (e.g., as described above) that includes one or more tracks, where each track is associated with a point in a video frame of a video and defines a spatio-temporal trajectory of the point through a plurality of time steps in the video. The system then processes a model input that includes the motion conditioning data using a machine learning model to generate a prediction output. Then, the system provides the prediction output. Atorney Docket No. 45288-0546WO1

[0194] In some cases, the prediction output defines a classification of the video into a class from a set of one or more classes.

[0195] For example, the machine learning model can perform an action classification task, and each of the one or more classes corresponds to a respective action. As a particular example, based on motion conditioning data defining the looping trajectories of a person's arms and legs, the model classifies the video with the class 'swimming'.

[0196] As another example, the machine learning model can perform a scene classification task, and each of the one or more classes corresponds to a respective type of scene. As a particular example, based on motion conditioning data defining many parallel, high-speed trajectories, the model classifies the video with the class 'highway traffic', distinguishing it from a 'city street' class which would have more erratic, slower trajectories.

[0197] As another example, the machine learning model performs an object detection task, and each of the one or more classes corresponds to a respective type of object. As a particular example, the model processes motion conditioning data defining a set of co-moving trajectories (i.e., points moving together) to detect and classify a 'pedestrian' as distinct from the static 'background'.

[0198] The machine learning model can have any of a variety of architectures configured to process a model input to generate a prediction output. For example, the model can be a 3D Convolutional Neural Network (3D CNN) that applies convolutional filters across both spatial dimensions (e.g., height, width) and the temporal dimension. As another example, the model can be a Video Transformer architecture that uses self-attention mechanisms to model relationships between different points in space and time as represented in the model input. In other cases, the model may be a combination of a spatial feature extractor (e.g., a 2D CNN) and a temporal model (e.g., a Recurrent Neural Network).

[0199] When the system provides the prediction output, the system can perform various operations, depending on the system's task. For example, in an action classification task, providing the output can include storing the predicted class label (e g., "running," "jumping") in a memory or database as metadata associated with the video. In an object detection task, providing the output can include rendering the prediction on a display, such as by drawing bounding boxes and class labels (e.g., "car," "pedestrian") over the corresponding video frames. For an agent control task, as described below, providing the prediction output can include transmitting the output as a control signal to another component, e.g., to the agent's actuators. Atorney Docket No. 45288-0546WO1

[0200] In some implementations, prior to the system processing the model input, the system generates a representation of the motion conditioning data as a spatio-temporal volume (e.g.. as described above), and the model input to the machine learning model includes the representation of the motion conditioning data as the spatio-temporal volume.

[0201] In some cases, the model input to the machine learning model also includes the video. That is. the machine learning model is configured to process multiple, distinct inputs simultaneously. The model can use the video (e.g., the raw pixel data) to understand the visual appearance, texture, and semantic context of the scene, while using the motion conditioning data (e.g., the spatio-temporal volume) as an explicit and direct representation of how points and objects are moving over time. For example, in an action classification task for "playing soccer," the model could process the video to identify the players, the ball, and the field, while simultaneously processing the spatio-temporal volume to understand the specific, long-range trajectories of the players running and the ball being kicked.

[0202] In some implementations, the video captures an environment in a vicinity of an agent at a time step. For these implementations the machine learning model can include an action selection neural network that processes the model input to generate an action selection output (as the prediction output). Additionally, after the system generates the action selection output, the system selects an action to be performed by the agent at the time step using the action selection output.

[0203] In some cases, the action selection output includes a score distribution over a set of actions. So, when the system selects the action to be performed by the agent at the time step using the action selection output, the system selects an action having a highest score under the score distribution as the action to be performed by the agent at the time step.

[0204] For example, an autonomous vehicle is the agent, and its machine learning model processes a video of the environment. The model input includes motion conditioning data showing the spatio-temporal trajectories of a nearby cyclist. Because the cyclist's trajectory is predicted to intersect the vehicle's path, the action selection output generates the highest score for the 'brake' action, and the system selects 'brake' over other available actions like 'accelerate' or 'maintain speed'.

[0205] More specifically, this specification describes methods for controlling an agent interacting with an environment over a sequence of time steps using an action selection neural network. At each of one or more time steps, the network can process a network input that includes a current observation of the environment at the time step as well as motion conditioning data as described herein. The network processes this input to generate an action Atorney Docket No. 45288-0546WO1

[0206] selection output that is used to select an action to be performed by the agent at the time step. The network input to the action selection neural network thus includes motion conditioning data that comprises one or more tracks (and / or an encoding of the one or more tracks in a spatio-temporal volume) defining spatio-temporal trajectories of points in a video frame of a video, e.g., that captures the environment over a duration of time preceding the current time step.

[0207] In some implementations, the environment is a real-world environment, the agent is a mechanical agent interacting with the real-world environment, e g., a robot or an autonomous or semi-autonomous land, air, or sea vehicle operating in or navigating through the environment, and the actions are actions taken by the mechanical agent in the real-world environment to perform the task. For example, the mechanical agent, e.g. robot, may be interacting with the environment to accomplish a specific task, e.g., to locate or manipulate an object of interest in the environment or to move an object of interest to a specified location in the environment or to navigate to a specified destination in the environment.

[0208] In these implementations, the observations may include, e.g., one or more of: images, video, object position data, and sensor data to capture observations as the agent interacts with the environment, for example sensor data from an image, distance, or position sensor or from an actuator. For example in the case of a robot, the observations may include data characterizing the current state of the robot, e.g., one or more of: joint position, joint velocity, joint force, torque or acceleration, e.g., gravity-compensated torque feedback, and global or relative pose of an item held by the robot. In the case of a robot or other mechanical agent or vehicle the observations may similarly include one or more of the position, linear or angular velocity', force, torque or acceleration, and global or relative pose of one or more parts of the agent. The observations may be defined in 1, 2 or 3 dimensions, and may be absolute and / or relative observations. The observations may also include, for example, sensed electronic signals such as motor current or a temperature signal; and / or image or video data for example from a camera or a LIDAR sensor, e.g., data from sensors of the agent or data from sensors that are located separately from the agent in the environment.

[0209] In these implementations, the actions may be control signals to control the robot or other mechanical agent, e g., torques for the joints of the robot or higher-level control commands, or the autonomous or semi-autonomous land, air, sea vehicle, e.g., torques to the control surface or other control elements e.g. steering control elements of the vehicle, or higher-level control commands. The control signals can include, for example, position, velocity, or force / torque / accel eration data for one or more joints of a robot or parts of another Atorney Docket No. 45288-0546WO1

[0210] mechanical agent. The control signals may also or instead include electronic control data such as motor control data, or more generally data for controlling one or more electronic devices within the environment the control of which has an effect on the observed state of the environment. For example in the case of an autonomous or semi-autonomous land or air or sea vehicle the control signals may define actions to control navigation e.g. steering, and movement e.g., braking and / or acceleration of the vehicle.

[0211] In some implementations the environment is a simulation of the above-described real-world environment, and the agent is implemented as one or more computers interacting with the simulated environment. For example the simulated environment may be a simulation of a robot or vehicle (or other mechanical agent) and the reinforcement learning system may be trained on the simulation and then, once trained, used in the real-world.

[0212] In some implementations the environment is a real-world manufacturing environment for manufacturing a product, such as a chemical, biological, or mechanical product, or a food product. As used herein a “manufacturing” a product also includes refining a starting material to create a product, or treating a starting material e g. to remove pollutants, to generate a cleaned or recycled product. The manufacturing plant may comprise a plurality of manufacturing units such as vessels for chemical or biological substances, or machines, e g. robots, for processing solid or other materials. The manufacturing units are configured such that an intermediate version or component of the product is moveable between the manufacturing units during manufacture of the product, e.g. via pipes or mechanical conveyance. As used herein manufacture of a product also includes manufacture of a food product by a kitchen robot.

[0213] The agent may comprise an electronic agent configured to control a manufacturing unit, or a machine such as a robot, that operates to manufacture the product. That is, the agent may comprise a control system configured to control the manufacture of the chemical, biological, or mechanical product. For example the control system may be configured to control one or more of the manufacturing units or machines or to control movement of an intermediate version or component of the product between the manufacturing units or machines.

[0214] As one example, a task performed by the agent may comprise a task to manufacture the product or an intermediate version or component thereof. As another example, a task performed by the agent may comprise a task to control, e.g. minimize, use of a resource such as a task to control electrical power consumption, or water consumption, or the consumption of any material or consumable used in the manufacturing process. Atorney Docket No. 45288-0546WO1

[0215] The actions may comprise control actions to control the use of a machine or a manufacturing unit for processing a solid or liquid material to manufacture the product, or an intermediate or component thereof, or to control movement of an intermediate version or component of the product within the manufacturing environment e.g. between the manufacturing units or machines. In general the actions may be any actions that have an effect on the observed state of the environment, e.g. actions configured to adjust any of the sensed parameters described below. These may include actions to adjust the physical or chemical conditions of a manufacturing unit, or actions to control the movement of mechanical parts of a machine or joints of a robot. The actions may include actions imposing operating conditions on a manufacturing unit or machine, or actions that result in changes to settings to adjust, control, or switch on or off the operation of a manufacturing unit or machine.

[0216] The rewards or return may relate to a metric of performance of the task. For example in the case of a task that is to manufacture a product the metric may comprise a metric of a quantity of the product that is manufactured, a quality of the product, a speed of production of the product, or to a physical cost of performing the manufacturing task, e.g. a metric of a quantity of energy, materials, or other resources, used to perform the task. In the case of a task that is to control use of a resource, the metric may comprise any metric of usage of the resource.

[0217] In general observations of a state of the environment may comprise any electronic signals representing the functioning of electronic and / or mechanical items of equipment. For example a representation of the state of the environment may be derived from observations made by sensors sensing a state of the manufacturing environment, e.g. sensors sensing a state or configuration of the manufacturing units or machines, or sensors sensing movement of material between the manufacturing units or machines. As some examples, such sensors may be configured to sense mechanical movement or force, pressure, temperature; electrical conditions such as current, voltage, frequency, impedance; quantity, level, flow / movement rate or flow / movement path of one or more materials; physical or chemical conditions e.g. a physical state, shape or configuration or a chemical state such as pH; configurations of the units or machines such as the mechanical configuration of a unit or machine, or valve configurations; image or video sensors to capture image or video observations of the manufacturing units or of the machines or movement; or any other appropriate type of sensor. In the case of a machine such as a robot the observations from the sensors may include observations of position, linear or angular velocity, force, torque or acceleration, or pose of Atorney Docket No. 45288-0546WO1

[0218] one or more parts of the machine, e.g. data characterizing the current state of the machine or robot or of an item held or processed by the machine or robot. The observations may also include, for example, sensed electronic signals such as motor cunent or a temperature signal, or image or video data for example from a camera or a LIDAR sensor. Sensors such as these may be part of or located separately from the agent in the environment.

[0219] As previously described the environment may be a simulated environment. Generally in the case of a simulated environment the observations may include simulated versions of one or more of the previously described observations or types of observations and the actions may include simulated versions of one or more of the previously described actions or types of actions. For example the simulated environment may be a motion simulation environment, e.g., a driving simulation or a flight simulation, and the agent may be a simulated vehicle navigating through the motion simulation. In these implementations, the actions may be control inputs to control the simulated user or simulated vehicle. Generally the agent may be implemented as one or more computers interacting with the simulated environment.

[0220] The simulated environment may be a simulation of a particular real-world environment and agent. For example, the system may be used to select actions in the simulated environment during training or evaluation of the system and, after training, or evaluation, or both, are complete, may be deployed for controlling a real-world agent in the particular real-world environment that was the subject of the simulation. This can avoid unnecessary wear and tear on and damage to the real-world environment or real-world agent and can allow the control neural network to be trained and evaluated on situations that occur rarely or are difficult or unsafe to re-create in the real-world environment. For example the system may be partly trained using a simulation of a mechanical agent in a simulation of a particular real-world environment, and afterwards deployed to control the real mechanical agent in the particular real-w orld environment. Thus in such cases the observations of the simulated environment relate to the real-world environment, and the selected actions in the simulated environment relate to actions to be performed by the mechanical agent in the real-world environment.

[0221] In some implementations the observations are observations of a real-world environment in which a human is performing a task, e.g. an image observation from an image sensor and / or a language observation from a speech recognition system; and the actions are language actions that control (instruct) the human, e.g. using natural language or images, to perform actions in the real-world environment to perform the task. A language action may be Atorney Docket No. 45288-0546WO1

[0222] an action that outputs a natural language sentence, e.g. by defining a sequence of language tokens, e g. words or wordpieces, to be emitted at sequential time steps.

[0223] Thus the agent may comprise a user interface device such as a digital device (a “digital assistant”), e.g. a smart speaker or smart display or other device, e.g. with a natural language input and / or output, that controls (instructs) a human user to perform a task. In general such a digital device can be a mobile device with a natural language interface to receive natural language requests from a human user and to provide natural language responses. It may also include a vision based input e.g. a camera and / or display screen. The digital device may include a language model or language generation neural network system either stored locally, or accessed remotely, or both. The user interface device may comprise, e.g., a mobile device, a keyboard (and optionally display), or a speech-based input mechanism, e.g. to input audio data characterizing a speech waveform of speech representing the input from the user in the natural or computer language and to convert the audio data into tokens representing the speech in the natural or computer language, i.e. representing a transcription of the spoken input. The user interface can also include a text or speech-based output, e.g. a display and / or a text-to-speech subsystem.

[0224] Thus in implementations the agent actions contribute to performing the task. A monitoring system, e.g. a video camera system, may be provided for monitoring the action (if any) which the user actually performs at each time step in case, e.g. due to human error, it is different from the action which the reinforcement learning system instructed the user to perform. The monitoring system can be used to determine whether the task has been completed. Training data may be collected by record the actions which the user actually performed based on the instruction. The reward value of an action may be generated, for example, by comparing the action the user took with a corpus of data showing a human expert performing the task, e.g. using techniques known from imitation learning, or in some other way, e.g. using a trained reward model. A system of this type can learn how to guide a human to perform a task, e.g. avoiding difficult to perform actions.

[0225] Optionally, in any of the above implementations, the observation at any given time step may include data from a previous time step that may be beneficial in characterizing the environment, e.g., the action performed at the previous time step, the reward received at the previous time step, or both.

[0226] FIG. 3 is a flow diagram of an example process 300 for generating motion conditioning data. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a video Atorney Docket No. 45288-0546WO1

[0227] generation system, e.g., the video generation system 100 of FIG. 1A, appropriately programmed in accordance with this specification, can perform the process 300.

[0228] The process 300 provides a specific example of how the system can obtain motion conditioning data, by generating it based on user input. The process is important because it provides a flexible and controllable way for a user to specify the desired motion in the video, rather than relying only on pre-existing or pre-calculated motion data. The system can therefore provide a framework for a wide spectrum of motion-conditioned generative video modeling tasks. For example, the user can provide “motion prompts” to control the movement of specific objects, define the path of a camera, or perform drag-based editing, as described above and shown by FIG. IB. The process 300 can translate these varied user inputs into the specific format of motion conditioning data (e.g., one or more tracks) that the generative model is configured to process.

[0229] The system obtains a motion prompt from a user by way of a user interface (step 302). In some implementations, to obtain the motion prompt from the user by way of the user interface, the system displays, to the user and on the user interface, an interactive user interface element that displays the conditioning video frame in the video. Then, the system receives a user input from the user, by way of user interaction with the interactive user interface element, that defines desired movements of one or more objects shown in the conditioning video frame over a plurality of time steps in the video.

[0230] For example, the system can present the conditioning video frame within the interactive user interface element. A user can then interact with this element by selecting a point on an object (e.g., by clicking on a pixel corresponding to an object’s handle, i.e., the user interface element that can be used to manipulate the object) and then defining a path for that point (e.g.. by dragging the cursor to a new location). This user interaction can correspond to a "drag-based" edit that defines the desired movement of that object point Additionally, in some implementations, to obtain the motion prompt from the user by way of the user interface, the system receives a user input from the user, by way of user interaction with the interactive user interface element, that defines a trajectory of camera poses.

[0231] For example, the interactive user interface element can provide a set of camera controls (e.g., virtual joysticks, timeline keyframes, or buttons for 'pan,' 'tilt,' 'zoom'). The user can interact with these controls to define a sequence of camera poses over the plurality of time steps (e.g., a five-second pan from left to right, or a slow zoom-in on the center of the conditioning video frame). This sequence of inputs defines the trajectory of camera poses. Atorney Docket No. 45288-0546WO1

[0232] The system processes the motion prompt obtained from the user to generate the motion conditioning data for the generative model (step 304).

[0233] In some implementations, to process the motion prompt obtained from the user to generate the motion conditioning data for the generative model, the system processes the user input defining the desired movements of the one or more objects shown in the conditioning video frame over the plurality of time steps in the video to generate a plurality of object control tracks associated with points in the conditioning video frame. Additionally, the plurality of object control tracks are included in the motion conditioning data.

[0234] For example, continuing the "drag-based" edit example above, the system processes this user input (the 'drag' motion) to generate a corresponding object control track. The system maps the path of the user's interaction to a spatio-temporal trajectory, which defines the (x, y) coordinates for that point (the object's handle) for each time step in the plurality of time steps. This generated trajectory is the object control track, which is then included in the motion conditioning data.

[0235] Additionally, in some implementations, to process the motion prompt obtained from the user to generate the motion conditioning data for the generative model, the system processes a trajectory of camera poses (e g., received as user input from a user) to generate motion conditioning data that causes the video generated by the generative model when conditioned on the motion conditioning data to be captured from a perspective of a camera that follows the trajectory of camera poses. That is, the system generates motion tracks for points in the scene such that the resulting video appears to have been filmed by a camera moving along the path defined by the user. For instance, to simulate a "pan left" camera trajectory , the system can generate tracks that move all points in the scene to the right.

[0236] In some cases, to process the trajectory’ of camera poses to generate motion conditioning data, the system generates a plurality of camera control tracks based on a 3D representation of the scene. That is, the system processes the conditioning video frame using a depth estimation model to generate a point cloud representing a geometry of a scene shown in the conditioning video frame. The system then projects the point cloud onto each camera pose in the trajectory of camera poses to generate a plurality of camera control tracks associated with points in the conditioning video frame, where the plurality of camera control tracks are included in the motion conditioning data.

[0237] In some cases, the system is configured to process motion prompts that define both object motion and camera motion simultaneously. That is, to process the motion prompt obtained from the user to generate the motion conditioning data for the generative model, the Atorney Docket No. 45288-0546WO1

[0238] system processes the motion prompt obtained from the user to generate: (i) a plurality' of object control tracks for controlling movement of objects in the video, and (ii) a plurality of camera control tracks for controlling a camera perspective from which the video is captured. The system then generates a plurality of combined tracks for inclusion in the motion conditioning data by combining the object control tracks and the camera control tracks.

[0239] For example, the system can obtain a motion prompt defining a camera trajectory (e.g., "pan left") and another motion prompt defining an object movement (e.g., a user input to make a dog's head turn to the right). The system then generates the plurality of camera control tracks for the pan (e.g., tracks moving the entire scene to the right) and the plurality of object control tracks for the dog's head (e.g., tracks moving the dog's nose to the right). The system then generates the combined tracks by combining these two sets of motion. For example, a track for a point on the dog's nose would be a combination of the 'pan left' motion and the 'turn right' motion, while a track for a point on the background would only have the 'pan left' motion.

[0240] Combining tracks, e.g., the object control tracks and the camera control tracks, can refer to any process that transforms more than one sets of tracks into one set of tracks. For example, the system can perform the combination by first representing the object control tracks as a set of spatial displacements (deltas) relative to their initial positions. These object motion displacements can then be added, via vector addition, to the spatial locations of the camera control tracks at each corresponding time step to produce the combined tracks.

[0241] In some implementations, processing the motion prompt includes expanding a sparse user input into a denser set of tracks. For example, a user input defining a single path (e.g., from a mouse drag) can be processed to generate a grid of point tracks centered on the cursor's location. This grid of tracks follows the user's defined path, allowing a single gesture to control an entire region or object patch.

[0242] In some implementations, the user interface can allow a user to define static tracks. These are tracks that do not move over time (i.e., their spatio-temporal trajectory remains at the same spatial location for all time steps). These static tracks can be used to 'pin' parts of the scene, such as the background, ensuring they remain still while other objects move in accordance with motion prompts.

[0243] FIG. 4 is a flow diagram of an example process 400 for generating a representation of motion conditioning data. For convenience, the process 400 will be described as being performed by a system of one or more computers located in one or more locations. For example, a video generation system, e.g., the video generation system 100 of FIG. 1A, Atorney Docket No. 45288-0546WO1

[0244] appropriately programmed in accordance with this specification, can perform the process 400.

[0245] The process 400 provides an example of how the system generates a representation of conditioning data that the system will process to generate a video, specifically by generating a representation of the motion conditioning data as a spatio-temporal volume. The process 400 is important because it converts the motion conditioning data, which includes one or more discrete tracks, into a dense, structured data format (the spatio-temporal volume) that the system’s generative model (e.g., a convolutional or attention-based neural network) can efficiently process as an input.

[0246] The system initializes the spatio-temporal volume with default values (step 402). In some implementations, the system initializes each entry in the spatio-temporal volume to have value zero.

[0247] This initialization step ensures that the spatio-temporal volume starts in a known, "empty" state, with no pre-existing data, before the system begins to encode the track information into it.

[0248] The system then updates the spatio-temporal volume to encode each track in the one or more tracks (step 404). In order for the system to update the spatio-temporal volume to encode each track in the one or more tracks, the system performs steps 406A and 406B for each track.

[0249] The system determines a respective embedding for the track (step 406A).

[0250] An '‘embedding” is an ordered set of numbers. The "embedding for the track" is a specific ordered set of numbers assigned to a single track. This embedding acts as a unique identifier for that track, allowing the generative model to distinguish it from other tracks that might be present in the same spatio-temporal volume.

[0251] In some cases, to determine a respective embedding for the track, the system assigns a unique and randomly chosen non-zero embedding to the track.

[0252] Afterwards, the system updates each location in the spatio-temporal volume that corresponds to a spatial location and time step visited by the track to include the embedding of the track (step 406B).

[0253] A “location in the spatio-temporal volume” can refer to a specific coordinate defined by a time step t and spatial coordinates (x, y). So, the volume itself can be considered a 3D grid (Time. Height, Width), where each point in this grid stores channel values corresponding to the channel dimension. For example, consider a spatio-temporal volume with a T(16)xH(64)xW(64)xC(3) dimensional space. At every coordinate (t, x, y) within the 16 time Atorney Docket No. 45288-0546WO1

[0254] steps and 64x64 spatial grid, the volume can store a 3 -dimensional vector. If a track has a 3-dimensional embedding (e.g., [0.1, 0.8, 0.3]) and its trajectory places it at spatial location (x=50, y=25) at time step t=3, the system can update the volume by setting the 3-dimensional vector at the coordinate (t=3, x=50, y=25) to be that embedding, [0.1, 0.8, 0.3],

[0255] In some implementations, the motion conditioning data defines, for each time step in each track, whether the track is off-screen or occluded at the time step. For these implementations, for any time step where the track is off-screen or occluded, the system, when performing step 406B, modifies the embedding included in the spatio-temporal volume at the spatial location and time step to reflect that the track is off-screen or occluded at the time step.

[0256] In some cases, to modify the embedding included in the spatio-temporal volume at the spatial location and time step, the system modifies the embedding to be a zero-embedding with a zero in each entry.

[0257] So, for example, if a track follows a person’s face, and that person is visible for the first few time steps, the system can update the spatio-temporal volume at the face’s coordinates with the track’s unique non-zero embedding. Then, if the motion conditioning data then defines that track as “occluded” (e.g., the person walks behind a building), the system can modify the embedding at the track’s corresponding locations to be a zeroembedding for those occluded time steps. When the person then becomes visible again later in the video, the system can resume updating the volume at the face’s new. visible coordinates with the original non-zero embedding.

[0258] In some implementations, the motion conditioning data defines, for each time step in each track, an adherence value that defines a degree to which the video should adhere to track at the time step. For these implementations, to update each location in the spatio-temporal volume that corresponds to a spatial location and time step visited by the track to include the embedding of the track for each track, the system can modify the embedding included in the spatio-temporal volume at the spatial location and time step based on a corresponding adherence value for the track at the time step.

[0259] In some cases, to modify the embedding included in the spatio-temporal volume at the spatial location and time step based on the corresponding adherence value for the track at the time step, the system modifies the embedding included in the spatio-temporal volume at the spatial location and time step to be a combination of: (i) the embedding for the track, and (ii) an embedding representing the adherence value for the track at the time step. Atorney Docket No. 45288-0546WO1

[0260] For example, if a track has a unique embedding vector, and at a particular time step, the motion conditioning data 104 indicates a strong degree of adherence, the system 100 determines the corresponding "strong adherence" embedding vector. The system 100 then combines these two vectors, for instance by performing an element-wise summation, and modifies the embedding in the spatio-temporal volume at the track's location for that time step to be this new combined vector. If at a subsequent time step, the adherence value for the same track changes to a weaker degree, the system 100 determines the corresponding "weaker adherence" embedding vector and combines it with the original track embedding. This second combined vector is then used to modify the embedding at the track's new location for that subsequent time step.

[0261] FIG. 5 is a flow diagram of an example process 500 for training a generative model. For convenience, the process 500 will be described as being performed by a system of one or more computers located in one or more locations. For example, a video generation system, e.g., the video generation system 100 of FIG. 1A, appropriately programmed in accordance with this specification, can perform the process 500.

[0262] The process 500 illustrates a generalized training loop for training the generative model. The process can be adapted depending on the specific architecture of the generative model and the training strategy.

[0263] For example, the process 500 can represent training the generative model from scratch (i. e. , randomly initialized parameters).

[0264] In other implementations, the generative model can be an adaptation of a large, pretrained generative model. In this case, the training process 500 can involve fine-tuning the parameters of the pre-trained model.

[0265] In yet another implementation, the system can keep the pre-trained model's parameters "locked" or "frozen" and training process 500 trains only a new, additional component. For example, in an architecture like ControlNet, the "trainable parameters" updated would be only those of the "trainable copy" and "zero convolutions", while the original pre-trained model's parameters are not updated.

[0266] The following description of the process 500 can apply to any of these strategies. The system obtains a training data set that includes training examples (step 502). In some implementations, each training example includes: (i) conditioning data that includes a conditioning video frame and motion conditioning data, and (ii) a target video that is consistent with the conditioning data.

[0267] The system can obtain the training data using any of a variety of methods. Atorney Docket No. 45288-0546WO1

[0268] For example, the system can obtain the training data set by loading it from a system-maintained memory (e.g., a local database or cloud storage), receiving it from a user (e.g., via a data upload interface), or obtaining it from another system (e.g., a data-sourcing pipeline).

[0269] In some implementations, for a plurality of the training examples, the system generates

[0270] the motion conditioning data for the training example. For example, the system can process the target video of the training example using a point tracking algorithm to generate one or more tracks for inclusion in the motion conditioning data of the training example. For example, the system can use the BootsTAP point tracking method, as described in arXiv:2402.00847, as the point tracking algorithm.

[0271] As an example, the system can take a target video from the training data set, select one of its frames as the conditioning video frame, and then apply a point tracking algorithm (e.g., BootsTAP) to the target video. The algorithm follows points from the conditioning video frame through the subsequent frames of the target video, thereby generating the spatiotemporal trajectories that form the one or more tracks. These generated tracks (as the motion conditioning data) and the selected conditioning video frame are then paired with the target video to form a complete training example.

[0272] In some implementations, the system further augments the motion conditioning data for the training examples in the training data. That is, for a plurality of the training examples, when generating the motion conditioning data, the system also, for one or more tracks: (i) selects a respective adherence value for one or more time steps in the track; and (ii) modifies the track by randomly displacing the spatial location of the track at that time step. The amount of this random displacement (e.g., the number of pixels) is dependent upon the selected adherence value.

[0273] For example, for a given track in a training example, the system can randomly select a respective adherence value for a particular time step, where the adherence value ranges from 0 to 1. If the adherence value is 1 (indicating perfect adherence), the system modifies the track by leaving the spatial location at that time step unchanged. If the adherence value is less than 1, the system modifies the track by randomly displacing the spatial location of the track at that time step. The direction of this displacement can be random. The amount of this random displacement (e.g., the number of pixels) is dependent upon the adherence value; for instance, the system can be configured such that adherence values closer to 0 result in a larger magnitude of displacement, while values closer to 1 result in a smaller magnitude of displacement. Atorney Docket No. 45288-0546WO1

[0274] The system, for each training example, generates an output (step 504). The “output” is the result of the generative model processing the conditioning data.

[0275] In the context of a generative model that is a diffusion generative model, the output can be the model's prediction of the noise that was added to the target video in a reverse diffusion process. For example, the system provides the model with a “noisy” version of the target video and the conditioning data, and the model's output is its estimation of the original noise.

[0276] The system evaluates an objective using all training examples and respective outputs (step 506).

[0277] For example, continuing with the example diffusion generative model example of step 504, the objective (e.g., a loss function) can measure the error between the model’s predicted noise (the “output” from step 504) and the actual noise that was added to the target video for the reverse diffusion process. For example, the objective can be the mean squared error between the predicted noise and the actual noise.

[0278] The system updates trainable parameters to optimize the objective (step 508).

[0279] The system can update the trainable parameters of the generative model to optimize the objective in any variety of ways, e g., gradient based method, evolutionary algorithmbased method, Bayesian optimization, etc.

[0280] For example, the system can optimize the objective using any of a variety of gradient descent techniques (e.g.. batch gradient descent, stochastic gradient descent, or mini-batch gradient descent) that include the use of a backpropagation technique to estimate the gradient of the loss with respect to trainable parameters of the neural network and to update the learnable parameters accordingly.

[0281] Generally, the system repeats the above steps until one or more criteria are satisfied (e.g., the system performs a pre-determined number of iterations, the updates to the trainable parameters no longer exceed a pre-determined magnitude of change, a metric regarding a validation dataset exceeds a pre-determined value, and so on).

[0282] FIG. 6 is an example 600 of the performance of the described techniques.

[0283] Example 600 is a table showing quantitative evaluation results from a performance comparison. The rows of the table correspond to different methods ("Image Conductor," "Drag Any thing," and "Ours," which represents the described techniques) and are grouped by the number of tracks (N) included in the motion conditioning data (e.g., N=l, N=16, N=512, N=2048). The columns correspond to various quantitative metrics used to evaluate the quality and accuracy of the generated video, including PSNR (Peak Signal-to-Noise Ratio), SSIM Atorney Docket No. 45288-0546WO1

[0284] (Structural Similarity Index Measure), LPIPS (Learned Perceptual Image Patch Similarity), FVD (Frechet Video Distance), and EPE (End-Point Error).

[0285] Example 600 shows that the described techniques ("Ours") achieves superior performance compared to the other, baseline methods ("Image Conductor," "Drag Any thing") across the quantitative metrics, particularly when the motion conditioning data is dense (i.e., includes a high number of tracks). For instance, at N=2048 tracks, the described techniques achieve the highest (best) PSNR and SSIM scores, and the lowest (best) FVD and EPE scores. Example 600 shows that the described technique’s ability to generate a video is consistent with the conditioning data improves as the number of tracks in the motion conditioning data increases.

[0286] FIG. 7 is an example 700 of the performance of the described techniques.

[0287] Example 700 is a table showing results from a human user study. The rows of the table list baseline methods ("Image Conductor," "Drag Anything") that were compared against the described techniques. The columns correspond to the subjective criteria that human users were asked to evaluate: "Motion Adherence" (e.g., how well the video adheres to the spatio-temporal trajectories defined by the motion conditioning data), "Motion Quality " (e g., the realism and smoothness of the motion), and "Visual Quality" (e g., the overall visual fidelity of the generated video).

[0288] Example 700 shows that the described techniques are strongly preferred by human users over the baseline methods. The values of the table represent the percentage of time users preferred the described techniques over the listed baseline for the corresponding metric. Example 700 shows that the described techniques not only perform better on quantitative metrics (as shown in FIG. 6) but also generates videos that are perceived by human evaluators as having higher quality and better adherence to the motion conditioning data.

[0289] FIG. 8 is an example 800 of the performance of the described techniques.

[0290] Example 800 is a table showing an ablation study. The rows of the table compare different aspects of the described techniques, grouped by the number of tracks (N=4, N=2048) and by the type of track representation used (e.g., "Sparse," "Dense + Sparse." "Dense"). The columns correspond to the same quantitative performance metrics described above in FIG. 6 (PSNR, SSIM, LPIPS, FVD, EPE).

[0291] Example 800 shows that the "Dense" configuration, particularly when used with a high number of tracks (N=2048), provides substantially better performance across all metrics compared to "Sparse" or "Dense + Sparse" configurations. This demonstrates that using dense motion conditioning data, e.g., motion conditioning data that includes a respective track for a Atorney Docket No. 45288-0546WO1

[0292] high percentage of points in the conditioning video frame, is a particularly useful aspect of the described techniques for achieving improved performance and generating a higher-fidelity video.

[0293] In this specification, the term "configured" is used in relation to computing systems and environments, as well as computer program components. A computing system or environment is considered "configured" to perform specific operations or actions when it possesses the necessary software, firmware, hardware, or a combination thereof, enabling it to carry out those operations or actions during operation. For instance, configuring a system might involve installing a software library with specific algorithms, updating firmware with new instructions for handling data, or adding a hardware component for enhanced processing capabilities. Similarly, one or more computer programs are "configured" to perform particular operations or actions when they contain instructions that, upon execution by a computing device or hardware, cause the device to perform those intended operations or actions.

[0294] The embodiments and functional operations described in this specification can be implemented in various forms, including digital electronic circuitry, software, firmware, computer hardware (encompassing the disclosed structures and their structural equivalents), or any combination thereof. The subject matter can be realized as one or more computer programs, essentially modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by or to control the operation of a computing device or hardware. The storage medium can be a storage device such as a hard drive or solid-state drive (SSD), a storage medium, a random or serial access memory device, or a combination of these. Additionally or alternatively, the program instructions can be encoded on a transmitted signal, such as a machine-generated electrical, optical, or electromagnetic signal, designed to carry information for transmission to a receiving device or system for execution by a computing device or hardware. Furthermore, implementations may leverage emerging technologies like quantum computing or neuromorphic computing for specific applications, and may be deployed in distributed or cloud-based environments where components reside on different machines or within a cloud infrastructure.

[0295] The term "computing device or hardware" refers to the physical components involved in data processing and encompasses all types of devices and machines used for this purpose. Examples include processors or processing units, computers, multiple processors or computers working together, graphics processing units (GPUs), tensor processing units (TPUs), and specialized processing hardware such as field-programmable gate arrays Atorney Docket No. 45288-0546WO1

[0296] (FPGAs) or application-specific integrated circuits (ASICs). In addition to hardware, a computing device or hardware may also include code that creates an execution environment for computer programs. This code can take the form of processor firmware, a protocol stack, a database management system, an operating system, or a combination of these elements. Embodiments may particularly benefit from utilizing the parallel processing capabilities of GPUs, in a General-Purpose computing on Graphics Processing Units (GPGPU) context, where code specifically designed for GPU execution, often called kernels or shaders, is employed. Similarly, TPUs excel at running optimized tensor operations crucial for many machine learning algorithms. By leveraging these accelerators and their specialized programming models, the system can achieve significant speedups and efficiency gains for tasks involving artificial intelligence and machine learning, particularly in areas such as computer vision, natural language processing, and robotics.

[0297] A computer program, also referred to as software, an application, a module, a script, code, or simply a program, can be written in any programming language, including compiled or interpreted languages, and declarative or procedural languages. It can be deployed in various forms, such as a standalone program, a module, a component, a subroutine, or any other unit suitable for use within a computing environment. A program may or may not correspond to a single file in a file system and can be stored in various ways. This includes being embedded within a file containing other programs or data (e.g., scripts within a markup language document), residing in a dedicated file, or distributed across multiple coordinated files (e.g., files storing modules, subprograms, or code segments). A computer program can be executed on a single computer or across multiple computers, whether located at a single site or distributed across multiple sites and interconnected through a data communication network. The specific implementation of the computer programs may involve a combination of traditional programming languages and specialized languages or libraries designed for GPGPU programming or TPU utilization, depending on the chosen hardware platform and desired performance characteristics.

[0298] In this specification, the term "engine" broadly refers to a software-based system, subsystem, or process designed to perform one or more specific functions. An engine is typically implemented as one or more software modules or components installed on one or more computers, which can be located at a single site or distributed across multiple locations. In some instances, one or more dedicated computers may be used for a particular engine, while in other cases, multiple engines may operate concurrently on the same one or more computers. Examples of engine functions within the context of Al and machine learning Atorney Docket No. 45288-0546WO1

[0299] could include data pre-processing and cleaning, feature engineering and extraction, model training and optimization, inference and prediction generation, and post-processing of results. The specific design and implementation of engines will depend on the overall architecture and the distribution of computational tasks across various hardware components, including CPUs, GPUs, TPUs, and other specialized processors.

[0300] The processes and logic flows described in this specification can be executed by one or more programmable computers running one or more computer programs to perform functions by operating on input data and generating output. Additionally, graphics processing units (GPUs) and tensor processing units (TPUs) can be utilized to enable concurrent execution of aspects of these processes and logic flows, significantly accelerating performance. This approach offers significant advantages for computationally intensive tasks often found in Al and machine learning applications, such as matrix multiplications, convolutions, and other operations that exhibit a high degree of parallelism. By leveraging the parallel processing capabilities of GPUs and TPUs, significant speedups and efficiency¬ gains compared to relying solely on CPUs can be achieved. Alternatively or in combination with programmable computers and specialized processors, these processes and logic flows can also be implemented using specialized processing hardware, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), for even greater performance or energy efficiency in specific use cases.

[0301] Computers capable of executing a computer program can be based on general-purpose microprocessors, special -purpose microprocessors, or a combination of both. They can also utilize any other ty pe of central processing unit (CPU). Additionally, graphics processing units (GPUs), tensor processing units (TPUs), and other machine learning accelerators can be employed to enhance performance, particularly for tasks involving artificial intelligence and machine learning. These accelerators often work in conjunction with CPUs, handling specialized computations while the CPU manages overall system operations and other tasks. Typically , a CPU receives instructions and data from read-only memory- (ROM), random access memory (RAM), or both. The elements of a computer include a CPU for executing instructions and one or more memory devices for storing instructions and data. The specific configuration of processing units and memory- will depend on factors like the complexity- of the Al model, the volume of data being processed, and the desired performance and latency requirements. Embodiments can be implemented on a wide range of computing platforms, from small embedded devices with limited resources to large-scale data center systems with Atorney Docket No. 45288-0546WO1

[0302] high-performance computing capabilities. The system may include storage devices like hard drives, SSDs, or flash memory for persistent data storage.

[0303] Computer-readable media suitable for storing computer program instructions and data encompass all forms of non-volatile memory, media, and memory devices. Examples include semiconductor memory devices such as read-only memory7(ROM), solid-state drives (SSDs), and flash memory devices: hard disk drives (HDDs); optical media; and optical discs such as CDs, DVDs, and Blu-ray discs. The specific type of computer-readable media used will depend on factors such as the size of the data, access speed requirements, cost considerations, and the desired level of portability7or permanence.

[0304] To facilitate user interaction, embodiments of the subject matter described in this specification can be implemented on a computing device equipped with a display device, such as a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display, for presenting information to the user. Input can be provided by the user through various means, including a key board), touchscreens, voice commands, gesture recognition, or other input modalities depending on the specific device and application. Additional input methods can include acoustic, speech, or tactile input, while feedback to the user can take the form of visual, auditory7, or tactile feedback. Furthermore, computers can interact with users by7exchanging documents with a user's device or application. This can involve sending web content or data in response to requests or sending and receiving text messages or other forms of messages through mobile devices or messaging platforms. The selection of input and output modalities will depend on the specific application and the desired form of user interaction.

[0305] Machine learning models can be implemented and deployed using machine learning frameworks, such as TensorFlow or JAX. These frameworks offer comprehensive tools and libraries that facilitate the development, training, and deployment of machine learning models.

[0306] Embodiments of the subject matter described in this specification can be implemented within a computing system comprising one or more components, depending on the specific application and requirements. These may include a back-end component, such as a back-end server or cloud-based infrastructure; an optional middleware component, such as a middleware server or application programming interface (API), to facilitate communication and data exchange: and a front-end component, such as a client device with a user interface, a web browser, or an app, through which a user can interact with the implemented subject matter. For instance, the described functionality could be implemented solely on a client Atorney Docket No. 45288-0546WO1

[0307] device (e.g., for on-device machine learning) or deployed as a combination of front-end and back-end components for more complex applications. These components, when present, can be interconnected using any form or medium of digital data communication, such as a communication network like a local area network (LAN) or a wide area network (WAN) including the Internet. The specific system architecture and choice of components will depend on factors such as the scale of the application, the need for real-time processing, data security requirements, and the desired user experience.

[0308] The computing system can include clients and servers that may be geographically separated and interact through a communication network. The specific ty pe of network, such as a local area network (LAN), a wide area network (WAN), or the Internet, will depend on the reach and scale of the application. The client-server relationship is established through computer programs running on the respective computers and designed to communicate with each other using appropriate protocols. These protocols may include HTTP, TCP / IP, or other specialized protocols depending on the nature of the data being exchanged and the security requirements of the system. In certain embodiments, a server transmits data or instructions to a user's device, such as a computer, smartphone, or tablet, acting as a client. The client device can then process the received information, display results to the user, and potentially send data or feedback back to the server for further processing or storage. This allows for dynamic interactions between the user and the system, enabling a wide range of applications and functionalities.

[0309] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0310] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be Attorney Docket No. 45288-0546WO1

[0311] performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0312] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0313] What is claimed is:

Claims

Attorney Docket No. 45288-0546WO1CLAIMS1. A method performed by one or more computers, the method comprising:generating a video using a generative model, comprising:obtaining conditioning data specifying desired characteristics of the video to be generated by the generative model, wherein the conditioning data comprises:a conditioning video frame for a time step in the video; and motion conditioning data that comprises one or more tracks, wherein each track is associated with a respective point in the conditioning video frame and defines a spatio-temporal trajectory of the point through a plurality of time steps in the video; and processing a representation of the conditioning data using the generative model to generate the video; andoutputting the video generated using the generative model.

2. The method of claim 1 , further comprising processing the conditioning data to generate the representation of the conditioning data that is processed using the generative model, comprising:generating a representation of the motion conditioning data as a spatio-temporal volume.

3. The method of claim 2, wherein the spatio-temporal volume includes a temporal dimension indexing time steps of the video and a plurality of spatial dimensions corresponding to spatial dimensions of the video.

4. The method of any one of claims 2-3, wherein generating the representation of the motion conditioning data as a spatio-temporal volume comprises:initializing the spatio-temporal volume with default values; andupdating the spatio-temporal volume to encode each track in the one or more tracks, comprising, for each track:determining a respective embedding for the track; andupdating each location in the spatio-temporal volume that corresponds to a spatial location and time step visited by the track to include the embedding of the track.

5. The method of claim 4, wherein for each track, determining the embedding for the track comprises:assigning a unique and randomly chosen non-zero embedding to the track.Attorney Docket No. 45288-0546WO16. The method of any one of claims 4-5, wherein initializing the spatio-temporal volume with default values comprises:initializing each entry in the spatio-temporal volume to have value zero.

7. The method of any one of claims 2-6, wherein the spatio-temporal volume includes a channel dimension.

8. The method of any one of claims 4-7, wherein the motion conditioning data defines, for each time step in each track, whether the track is off-screen or occluded at the time step; andwherein for each track, updating each location in the spatio-temporal volume that corresponds to a spatial location and time step visited by the track to include the embedding of the track comprises, for any time step where the track is off-screen or occluded:modifying the embedding included in the spatio-temporal volume at the spatial location and time step to reflect that the track is off-screen or occluded at the time step.

9. The method of claim 8, wherein modifying the embedding included in the spatiotemporal volume at the spatial location and time step comprises modifying the embedding included in the spatio-temporal volume at the spatial location and time step to be a zeroembedding with a zero in each entry.

10. The method of any one of claims 4-9, wherein the motion conditioning data defines, for each time step in each track, an adherence value that defines a degree to which the video should adhere to the track at the time step; andwherein for each track, updating each location in the spatio-temporal volume that corresponds to a spatial location and time step visited by the track to include the embedding of the track comprises:modifying the embedding included in the spatio-temporal volume at the spatial location and time step based on a corresponding adherence value for the track at the time step.

11. The method of claim 10, wherein modifying the embedding included in the spatiotemporal volume at the spatial location and time step based on the corresponding adherence value for the track at the time step comprises:modifying the embedding included in the spatio-temporal volume at the spatialAttorney Docket No. 45288-0546WO1location and time step to be a combination of: (i) the embedding for the track, and (ii) an embedding representing the adherence value for the track at the time step.

12. The method of any preceding claim, wherein the motion conditioning data comprises a respective track for at least 10%, or at least 20%, or at least 30%, or at least 40%, or at least 50%, or at least 60%, or at least 70%, or at least 80%. or at least 90% of points in the conditioning video frame.

13. The method of any preceding claim, wherein the motion conditioning data comprises a respective track for even’ point in the conditioning video frame.

14. The method of any preceding claim, wherein the generative model is a generative diffusion model.

15. The method of any preceding claim, wherein the generative model has been trained on a set of training examples; andwherein each training example comprises: (i) conditioning data comprising a conditioning video frame and motion conditioning data, and (ii) a target video that is consistent with the conditioning data.

16. The method of claim 15, wherein for a plurality of the training examples, the motion conditioning data for the training example has been generated by performing operations comprising:processing the target video of the training example using a point tracking algorithm to generate one or more tracks for inclusion in the motion conditioning data of the training example.

17. The method of any one of claims 15-16, wherein for a plurality of the training examples, generating the motion condition data for the training example further comprises, for one or more tracks included in the motion conditioning data:selecting a respective adherence value for one or more time steps in the track; and modifying the track by, for each time step in the track, randomly displacing a spatial location of the track at the time step by a number of pixels that is dependent upon the adherence value for the time step.Attorney Docket No. 45288-0546WO118. The method of any preceding claim, further comprising:obtaining a motion prompt from a user by way of a user interface; and processing the motion prompt obtained from the user to generate the motion conditioning data for the generative model.

19. The method of claim 18, wherein obtaining the motion prompt from the user by way of the user interface comprises:displaying, to the user and on the user interface, an interactive user interface element that displays the conditioning video frame in the video; andreceiving a user input from the user, by way of user interaction with the interactive user interface element, that defines desired movements of one or more objects shown in the conditioning video frame over a plurality of time steps in the video.

20. The method of claim 19, wherein processing the motion prompt obtained from the user to generate the motion conditioning data for the generative model comprises:processing the user input defining the desired movements of the one or more objects shown in the conditioning video frame over the plurality of time steps in the video to generate a plurality' of object control tracks associated with points in the conditioning video frame; wherein the plurality of object control tracks are included in the motion conditioning data.

21. The method of any one of claims 19-20, wherein outputting the video generated using the generative model comprises:outputting only a final video frame in the video generated using the generative model in order to perform drag-based image editing of an image represented by the conditioning video frame.

22. The method of any one of claims 18-21, wherein obtaining the motion prompt from the user by way of the user interface comprises:receiving a user input from the user, by way of user interaction with the interactive user interface element, that defines a trajectory' of camera poses; andwherein processing the motion prompt obtained from the user to generate the motion conditioning data for the generative model comprises:processing the trajectory of camera poses to generate motion conditioning data that causes the video generated by the generative model when conditioned on the motionAtorney Docket No. 45288-0546WO1conditioning data to be captured from a perspective of a camera that follows the trajectory of camera poses.

23. The method of claim 22, wherein processing the trajectory of camera poses to generate motion conditioning data that causes the video generated by the generative model when conditioned on the motion condition data to be captured from the perspective of the camera that follows the trajectory of camera poses comprises:processing the conditioning video frame using a depth estimation model to generate a point cloud representing a geometry7of a scene shown in the conditioning video frame;projecting the point cloud onto each camera pose in the trajectory of camera poses to generate a plurality of camera control tracks associated with points in the conditioning video frame;wherein the plurality7of camera control tracks are included in the motion conditioning data.

24. The method of any one of claims 18-23, wherein processing the motion prompt obtained from the user to generate the motion conditioning data for the generative model comprises:processing the motion prompt obtained from the user to generate: (i) a plurality of object control tracks for controlling movement of objects in the video, and (ii) a plurality7of camera control tracks for controlling a camera perspective from which the video is captured; andgenerating a plurality of combined tracks for inclusion in the motion conditioning data by combining the object control tracks and the camera control tracks.

25. The method of any preceding claim, wherein outputting the video comprises storing the video in a memory, or presenting the video on a display of a user device, or transmitting the video over a data communication network.

26. The method of any preceding claim, wherein the generative model has been configured through training to cause the video to have characteristics that are consistent with the conditioning data.

27. The method of claim 20, wherein the user input defines a manipulation of a geometric primitive associated with an object in the conditioning video frame, and wherein processingAttorney Docket No. 45288-0546WO1the user input comprises translating the manipulation of the geometric primitive into the plurality of object control tracks.

28. The method of claim 1, wherein obtaining the conditioning data comprises:processing a source video to generate the one or more tracks comprising the motion conditioning data; and wherein the conditioning video frame is obtained from a source that is distinct from the source video, such that the video generated using the generative model included transferred motion from the source video to the conditioning video frame.

29. A method performed by one or more computers, the method comprising:obtaining motion conditioning data that comprises one or more tracks, wherein each track is associated with a point in a video frame of a video and defines a spatio-temporal trajectory’ of the point through a plurality’ of time steps in the video;processing a model input that comprises the motion conditioning data using a machine learning model to generate a prediction output; andproviding the prediction output.

30. The method of claim 29, further comprising generating a representation of the motion conditioning data as a spatio-temporal volume; andwherein the model input to the machine learning model comprises the representation of the motion conditioning data as the spatio-temporal volume.

31. The method of any one of claims 29-30, wherein the model input to the machine learning model further comprises the video.

32. The method of any one of claims 29-31, wherein the video captures an environment in a vicinity of an agent at a time step; andwherein the machine learning model comprises an action selection neural network that processes the model input to generate an action selection output; andwherein the method further comprises:selecting an action to be performed by the agent at the time step using the action selection output.

33. The method of claim 32, wherein the action selection output comprises a score distribution over a set of actions; andwherein selecting the action to be performed by the agent at the time step using theAttorney Docket No. 45288-0546WO1action selection output comprises:selecting an action having a highest score under the score distribution as the action to be performed by the agent at the time step.

34. The method of any one of claims 32-33, wherein the environment is a real-world environment, the agent is a mechanical agent interacting with the environment, and the action is to be performed by the agent at the time step in an attempt to accomplish a specified task.

35. The method of any one of claims 29-31, wherein the prediction output defines a classification of the video into a class from a set of one or more classes.

36. The method of claim 35, wherein machine learning model performs an action classification task, and wherein each of the one or more classes corresponds to a respective action.

37. The method of claim 35, wherein the machine learning model performs a scene classification task, and wherein each of the one or more classes corresponds to a respective type of scene.

38. The method of claim 35, wherein the machine learning model performs an object detection task, and wherein each of the one or more classes corresponds to a respective type of object.

39. The method of any one of claims 35-38, wherein the video captures a real-world environment and the prediction output characterizes one or more of an action performed in the environment, a scene of the environment, and an object in the environment.

40. A system comprising:one or more computers; andone or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the respective method of any one of claims 1-39.

41. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations of the respective method of any one of claims 1-39.

Images