Generative ai models using inductive moment matching

Abstract

Described are methods and systems for training generative artificial intelligence (Al) models. A method for training a generative Al model can include obtaining a first dataset comprising an observed data sample from a data distribution of observed data and a second dataset comprising a prior sample from a prior distribution; synthesizing first data at a first time step based on the observed data sample and the prior sample; synthesizing second data at a second time step based on the observed data sample and the first data; generating, using a neural network, a first output and a second output at the target time step; and obtaining a trained generative Al model by updating parameters of the neural network, wherein the updating comprises at least computing a loss between a first distribution of the first output and a second distribution of the second output.

Description

WSGRNo. 67054-701.601GENERATIVE Al MODELS USING INDUCTIVE MOMENT MATCHINGCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 760,396, filed February 19, 2025, which is incorporated by reference herein in its entirety.BACKGROUND

[0002] Generative artificial intelligence (Al) models can enable technical applications related to generating, e.g., images, videos, audio, and the like. However, achieving efficient computing inference, high-fidelity outputs, and stable model training remain technical challenges. For example, diffusion models and flow matching models can generate high-quality samples but are slow at inference. Further, distilling them into one or few-step models can lead to models that are unstable and require extensive tuning.

[0003] Recognized herein is a need for generative Al models that can at least be trained in one or few-steps or stages while remaining stable and generating high-fidelity outputs.SUMMARY

[0004] Provided herein are methods and systems that can at least train generative Al models in one or few-steps while remaining stable and generating high-fidelity outputs. To resolve technical trade-offs between, e.g., efficient inference and stability, the present disclosure provides methods and systems for inductive moment matching (IMM), a new class of generative Al models for one-or few-step sampling with single-stage training. Unlike distillation, IMM does not require pretraining initialization and optimization of two or more networks, which can waste computing and network resources. Unlike consistency models, IMM can provide distribution-level convergence and remain stable under various hyperparameters and standard model architectures. As demonstrated herein, IMM can surpass diffusion models on ImageNet-256×256 with 1.99 Fréchet inception distance (FID) using only one or few stages of 8 inference steps. Further, IMM can achieve state-of-the-art 2-step FID of 1.98 with one or few stages on Canadian Institute for Advanced Research (CIFAR)-10 for a model trained from scratch.

[0005] In an aspect, disclosed herein is a method for training a generative artificial intelligence (AI) model, the method comprising: (a) obtaining (i) a first dataset comprising an observed data sample from a data distribution of observed data and (ii) a second dataset comprising a prior sample from a prior distribution; (b) synthesizing first data at a first time step based on the observed data sample and the prior sample; (c) synthesizing second data at a second time step based on the observed data sample and the first data, wherein the second time step is between a target time step and the first time step; (d) generating, using a neural network, a first output and a second output at the target time step, wherein (i) the first output is obtained using the first dataWSGRNo. 67054-701.601with the target time step and the first time step as conditioning inputs and (ii) the second output is obtained using the second data with the target time step and the second time step as conditioning inputs; and (e) obtaining a trained generative Al model by updating parameters of the neural network, wherein the updating comprises at least computing a loss between a first distribution of the first output and a second distribution of the second output. In some embodiments, the method further comprises generating the first dataset and the second dataset. In some embodiments, the observed data comprises data obtained from image data, video data, audio data, text data, sensor data, geospatial data, or metadata. In some embodiments, the observed data comprises continuous data, structured data, or unstructured data. In some embodiments, the prior sample comprises random noise. In some embodiments, the prior distribution comprises a Gaussian distribution, a uniform random distribution, or a Dirichlet distribution. In some embodiments, synthesizing the first data comprises sampling from a first marginal interpolating distribution parametrized by the observed data sample and the prior sample. In some embodiments, the first marginal interpolating distribution is determined using a denoising diffusion implicit model. In some embodiments, synthesizing the second data comprises sampling from a second marginal interpolating distribution parametrized by the observed data sample and the first data. In some embodiments, the second marginal interpolating distribution is determined using a denoising diffusion implicit model. In some embodiments, the first output comprises a first marginal preserving interpolant between the first data and the observed data. In some embodiments, the first marginal preserving interpolant preserves a marginal distribution of the first output for a plurality of time steps. In some embodiments, the second output comprises a second marginal preserving interpolant between the second data and the observed data. In some embodiments, the second marginal preserving interpolation preserves a marginal distribution of the second first output for a plurality of time steps. In some embodiments, the first time step is sampled from a first time distribution. In some embodiments, the target time step is sampled from a second time distribution, parametrized by the first time step. In some embodiments, the second time step is determined using a mapping function parametrized by the first time step and the target time step. In some embodiments, the computed loss is determined using maximum mean discrepancy. In some embodiments, the maximum mean discrepancy is determined using a kernel function. In some embodiments, the method further comprises applying a stop gradient operation on the second output when computing the loss. In some embodiments, the method further comprises prompting the trained generative Al model with at least a request for a type of data and a category of data. In some embodiments, the method further comprises generating, using the trained generative Al model, new data based at least on the prompting. In some embodiments, the new data is determined to be statistically similar to the observed data. In some embodiments, the new data isWSGRNo. 67054-701.601determined to be statistically similar to the observed data with a Fréchet inception distance (FID) of less than about 2.5. In some embodiments, the method further comprises automatically rendering the new data on a graphical user interface (GUI), wherein the GUI comprises dynamically selectable objects or elements to modify the new data or append additional data to the new data in real time. In some embodiments, the trained generative Al model comprises a stable model trained in one stage. In some embodiments, the method further comprises repetitively synthesizing a third data or more data at successive time steps, based on data synthesized at a previous time step. In some embodiments, the first dataset comprises a first training dataset, wherein the second dataset comprises a second training dataset, and wherein each dataset is used to obtain the trained generative Al model.

[0006] In another aspect, disclosed herein is a method comprising: (a) training a generative artificial intelligence (AI) model in one stage, wherein the model is stable; and (b) generating, using the trained generative AI model, new data, wherein the new data comprises high fidelity data.

[0007] In another aspect, disclosed herein is a method comprising: (a) obtaining a trained generative artificial intelligence (Al) model according to any one of claims herein; (b) prompting the trained generative AI model with at least a request for a type of data and a category of data; and (c) generating, using the trained generative AI model, new data based at least on the prompting in (b).

[0008] In another aspect, disclosed herein is a system comprising at least one processor and instructions executable by the at least one processor to cause the at least one processor to perform operations comprising: (a) obtaining (i) a first dataset comprising an observed data sample from a data distribution of observed data and (ii) a second dataset comprising a prior sample from a prior distribution; (b) synthesizing first data at a first time step based on the observed data sample and the prior sample; (c) synthesizing second data at a second time step based on the observed data sample and the first data, wherein the second time step is between a target time step and the first time step; (d) generating, using a neural network, a first output and a second output at the target time step, wherein (i) the first output is obtained using the first data with the target time step and the first time step as conditioning inputs and (ii) the second output is obtained using the second data with the target time step and the second time step as conditioning inputs; and (e) obtaining a trained generative Al model by updating parameters of the neural network, wherein the updating comprises at least computing a loss between a first distribution of the first output and a second distribution of the second output. In some embodiments, the system further comprises generating the first dataset and the second dataset. In some embodiments, the observed data comprises data obtained from image data, video data, audio data, text data, sensor data, geospatial data, orWSGRNo. 67054-701.601metadata. In some embodiments, the observed data comprises continuous data, structured data, or unstructured data. In some embodiments, the prior sample comprises random noise. In some embodiments, the prior distribution comprises a Gaussian distribution, a uniform random distribution, or a Dirichlet distribution. In some embodiments, synthesizing the first data comprises sampling from a first marginal interpolating distribution parametrized by the observed data sample and the prior sample. In some embodiments, the first marginal interpolating distribution is determined using a denoising diffusion implicit model. In some embodiments, synthesizing the second data comprises sampling from a second marginal interpolating distribution parametrized by the observed data sample and the first data. In some embodiments, the second marginal interpolating distribution is determined using a denoising diffusion implicit model. In some embodiments, the first output comprises a first marginal preserving interpolant between the first data and the observed data. In some embodiments, the first marginal preserving interpolant preserves a marginal distribution of the first output for a plurality of time steps. In some embodiments, the second output comprises a second marginal preserving interpolant between the second data and the observed data. In some embodiments, the second marginal preserving interpolation preserves a marginal distribution of the second first output for a plurality of time steps. In some embodiments, the first time step is sampled from a first time distribution. In some embodiments, the target time step is sampled from a second time distribution, parametrized by the first time step. In some embodiments, the second time step is determined using a mapping function parametrized by the first time step and the target time step. In some embodiments, the computed loss is determined using maximum mean discrepancy. In some embodiments, the maximum mean discrepancy is determined using a kernel function. In some embodiments, the system further comprises applying a stop gradient operation on the second output when computing the loss. In some embodiments, the system further comprises prompting the trained generative Al model with at least a request for a type of data and a category of data. The system of claim 51, further comprises generating, using the trained generative Al model, new data based at least on the prompting. In some embodiments, the new data is determined to be statistically similar to the observed data. In some embodiments, the new data is determined to be statistically similar to the observed data with a Fréchet inception distance (FID) of less than about 2.5. In some embodiments, the system further comprises automatically rendering the new data on a graphical user interface (GUI), wherein the GUI comprises dynamically selectable objects or elements to modify the new data or append additional data to the new data in real time. In some embodiments, the trained generative Al model comprises a stable model trained in one stage. In some embodiments, the system further comprises repetitively synthesizing a third data or more data at successive time steps, based on data synthesized at a previous time step. In someWSGRNo. 67054-701.601embodiments, the first dataset comprises a first training dataset, wherein the second dataset comprises a second training dataset, and wherein each dataset is used to obtain the trained generative Al model.

[0009] In another aspect, disclosed herein is a computer program product, the computer program product comprising at least one non-transitory computer-readable medium having computer-readable program code portions embodied therein, the computer-readable program code portions comprising: an executable portion configured to obtain (i) a first dataset comprising an observed data sample from a data distribution of observed data and (ii) a second dataset comprising a prior sample from a prior distribution; an executable portion configured to synthesize first data at a first time step based on the observed data sample and the prior sample; an executable portion configured to synthesize second data at a second time step based on the observed data sample and the first data, wherein the second time step is between a target time step and the first time step; an executable portion configured to generate, using a neural network, a first output and a second output at the target time step, wherein (i) the first output is obtained using the first data with the target time step and the first time step as conditioning inputs and (ii) the second output is obtained using the second data with the target time step and the second time step as conditioning inputs; and an executable portion configured to obtain a trained generative AI model by updating parameters of the neural network, wherein the updating comprises at least computing a loss between a first distribution of the first output and a second distribution of the second output.

[0010] Additional aspects and advantages of the present disclosure will become readily apparent from the following detailed description, wherein only illustrative embodiments of the present disclosure are shown and described. As will be realized, the present disclosure is capable of other and different embodiments, and its several details are capable of modifications in various obvious respects, all without departing from the present disclosure. Accordingly, the drawings and description are to be regarded as illustrative in nature and not as restrictive.INCORPORATION BY REFERENCE

[0011] All publications, patents, and patent applications mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent, or patent application was specifically and individually indicated to be incorporated by reference. To the extent publications and patents or patent applications incorporated by reference contradict the present disclosure contained in the specification, the specification is intended to supersede and / or take precedence over any such contradictory material.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The novel features of the present disclosure are set forth with particularity in the appended claims. A better understanding of the features and advantages of the present disclosure will beWSGRNo. 67054-701.601obtained by reference to the following detailed description that sets forth illustrative embodiments, in which the principles of the present disclosure are utilized, and the accompanying drawings of which:

[0013] FIG. 1 shows examples of generated samples on ImageNet-256×256 using 8 steps, in some embodiments.

[0014] FIG. 2 shows an example method of using an interpolation from data to prior to define a one-step sampler that moves from any t to s < t, directly transforming ^t(xt) to qs(xs), in some embodiments. This can be repeated by jumping to an intermediate r < t before moving to s < r.

[0015] FIG. 3 shows an example model that, with self-consistent interpolants, uses M particle samples M= 2 is shown) for moment matching, in some embodiments. Samples from pf|t(xs) are obtained by drawing from pθ|t(x|xt) followed by qs\t(xs\x,xt). Solid and dashed lines indicate sampling with and without gradient propagation, respectively. After AT samples are drawn, sample Xi is repulsed by xsand attracted towards samples of xT and x~'sthrough kernel function &(-, ).

[0016] FIGs. 4A-4B show example data illustrating model training stability, in some embodiments. FIG. 4A shows FID convergence for different embeddings, in some embodiments.FIG. 4B shows CIFAR-10 samples from Fourier embedding (scale = 16), in some embodiments.

[0017] FIG. 5 shows example data illustrating that more particles indicate more stable training on ImageNet-256x256, in some embodiments.

[0018] FIG. 6 shows example data of ImageNet-256x256 FID with different sampler types, in some embodiments.

[0019] FIGs. 7A-7C show example data illustrating the performance of a model trained using moment matching self-distillation (MMSD), in some embodiments. FIG. 7A shows data illustrating that the model scales with training compute, in some embodiments. FIG. 7B shows data illustrating that the model scales with inferences compute, in some embodiments. FIG. 7C shows data illustrating that the model exhibits strong correlation between model size and performance, in some embodiments.

[0020] FIG. 8 shows example data illustrating that sample visual quality increases with increase in both model size and sampling compute, in some embodiments.

[0021] FIGs. 9A-9B show example data illustrating a performance of a model based on a value of a noise embedding, in some embodiments. FIG. 9A shows data illustrating log distance in embedding space for Cnoise(f) = ct, in some embodiments. FIG. 9B shows data illustrating similar ImageNet-256×256 convergence across different c, in some embodiments.

[0022] FIG. 10 shows example data of ImageNet-256×256 FID progression with different t,r gap with M = 4, in some embodiments.WSGRNo. 67054-701.601

[0023] FIGs. 11A-11B show example data of FID progression on different types of mapping function r(s,t), in some embodiments. FIG. 11A shows example data of FID progression on different types of mapping function r(s,t) for CIFAR-10, in some embodiments. FIG. 11B shows example data of FID progression on different types of mapping function r(s,t) for ImageNet-256x256, in some embodiments.

[0024] FIG. 12 shows examples of uncurated samples on CIFAR-10, unconditional, 2 steps, in some embodiments.

[0025] FIG. 13 shows examples of uncurated samples on ImageNet-256x256 using DiT-XL / 2 architecture, guidance w = 1.5, 8 steps, in some embodiments.

[0026] FIG. 14 shows examples of uncurated samples on ImageNet-256x256 using DiT-XL / 2 architecture, guidance R = 1.5, 8 steps, in some embodiments.

[0027] FIG. 15 depicts a non-limiting example of a computing device configured to perform methods herein, in some embodiments.

[0028] FIG. 16 depicts a non-limiting example of a web / mobile application provision system configured to perform methods herein, in some embodiments.

[0029] FIG. 17 depicts a non-limiting example of a cloud-based web / mobile application provision system configured to perform methods herein, in some embodiments.DETAILED DESCRIPTION

[0030] While various embodiments of the present disclosure have been shown and described herein, such embodiments are provided by way of example only. Numerous variations, changes, or substitutions may occur without departing from the present disclosure. It should be understood that various alternatives to the embodiments of the present disclosure described herein may be employed.1 Introduction

[0031] Generative models for domains, e.g., continuous domains, may enable applications in images (Rombach et al., 2022; Saharia et al., 2022; Esser et al., 2024; each of which is incorporated by reference herein in its entirety), videos (Ho et al., 2022a; Blattmann et al., 2023; OpenAI, 2024; each of which is incorporated by reference herein in its entirety), or audio (Chen et al., 2020; Kong et al., 2020; Liu et al., 2023; each of which is incorporated by reference herein in its entirety). Yet achieving high-fidelity outputs, efficient inference, and stable training remains an unmet technical challenge. Diffusion models (Sohl-Dickstein et al., 2015; Ho et al., 2020; Song et al., 2020b; each of which is incorporated by reference herein in its entirety) may require many inference steps for high-fidelity outputs. Further, step-reduction methods, such as diffusion distillation (Yin et al., 2024; Sauer et al., 2025; Zhou et al., 2024; Luo et al., 2024a; each of which is incorporated by reference herein in its entirety) and consistency models (Song et al., 2023; GengWSGRNo. 67054-701.601et al., 2024; Lu & Song, 2024; Kim et al., 2023; each of which is incorporated by reference herein in its entirety) may risk training collapse without careful tuning and regularization, e.g., pregenerating data-noise pair and early stopping.

[0032] To solve at least the aforementioned technical challenges, the present disclosure provides methods and systems for inductive moment matching (IMM). IMM can provide technical solutions for at least a stable, single-stage training procedure that learns generative Al models from scratch for single- or multi-step inference. IMM can operate on the time-dependent marginal distributions of stochastic interpolants (Albergo et al., 2023; which is incorporated by reference herein in its entirety), described as continuous-time stochastic processes that connect two arbitrary probability density functions (data at t = 0 and prior at t = 1). As illustrated in FIG.2, by learning a mapping, e.g., a stochastic or deterministic mapping from a first marginal at time t to a second marginal at time.s < / , IMM can provide technical solutions for one- or multi-step generation of trained generative Al models.

[0033] For example, IMM models can be trained efficiently from mathematical induction. For time 5 < r < / , two distributions can be formed at 5 by running a one-step IMM from samples at r and t. Their divergence can be minimized, thereby enforcing that the distributions at 5 are independent of the starting time-steps. This construction by induction can enable convergence to the data distribution. To improve training stability, IMM can be modeled based on certain stochastic interpolants, and the objective can be optimized with stable sample-based divergence estimators, e.g., moment matching (Gretton et al., 2012; which is incorporated by reference herein in its entirety). The present disclosure demonstrates that consistency models (CMs) are a singleparticle, first-moment matching special case of IMM, which can at least explain the training instability of CMs.

[0034] To summarize, models such as diffusion models and flow matching models are slow at sampling. Even if perfectly trained, they require many inefficient computing steps, e.g., tens to more than hundreds of steps, to produce high-fidelity samples. Compared to the present disclosure, methods which focus on distilling these slow samplers into fast one-step or few-step ones as a post-training step in a two-stage approach often require extensive tuning, e.g., balancing training of two or more networks or tuned training schedules.

[0035] In contrast, the present disclosure can provide a single-stage training approach without adversarial losses to directly achieve few-step sampling during inference yet achieve both high-fidelity outputs and sampling efficiency. Further, the present disclosure provides broader implications for generative modeling by enabling end-to-end training of models that support both few-step and many-step inference. By eliminating the need for an additional distillation step during deployment, methods and systems herein can streamline the training-to-inference pipeline,WSGRNo. 67054-701.601thereby at least reducing computing and network resources to train and deploy generative Al models. IMM can learn by using maximum mean discrepancy (MMD) as a stable divergence metric that matches two probability distributions using samples. Such methods can incorporate a learning strategy that allows the model to learn from its own samples inspired by mathematical induction. IMM is demonstrated herein to converge to the data distribution and is empirically more stable than other few-step approaches, e.g., consistency training, while achieving improved technical performance on ImageNet-256x256. As demonstrated, on ImageNet-256×256, IMM surpasses diffusion models and achieves 1.99 Fréchet inception distance (FID) with only 8 inference steps in one or few stages using standard transformer architectures. On (CIFAR)-10, IMM similarly achieves state-of-the-art of 1.98 FID with 2-step generation in one or few stages for a model trained from scratch.

[0036] In an aspect, the present disclosure provides methods for training a generative artificial intelligence (Al) model. In some embodiments, a method comprises obtaining (i) a first training set comprising an observed data sample from a data distribution of observed data (e.g., distribution at t = 0) and (ii) a second training set comprising a prior sample from a prior distribution (e.g., distribution at t = 1). In some embodiments, the method comprises synthesizing first data at a first time step (e.g., time f) based on the observed data sample and the prior sample. In some embodiments, the method comprises synthesizing second data at a second time step (e.g., time r) based on the observed data sample and the first data. In some embodiments, the second time step is between a target time step (e.g., time 5) and the first time step. In some embodiments, the method comprises generating, using a neural network, a first output and a second output at the target time step. In some embodiments, the first output is obtained using the first data with the target time step and the first time step as conditioning inputs and (ii) the second output is obtained using the second data with the target time step and the second time step as conditioning inputs. In some embodiments, the method comprises obtaining a trained generative Al model by updating parameters of the neural network. In some embodiments, the updating comprises at least computing a loss between a first distribution of the first output and a second distribution of the second output.

[0037] In some embodiments, a method comprises (a) training a generative artificial intelligence (Al) model in one stage. In some embodiments, the model is stable. In some embodiments, the method comprises generating, using the trained generative AI model, new data, wherein the new data comprises high fidelity data.

[0038] In some embodiments, a method comprises obtaining a trained generative artificial intelligence (Al) model according to any of the method disclosed herein. In some embodiments, the method comprises prompting the trained generative Al model with at least a request for a typeWSGRNo. 67054-701.601of data and a category of data. In some embodiments, the method comprises generating, using the trained generative Al model, new data based at least on the prompting.

[0039] In some embodiments, the method comprises generating the first training set and the second training set. In some embodiments, generating the first training set comprises obtaining the observed data. In some embodiments, the observed data is obtained from a memory, a database, a storage, a cloud, or any combination thereof. In some embodiments, generating the first training set comprises generating the data distribution, based on the observed data. In some embodiments, generating the first training set comprises sampling or selecting the observed data sample from the data distribution of observed data. In some embodiments, generating the second training set comprises generating the prior distribution. In some embodiments, generating the second training set comprises sampling the prior sample from the prior distribution.

[0040] In some embodiments, the observed data comprises data obtained from image data, video data, audio data, text data, sensor data, geospatial data, or metadata. In some embodiments, the observed data is obtained using a sensor. In some embodiments, the image data or the video data is obtained using a camera or a scanner. In some embodiments, the audio data is obtained using a microphone. In some embodiments, the text data is obtained from a user using a keyboard, a graphical user interface (GUI), a voice recognition software, an optical character recognition (OCR) software, or any combination thereof. In some embodiments, the geospatial data is obtained using a global positioning system (GPS) sensor, a satellite imaging sensor, a radar sensor, a light detection and ranging (LiDAR) sensor, or any combination thereof. In some embodiments, the metadata comprises temporal metadata, geospatial metadata, descriptive metadata, structural metadata, or any combination thereof.

[0041] In some embodiments, the observed data comprises continuous data, structured data, or unstructured data. In some embodiments, the continuous data comprises measured data. In some embodiments, the measured data may take on an infinite number of values within a specified range. In some embodiments, the structured data is organized in a data schema. In some embodiments, the data schema comprises a table, a chart, a graph, a matrix, or any combination thereof. In some embodiments, the unstructured data is not organized in the data schema. In some embodiments, the unstructured data comprises image data, audio data, text data, or any combination thereof.

[0042] In some embodiments, the first dataset comprises a first training dataset. In some embodiments, the second dataset comprises a second training dataset. In some embodiments, each dataset is used to obtain the trained generative Al model. In some embodiments, the first training dataset or the second training dataset may comprise data obtained from one or more sensors. In some embodiments, the first training dataset or the second training dataset may comprise labeledWSGRNo. 67054-701.601data. The labeled data may comprise a label corresponding to a type, a structure, a content, or a topic of the first training dataset or the second training dataset. In some embodiments, the trained generative Al model is configured to generate data that is similar to the first training dataset or the second training dataset.

[0043] In another aspect, the present disclosure provides systems configured to implement methods herein. In some embodiments, a system comprises at least one processor. In some embodiments, the system comprises instructions executable by the at least one processor to cause the at least one processor to perform operations. In some embodiments, the operations comprise obtaining (i) a first dataset comprising an observed data sample from a data distribution of observed data and (ii) a second dataset comprising a prior sample from a prior distribution. In some embodiments, the operations comprise synthesizing first data at a first time step based on the observed data sample and the prior sample. In some embodiments, the operations comprise synthesizing second data at a second time step based on the observed data sample and the first data. In some embodiments, the second time step is between a target time step and the first time step. In some embodiments, the operations comprise generating, using a neural network, a first output and a second output at the target time step. In some embodiments, (i) the first output is obtained using the first data with the target time step and the first time step as conditioning inputs and (ii) the second output is obtained using the second data with the target time step and the second time step as conditioning inputs. In some embodiments, the operations comprise obtaining a trained generative Al model by updating parameters of the neural network. In some embodiments, the updating comprises at least computing a loss between a first distribution of the first output and a second distribution of the second output.

[0044] In some embodiments, the present disclosure provides computer program products. In some embodiments, a computer program product comprising at least one non-transitory computer-readable medium having computer-readable program code portions embodied therein. In some embodiments, the computer-readable program code portions comprise an executable portion configured to obtain (i) a first dataset comprising an observed data sample from a data distribution of observed data and (ii) a second dataset comprising a prior sample from a prior distribution. In some embodiments, the computer-readable program code portions comprise an executable portion configured to synthesize first data at a first time step based on the observed data sample and the prior sample. In some embodiments, the computer-readable program code portions comprise an executable portion configured to synthesize second data at a second time step based on the observed data sample and the first data. In some embodiments, the second time step is between a target time step and the first time step. In some embodiments, the computer-readable program code portions comprise an executable portion configured to generate, using a neural network, aWSGRNo. 67054-701.601first output and a second output at the target time step. In some embodiments, (i) the first output is obtained using the first data with the target time step and the first time step as conditioning inputs and (ii) the second output is obtained using the second data with the target time step and the second time step as conditioning inputs. In some embodiments, the computer-readable program code portions comprise an executable portion configured to obtain a trained generative Al model by updating parameters of the neural network. In some embodiments, the updating comprises at least computing a loss between a first distribution of the first output and a second distribution of the second output2 Preliminaries2.1 Diffusion, Flow Matching, and Interpolants

[0045] Methods and systems herein can provide technical improvements over diffusion models, flow matching models, and stochastic interpolants. For a data distribution q(x), diffusion models, e.g., variance-preserving (VP) diffusion models (Ho et al., 2020; Song et al., 2020b; each of which is incorporated by reference herein in its entirety) and flow matching (FM) (Lipman et al., 2022; Liu et al., 2022; each of which is incorporated by reference herein in its entirety), may be used to construct time-augmented variables xt as an interpolation between data x ~ q(x) and prior e ~ N(0, I) such that xt = atx + otc where oco = <ri = l,«i = uo = 0. VP diffusion may choose at= (7T 7T-t \,ot= sin (-t) and FM chooses at= 1 - t,ot = t. Both v-prediction diffusion (Salimans & Ho, 2022; which is incorporated by reference herein in its entirety) and FM can be trained by matching the conditional velocity vt= aix + of such that a generative Al model, e.g., a neural network Gfxt,i) approximates Ex., [vt|xt]. Samples can then be generated via an ordinary differential equation (ODE), e.g., a probability-flow ordinary differential equation (PF-ODE) = G0(xt, t) starting from e ~ N(0, I).

[0046] Stochastic interpolants. Unifying diffusion models and FM, interpolants, e.g., stochastic interpolants (Albergo et al., 2023; Albergo & Vanden-Eijnden, 2022; each of which is incorporated by reference herein in its entirety) construct a conditional interpolation qt(xt\x,c) = Nffx ),yt2T) between any data x ~ q(x) and prior e ~ pf) and sets constraints h(x,c) = c, Io(,e) = x, and yi = yo = 0. Similar to FM, a sampler, e.g., a deterministic sampler can be learned by explicitly matching the conditional interpolant velocity vt= dtlt(x,c) + y'a. where z ~ N(0, I) such dxt-that Ge(xt,f) ~ Ex,e,z[vt|xt]. Sampling can be performed following the PF-ODE — = Ge(xt ) similarly starting from prior e ~ p(e).

[0047] In some embodiments, the prior sample comprises random noise. In some embodiments, the random noise is sampled from a random distribution. In some cases, the random noise isWSGRNo. 67054-701.601generated using a random number generator (RNG). In some embodiments, the prior distribution comprises a Gaussian distribution, a uniform random distribution, or a Dirichlet distribution.

[0048] When yt= 0 and 7t(x,e) = atx + m for at„ot defined in FM, the intermediate variable xt = atx + me becomes a deterministic interpolation and its interpolant velocity vt= aix + o>( can reduce to FM velocity. Thus, its training and inference can both reduce to that of FM. When e ~ N(0, I), stochastic interpolants can reduce to v-prediction diffusion.2.2 Maximum Mean Discrepancy

[0049] Methods and systems herein can train models based on computing loss using maximum mean discrepancy. In some embodiments, the computed loss is determined using maximum mean discrepancy. In some embodiments, the computed loss is determined based at least in part on a generative adversarial network (GAN) objective. In some embodiments, the maximum mean discrepancy is determined using a kernel function. In some embodiments, the kernel function comprises a Laplacian kernel. In some embodiments, the kernel function comprises a radial basis function (RBF) kernel.

[0050] Maximum mean discrepancy (MMD, Gretton et al. (2012); which is incorporated by reference herein in its entirety) between distribution / ?(x), (y) for x,y G Rnis an integral probability metric (Muller, 1997; which is incorporated by reference herein in its entirety). MMD can be defined on Reproducing Kernel Hilbert Space (RKHS) H with a positive definite kernel k • R^5x R° — R asMMD2(p(x), q(y)) = ||Ex[ / c(x,-)] - Ey[fc(y,-)] ||%(1)where the norm is in H. Choices such as the radial basis function (RBF) kernel may imply an inner product of infinite-dimensional feature maps consisting of all moments of p(x) and t / (y), e.g., Efx7] and E[y'] for integer j > 1 (Steinwart & Christmann, 2008; which is incorporated by reference herein in its entirety).3 Inductive Moment Matching

[0051] The present disclosure provides methods and systems for inductive moment matching (IMM), a method that can train a model of both high fidelity and sampling efficiency in a single stage. To do so, a time-augmented interpolation between data (distribution at t = 0) and prior (distribution at t = 1) can be assumed. An implicit one-step model, e.g., a one-step sampler, that transforms the distribution at time t to the distribution at time 5 for any.s < t may be learned (Section 3.1). The model can enable direct one-step sampling from t = 1 to 5 = 0, and few-step sampling via recursive application from a t to an r < t and then to an.s < r until 5 = 0; this allows the model to learn from its own samples via bootstrapping (Section 3.2).3.1 Model Construction via Interpolants

[0052] Methods and systems herein can provide for model construction via interpolants.WSGRNo. 67054-701.601

[0053] Given data x ~ q(x) and prior e ~ p(e the time-augmented interpolation xt, e.g., defined in Albergo et al. (2023) (which is incorporated by reference herein in its entirety), follows xt~ < / t(xt|x, r). This implies a marginal interpolating distribution= JT qt(xt|x,e)q(x)p(e)dxde. (2)

[0054] A model distribution, implicitly defined by a one-step sampler that transforms ^t(xt) into ^(xi) for some.s < / , can be learned. This can be done via a special class of interpolants, which preserves the marginal distribution ^(x0 while interpolating between x and xt. These may be termed marginal-preserving interpolants among a class of generalized interpolants. Formally, Xs may be defined as a generalized interpolant between x and xtif, for all 5 G [0, / ], its distribution follows qs\t(xs\x,xl) = N(Is\t(x,xt),yS2\tT) (3) and satisfies constraints It\t(x,xt) = xt, Io\t(x,xt) = x, yq= yo\t = 0, and < / t|(xt|x,6) = qt(xt\x,e). When t = 1, it can reduce to regular stochastic interpolants. Next, marginal-preserving interpolants can be defined.

[0055] Definition 1 (Marginal-preserving interpolants). A generalized interpolant Xs is marginal-preserving if for all t G [0,1] and for all.s 6 [0, / ], the following equality holds:Qs(xs) = ff qs^x^x.x^qt^x^qtCx^dXtdx, (4) where(X|X 5 =J- „(xt|x,e),Mp(e)dedxv 1 7QtOt)v 7

[0056] That is, this class of interpolants has the same marginal at 5 regardless of t. For all t E [0, 1], the noisy model distribution may be defined at.s G [0, / ] asPs\t(Xs) = qs^xAx.x^p^x^q^x^dXtdx, (6) where the interpolant is marginal preserving and p |t(x|xt)is the clean model distribution implicitly parameterized as a one-step sampler. This definition can also enable multistep sampling. To produce a clean sample x given xt~ qt(xi) in two steps via an intermediate 5: (1) x ~ p |t(x|xt) can be sampled followed by xs~ qs|t(xs|x, xt) and (2) if the marginal of xsmatches qs(xs), x can be obtained by x ~ pQ|t(x|xt). Therefore, it can be useful to minimize divergence between Eq. (4) and (6) using the objective below.

[0057] Naive objective. As one can easily draw samples from the model, it can be naively learned by directly minimizing£(0) = Es,t[D(qs(xs),pf|t(xs))] (7) with time distribution p(s,f) and a sample-based divergence metric £>(•, •) such as MMD or GAN (Goodfellow et al., 2020; which is incorporated by reference herein in its entirety). If an interpolant Xsis marginal-preserving, then the minimum loss is 0 (see Lemma 3). One might also notice the similarity between right-hand sides of Eq. (4) and (6).WSGRNo. 67054-701.601

[0058] However, qs(xs) = pf|t(%s) does not necessarily imply pf|t(x|xt) = qt(x|xt). In fact, the minimizer pf|t(x|xt) is not unique and, under mild assumptions, a minimizer, e.g., a deterministic minimizer exists (see Section 4).3.2 Learning via Inductive Bootstrapping

[0059] Methods and systems herein can improve model training by using inductive learning methods that involve bootstrapping.

[0060] In some embodiments, the first time step is sampled from a first time distribution. In some embodiments, the target time step is sampled from a second time distribution, parametrized by the first time step. In some embodiments, the second time step is determined using a mapping function parametrized by the first time step and the target time step.

[0061] While sound, the naive objective in Eq. (7) is difficult to optimize in practice because when t is far from.s; the input distribution ^t(xt) can be far from the target ^(x0- Fortunately, the interpolant construction implies that the model definition in Eq. (6) satisfies boundary condition qs(xs) = Ps\s (%s) regardless of 3 (see Lemma 4). This indicates that pf|t(xs) « qs(xs) when t is close to 5. Furthermore, the interpolant enforces pf|t(xs) ~ p^r(xs) for any r < t close to t as long as the model is continuous around t. Therefore, an inductive learning algorithm forpf|t(xs) can be constructed using samples from pf|r(xs).

[0062] For improved analysis, a sequence number n for parameter 0nand function r(s,f) can be defined, where 5 < r(s,f) < t such that pS|”(xs) learns to matchpS|"-1(xs). r’s arguments can be omitted when context is clear and r(s,f) may be a finite decrement from t but truncated at.s < t (see Example 2.3 for well-conditioned r(s,ff).

[0063] General objective. With marginal-preserving interpolants and mapping r(s,t), 0nm the following objective can be learned:L(0n) = ES / t[w(s,t)MMD2(ps^-1(xs),p^(xs))] (8)where w(s,f) is a weighting function. MMD may be chosen as the objective due to its superior optimization stability and this objective may be shown to learn the correct data distribution.

[0064] Theorem 1. Assuming r(s,f) is well-conditioned, the interpolant is marginal-preserving, and 0^ is a minimizer of Eq. (8) for each n with infinite data and network capacity, for all t G [0,1], 5 e [0,t],lim MMZ)2(qs(xs),p0"(xs)) = 0 (9) n->oo

[0065] In other words, 0neventually learns the target distribution ^(x0 by parameterizing a one-step sampler pS|"(x|xt).4 Simplified Formulation and PracticeWSGRNo. 67054-701.601

[0066] The present disclosure provides algorithmic and practical decisions in their utility.4.1 Algorithmic Considerations

[0067] Methods and systems herein can choose marginal-preserving interpolants to enable iterative model training. Despite theoretical soundness, it can be technically challenging to empirically choose a marginal-preserving interpolant. First, a sufficient condition for marginal preservation is presented.

[0068] In some embodiments, synthesizing the first data comprises sampling from a first marginal interpolating distribution parametrized by the observed data sample and the prior sample. In some embodiments, the first marginal interpolating distribution is determined using a denoising diffusion implicit model. In some embodiments, synthesizing the second data comprises sampling from a second marginal interpolating distribution parametrized by the observed data sample and the first data. In some embodiments, the second marginal interpolating distribution is determined using a denoising diffusion implicit model. In some embodiments, the first output comprises a first marginal preserving interpolant between the first data and the observed data. In some embodiments, the first marginal preserving interpolant preserves a marginal distribution of the first output for a plurality of time steps. In some embodiments, the second output comprises a second marginal preserving interpolant between the second data and the observed data. In some embodiments, the second marginal preserving interpolation preserves a marginal distribution of the second first output for a plurality of time steps.

[0069] Definition 2 (Self-consistent interpolants). Given s,t G[0, 1], s < / . an interpolant Xs ~ ^|t(x^|x,xt) is self-consistent if for all r G [x,t], the following holds:qs|t(xs|x,xt) = f qslr(xslx, xr)qrlt(xrlx, xt)dxr(10)

[0070] In other words, Xs has the same distribution if one (1) directly samples it by interpolating x and xt and (2) first samples any xr (given x and xt) and then samples Xs (given x and Xr). Furthermore, self-consistency implies marginal preservation (Lemma 5).

[0071] DDIM interpolant. Denoising diffusion implicit models (Song et al., 2020a; which is incorporated by reference herein in its entirety) can provide a fast ODE sampler for diffusion models, defined asDDIM(xt,x,s, t) = (<zs— y at^ x + ^-xt(11)and sample Xs = DDIM(xf, Ex[x|xf],s, ) can be drawn when Ex[x|xf] is approximated by a network. Example 3.1 shows thatDDIM as an interpolant, e.g., ys\t= 0 and Is\t(x,xt) = DDIM(x(,x,.s, / ), is self-consistent. Moreover, with deterministic interpolants such as DDIM, there exists a minimizer, e.g., a deterministic minimizer p |t(x|xt) of Eq. (7).WSGRNo. 67054-701.601

[0072] Proposition 1. (Informal) If ys\t= 0 and A|t(x,xt) satisfies mild assumptions, there exists a deterministic pf|t(x|xt) that attains 0 loss for Eq. (7).

[0073] See Example 2.6 for formal statement and proof. This allows pf|t(x|xt) = <5(% — ge(xt, s, t)) to be defined for a generative Al model, e.g., a neural network ge(xt,s,f) with parameter 0 by default.

[0074] Eliminating stochasticity. DDIM interpolant, deterministic model, and prior / ?(e) = N(Q,od2‘I) may be used, where oa is the data standard deviation (Lu & Song, 2024; which is incorporated by reference herein in its entirety). As a result, one can draw xArom model via x = fset(.xt) = DDIM(xt, g0(xt, s, t), s, t) where xt ~ qt(xi).

[0075] Re-using xtfor xr. Inspecting Eq. (8) and (6), xr~ t / / (x,) can be used to generate samples from the target distribution. Instead of sampling xrgiven a new (x,e) pair, variance can be reduced by reusing xt and x such that xr= DDIM(x(,x, / ', / ). This is justified because xrderived from xt preserves the marginal distribution qr(x / ) (see Example 3.2).

[0076] In some embodiments, the method further comprises applying a stop gradient operation on the second output when computing the loss.

[0077] Stop gradient, n can be set to optimization step number, e.g., advancing from n - 1 to n is a single optimizer step where / Lis initialized from 0n-i. Equivalently, n can be omitted from f)nand 0n- may be written as the stop-gradient parameter 0~.

[0078] Simplified objective. Let xt, t be i.i.d. random variables from qt(xt) and xt,x are variables obtained by reusing xt,xt respectively, the training objective can be derived from the MMD definition in Eq. (1) (see Example 3.3) ass,t (w(^ A) ) T k(ysr,ysr) — k(yys t,ys r^ — k(jys r,ySf)j](12) where ys-t= fs^x^.y^' = fset(xt'\ysx= fs,rM>ys',r = / s0r(x(),k(-,-)is a kernel function, and w(s,t) is a prior weighting function.

[0079] An empirical estimate of the above objective uses M particle samples to approximate each distribution indexed by t. In practice, a batch of model output with size B may be divided into B / M groups within which share the same (s,f) sample, and the objective is approximated by instantiating B / M number of M x M matrices. Note that the number of model passes does not change with respect to M (see Example 3.4). A M = 2 version is visualized in FIG. 3 and a simplified training algorithm is shown in Algorithm 1. A full training algorithm is shown in Example 4.4.2 Other Implementation ChoicesWSGRNo. 67054-701.601

[0080] Methods and systems herein can provide various functions and mathematical relations for use in training the generative Al model. Detailed analysis of each decision is described in Example 3.

[0081] Flow trajectories. Two flow trajectories (Nichol & Dhariwal, 2021; Lipman et al., 2022; each of which is incorporated by reference herein in its entirety) can be investigated for utility.1 1

[0082] Cosine. at= cos(-7Tt), ot= sin(-7Tt).

[0083] Optimal transport (OT)-FM. at= 1 - / . ot = t.

[0084] Network ge(xt,s,t). ge( t,s,f) = Csklp( / )x(+ Cout(Z)G:0(cin( xf, Cnoise(5), Cnoise( ) may be set with a generative Al model, e.g., a neural network Ge, following elucidating the design Space of diffusion-based generative models (EDM, Karras et al., 2022; which is incorporated by reference herein in its entirety). Let an(t) = I A + ot2) / od (Lu & Song, 2024; which is incorporated by reference herein in its entirety). Listed below are valid choices for other coefficients.

[0085] Identity. Cskip(t) = 0, Cout(f) = 1.

[0086] Simple-EDM (Lu & Song, 2024; which is incorporated by reference herein in its entirety). Cskip(t) = at / (a2+ (7a), Cout(t) = ~(7d(7t / paa + oa.

[0087] Euler-FM. cSkiP(t) = 1, cOut(t) = -tod. This can be specific to OT-FM schedule.

[0088] Example 3.5 shows that fs,te(xr) similarly follows the EDM parameterization of the form(X() C‘skip(. S, / )X A Cout(s, t)Gff(Cin(t)Xt, Cnoise(. S'), Cnoise(f)).

[0089] Noise conditioning cnoiseC). Cnoise(t) = ct can be chosen for some constant c > 1. The model convergence can be relatively insensitive to c, but larger c, e.g., 1000, can be useful (Song et al., 2020b; Peebles & Xie, 2023; each of which is incorporated by reference herein in its entirety), because it enables sufficient distinction between nearby r and t.

[0090] Mapping function r(s,t). r(s,f) via constant decrement in rjt = ot / at works well where the decrement is chosen in the form of (z max -z minj / Z^for some appropriate k (details in Example 3.7).

[0091] Kernel function. Kernel functions, e.g., time-dependent Laplace kernels of the form ks,t(x,y) = exp(-vi ~(.s, / )max(||. Y- -ylh,e) / D) for x,y G R / f some e > 0 may be used to avoid undefined gradients, and w~(s,f) = l / jcout(x,t)|. Laplace kernels can provide better gradient signals than RBF kernels, (see Example 3.8).

[0092] Weighting w(s,t) and distribution p s,t). Diffusion models, e.g., variational diffusion models (VDM) may be used (Kingma et al., 2021; Kingma & Gao, 2024; each of which is incorporated by reference herein in its entirety) and p(t) = U(e, T) and p(s\f) = U(e, may be defined for constants e, T [0, 1], Similarly, weighting can be defined as(13)WSGRNo. 67054-701.601where cr( ) is sigmoid function, At denotes log-signal-noise ratio (SNR)tand a G {1,2}, cr( ), b G R are constants (see Example 3.9).4.3 Sampling

[0093] Methods and systems herein can provide various sampling strategies for more efficient model training.

[0094] Pushforward sampling. A sample Xs can be obtained by directly pushing xt ~ ^t(xt) through fs,te(xt). This can be iterated for an arbitrary number of steps starting from e ~until 5 = 0. By definition, one application of fs,te(xt) can be equivalent to one DDIM step using the learned network ge(xt,s,f) as the x prediction. This sampler can then be viewed as a few-step sampler using DDIM where ge(xt, s,i) outputs a realistic sample x instead of its expectation Ex[x|xt], as in diffusion models.

[0095] Restart sampling. Stochasticity can be introduced during sampling (Xu et al., 2023; Song et al., 2023; each of which is incorporated by reference herein in its entirety) by re-noising a sample to a higher noise-level before sampling again. For example, a two-step restart sampler from xt requires.s G (0, / ) for drawing sample x = foiS(xs) where x, ~ ^(Xxl / o0t(%t))-

[0096] Classifier-free guidance. Given a data-label pair (x,c), during inference time, guidance, e.g., classifier-free guidance (Ho & Salimans, 2022; which is incorporated by reference herein in its entirety) with weight w can replace conditional model output Ge(xt,s,t,c) by a reweighted model output viawGe(xt,s,t,c) + (1 - w)G (xi,s, / ,0) (14) where 0 denotes the null-token indicating unconditional output. Similarly, the guided model can be defined as fs,t,we(xt) = cSkip(s,t)xt+cOut s,t)Gw0(xt,s,t,c), where Gwe(xt,s,t,c) is as defined in Eq. (14) and drop Cin( ) and cnoise( ) can be dropped for notational simplicity. This decision is justified in Example 5. Similar to diffusion models, c can be randomly dropped with probability p during training without special practices.

[0097] Algorithm 1. Training (see Example 4 for complete description):Input: parameter 0, DDIM(x(,x,.s, / ), B, M, pOutput: learned 0while model not converged doSample data x, label c, and prior e with batch sizeB and split into B / M groups. Each group shares a (s,r,f) sample.For each group, xt<— DDIM(r,x, / , 1).For each group, xr<— DDIM(x(,x, / ', / ).For each instance, set c = 0 with prob. p.Minimize the empirical loss L IMM(0) in Eq. (67).WSGRNo. 67054-701.601end while

[0098] Algorithm 2. Pushforward sampling (see Example 6 for complete description):Input: model fe,J\T(O, (optional) wOutput: xtSample xtN~ 7^(0, cr / )for z = N,..., 1 door / t'Lti.wfe)- end for

[0099] Pushforward sampling is presented in Algorithm 2, and both samplers are described in Example 6.5 Consistency of Methods Herein

[0100] Methods and systems herein can provide machine learning models that are consistent in formulation or implementation with additional generative Al models.

[0101] Detailed description and analysis are found in Example 7.

[0102] Consistency models. Consistency models (CMs) (Song et al., 2023; Song & Dhariwal, 2023; Lu & Song, 2024; which is incorporated by reference herein in its entirety) can use a network ge( t,f) that outputs clean data given noisy input xt. It can utilize point- wise consistency ge( t,f) = g0(xr,r) for any r < t where xris obtained via an ODE solver from xt using a pretrained model or ground truth data. CMs, e.g., discrete-time CM can satisfy g6»(xo, O) = xo and train via loss Ext,x,t[t / (g6»(xt, / ),g(9(xr,r))] where t / (-, ) can be chosen as L2 or learned perceptual image patch similarity (LPIPS) (Zhang et al., 2018; which is incorporated by reference herein in its entirety).

[0103] It may be shown in the following Lemma that CM objective with L2 distance is a singleparticle estimate of IMM objective with energy kernel.

[0104] Lemma 1. When xt= xt',xr= xr', k x,y) = — ||x — y||2, and s > 0 is a small constant, Eq. (12) reduces to CM loss Ex^x,t[w( IIZ70( t< 0—ge-(xr, r)||2] f°rsome valid mapping r(t) < t.

[0105] This single-particle estimate ignores the repulsion force imposed by k -, •). Energy kernel matches the first moment, ignoring all higher moments. These decisions can be significant contributors to training instability and performance degradation of CMs.

[0106] Improved CMs (Song & Dhariwal, 2023; which is incorporated by reference herein in its entirety) propose a loss, e.g., pseudohuber loss as t / (-, ) which may be justified in the Lemma below.WSGRNo. 67054-701.601

[0107] Lemma 2. Negative pseudo-huber loss kc(x,y) = c — √||x — y ||2+ c2for c > 0 is a conditionally positive definite kernel that matches all moments of x and y where weights on higher moments depend on c.

[0108] From a moment-matching perspective, the improved performance can be explained by the loss matching all moments of the distributions. In addition to pseudo-huber loss, many other kernels (e.g., Laplace, RBF, etcl) are all valid choices in the design space.

[0109] IMM loss can be extended to the differential limit by taking r(s,t) → t. The result can subsume the continuous-time CM (Lu & Song, 2024; which is incorporated by reference herein in its entirety) as a single-particle estimate (see Example 8).

[0110] Diffusion GAN and adversarial consistency distillation. GAN, e g., diffusion GAN (Xiao et al., 2021; which is incorporated by reference herein in its entirety) can parameterize the generative distribution as pf|t(xs|xt) = ∬ qs|t(xs|x, xt)δ(x — Gθ(xt,z, t))dzdx for s as a fixed decrement from t and p(z) a noise distribution. It defines the interpolant qs\t(xs\x,xt) as the DDPM posterior distribution, which is self-consistent (see Example 7.2) Diffusion GAN can introduce randomness to the sampling process to match qt(x\xt) instead of the marginal. Both Diffusion GAN and adversarial consistency distillation (Sauer et al., 2025; which is incorporated by reference herein in its entirety) can use GAN objective, which shares similarity to MMD in that MMD is defined as an integral probability metric where the discriminator can be chosen in RKHS. This can eliminate the need for explicit adversarial optimization of a neural -network discriminator.

[0111] Generative moment matching network (GMMN). GMMN (Li et al., 2015; which is incorporated by reference herein in its entirety) directly applies MMD to train a generator Ge(fi) where z ~ N(0,I) to match the data distribution. It is a special case of IMM in that when t = 1 and r(s,t) ≡ s = 0, the loss reduces to naive GMMN objective.6 Further Description of Technical Problems and Solutions

[0112] Methods and systems herein can provide technical improvements in training and generative performance, compared to other generative models.

[0113] Diffusion, flow matching, and stochastic interpolants. Diffusion models (Sohl-Dickstein et al., 2015; Song et al., 2020b; Ho et al., 2020; Kingma et al., 2021; each of which is incorporated by reference herein in its entirety) and flow matching (Lipman et al., 2022; Liu et al., 2022; each of which is incorporated by reference herein in its entirety) are generative frameworks that can learn a score or velocity field of a noising process from data into a simple prior. They can be scaled for text-to-image tasks (Rombach et al., 2022; Saharia et al., 2022; Podell et al., 2023; Chen et al., 2023; Esser et al., 2024; each of which is incorporated by reference herein in its entirety) or text-to-video tasks (Ho et al., 2022a; Blattmann et al., 2023; OpenAI, 2024; each of which is incorporated by reference herein in its entirety).WSGRNo. 67054-701.601

[0114] Interpolants, e.g., stochastic interpolants (Albergo et al., 2023; Albergo & Vanden-Eijnden, 2022; each of which is incorporated by reference herein in its entirety) can extend these ideas by defining a stochastic path between data and prior, then matching its velocity to facilitate distribution transfer. IMM can provide technical solutions that improve on top of the interpolant construction by directly learning one-step mappings between any intermediate marginal distributions.

[0115] Diffusion distillation. To resolve diffusion models’ sampling inefficiency, methods (Salimans & Ho, 2022; Meng et al., 2023; Yin et al., 2024; Zhou et al., 2024; Luo et al., 2024a; Heek et al., 2024; each of which is incorporated by reference herein in its entirety) may focus on distilling one-step or few-step models from pre-trained diffusion models. Some approaches (Yin et al., 2024; Zhou et al., 2024; each of which is incorporated by reference herein in its entirety) propose jointly optimizing two networks, but the training relies on careful tuning in practice and can lead to mode collapse (Yin et al., 2024; which is incorporated by reference herein in its entirety). Another method (Salimans et al., 2024; which is incorporated by reference herein in its entirety) matches the first moment of the data distribution available from pre-trained diffusion models. IMM can improve on the diffusion distillation methods by implicitly matching moments using MMD, and allow for training from scratch with a single model.

[0116] Few-step generative Al models from scratch. Early one-step generative Al models may rely on GANs (Goodfellow et al., 2020; Karras et al., 2020; Brock, 2018; each of which is incorporated by reference herein in its entirety) and MMD (Li et al., 2015; 2017; each of which is incorporated by reference herein in its entirety) (or their combination) but scaling adversarial training may remain challenging. Independent classes of few-step models, e.g. consistency models (CMs) (Song et al., 2023; Song & Dhariwal, 2023; Lu & Song, 2024; each of which is incorporated by reference herein in its entirety), consistency trajectory models (CTMs) (Kim et al., 2023; Heek et al., 2024; each of which is incorporated by reference herein in its entirety) and shortcut models (SMs) (Frans et al., 2024; which is incorporated by reference herein in its entirety) may face training instability and may require specialized components (Lu & Song, 2024; which is incorporated by reference herein in its entirety) (e.g., Jacobian-vector product (JVP) for flash attention) or other special practices. High weight decay may be used for SMs, combined LPIPS (Zhang et al., 2018; which is incorporated by reference herein in its entirety) or GAN losses may be used for CTMs, and special training schedules (Geng et al., 2024; which is incorporated by reference herein in its entirety) can be used to improve stability. IMM improves on the GANs and few-step models, as it can be trained stably with a single loss and can achieve strong performance without special training practices.7 Demonstration of Methods and Systems HereinWSGRNo. 67054-701.601

[0117] IMM’s empirical performance (Section 7.1), training stability (Section 7.2), sampling choices (Section 7.3), scaling behavior (Section 7.4), and practical decisions ablation (Section 7.5) are demonstrated herein.7.1 Image Generation

[0118] Methods and systems herein can provide trained generative Al models that are capable of generating high-fidelity data, e.g., images, video, audio, text, and the like, that are statistically similar to training datasets.

[0119] In some embodiments, the method comprises prompting the trained generative Al model with at least a request for a type of data and a category of data. In some embodiments, the type of data comprises a data modality (e.g., image, video, text, etc.). In some embodiments, the category of data comprises a content, a structure, or a topic of the data. In some embodiments, the type of data and the category of data matches a type and a category of the observed data. In some embodiments, the method further comprises generating, using the trained generative Al model, new data based at least on the prompting. In some embodiments, the new data comprises the type of data and the category of data.

[0120] In some embodiments, the new data is determined to be statistically similar to the observed data. The new data may be statistically similar to the observed data if a distribution of the new data is statistically similar to the data distribution of the observed data. In some embodiments, the statistical similarity between the new data and the observed data may be determined based on comparing one or more values of the new data with one or more values of the observed data. In some embodiments, the statistical similarity between the new data and the observed data may be determined based on comparing one or more parameters of the data distribution of the observed data and the distribution of the new data.

[0121] In some embodiments, the new data is determined to be statistically similar to the observed data with a Fréchet inception distance (FID) of less than about 2.5. The FID may measure a distance between the distribution of the new data and the observed data distribution. The FID may be determined based on determining a mean or a covariance of the distribution of the new data and the observed data distribution. The FID may be computed based at least in part on the mean or the covariance of the distribution of the new data and the observed data distribution. In some embodiments, the FID is at least or at most about 1, 1.5, 2, 2.5, 3, 3.5, 4, 4.5, 5, 5.5, 6, 6.5, 7, 7.5, 8, 8.5, 9, 9.5, 10, or any increment therein.

[0122] In some embodiments, the method further comprises automatically rendering the new data on a graphical user interface (GUI), wherein the GUI comprises dynamically selectable objects or elements to modify the new data or append additional data to the new data in real time. In some embodiments, the new data may be modified to update one or more features, to save the new data,WSGRNo. 67054-701.601or to delete the new data. In some embodiments, the additional data comprises additionally generated data or additionally obtained data, e.g., with one or more sensors.

[0123] FID (Heusel et al., 2017; which is incorporated by reference herein in its entirety) results may be presented for unconditional CIFAR-10 and class-conditional ImageNet-256×256 in Table 1 and 2. For CIFAR-10, baselines may be separated into diffusion and flow models, distillation models, and few-step models from scratch. IMM belongs to the last category in which it can achieve state-of-the-art performance of 1.98 using pushforward sampler. For ImageNet-256×256, transformer architectures, e.g., the popular Diffusion models with Transformers (DiT) (Peebles & Xie, 2023; which is incorporated by reference herein in its entirety) architecture may be used because of its scalability. IMM can be compared with GANs, masked and autoregressive models, diffusion and flow models, and few-step models trained from scratch.Table 1. CIFAR-10 results trained without label conditionsWSGRNo. 67054-701.601Family Method FID (4) Steps (|)DDPM (Ho et al, 2020) 3.17 1000 DDPM+ + (Song et al., 2020b) 3.16 1000 CS - - (Song et al., 2020b) 2.38 1000 Diffusion DP. M-Solver(Lu etal., 2022) 4.70 1.0 & How 1DDP (Nichol & Dhariwal, 2021) 2.90 4000 EDM (Kamas et al.. 2022) 2.05 35 Flow Matching (Lipman et al, 2022) 6.35 142 Rectified How (Li u et al., 20221 2.58 127 PD (Salimans & Ho, 2022)4.51 2 2~Rectifed Flow ( Salim am & Ho, 2022) 4.85 1 DFNO (Zheng et al., 2023) 3.78 1 KD (Lehman <& Ltihman, 2021 ) 9.3 1.TRACT (Berthelot et al, 2023) 3.32 2 „. DifLInstract (Luo et al, 2024a) 5.57 1 Few-Step PID (LP1PS) ( lee i.a aL. 2024} 3.92 1 via Distillation,.,,DMD (Yin et al, 2024) 3.77 1 CD (LPIPS) (Sune et al., 2023) 2.93 2 CTM (w / GAN) ( Kim st al, 2023) 1. 7 2 SiD(ZtouetaL 2(C ) L92 1 SIM (Lao st al.., 2024b) 2.06 1 sCD ( Song, 2024) 2.52 2 iCT (Song & Dhariwal, 2023) 2.83 12.46 2 ECT (Geng et al, 2024) 3.60 1 Few “Step 2.11 2 from Scratch sCT (Lu & Song, 2024) 2.97 12.06 2 MMSD (wis) 2.93 12.05 2Table 2. Class-conditional ImageNet-256×256 resultsWSGRNo. 67054-701.601Family Method FID(.) Steps (.) SPaiams BigGAN (Brock, 2018) 6.95 1 112M GAN GigaGAN (Kang et al, 2023) 3-45 1 569M StyJeGAN-XL ■ (Karras et at, 2020) 230 1 166M VQGAN (Esser e tai., 2021) 26.52 1024 227M. MaskGIT (Chang fit at. 2022) 6J3 8 227M 1 MAR (Li et al, 2024) 1.98 64 166M A Ak VAR-d2(MTm et al. 2024a) 2.57 10 600M VAR-d30tTmet aL.2O24a) 1.92 10 2B ADM (Dhariwal & Nichol, 2021) 10.94 250 554M CDM (Ho el al, 2022b) 438 8100 SimDiff (&x>geboometaL, 2023) 2.77 512 2B LDM-4-G (Rombach et al, 2022) 3.60 250 400M Diffusion U-DiT-L (Tm fit aL 2024b) 337 250 916M & Fkiw DiT-XL / 2 (w - 1.0) (Peebles & Xie, 2023) 9.62 250 675M DiT-XL / 2 (w = 1.25) (Peebles & Xis, 2023) 3.22 250 675M DiT-XL / 2 (w --M.5) ( Peebles & Xie, 2023) 2.27 250 675M SiT-XU2 (w - 1.0) (Ma et al, 2024) 9.35 250 675 M SiT-XU2 (w - 1.5)(Ma et al.,2024) 2.15 250 675 M ICT C&weuL, 2023) 34.24 1 675M 20.3 2 675M Shortcut (Frans etal, 2024) 10.60 1 675M 730 4 675 M 330 128 675MeMMSD (ours) (XL / 2, w - 1.26) 7.12 1 675 M ' 4. S7 2u6 4 WJM2.80 8 675M MMSD (ours) (XL / 2, w = 1.5) 7.97 1 675M 4.01 2 675M 2.61 4 675M 2.13 8 675M

[0124] Decreasing FID can be observed with more steps and IMM can achieve 1.99 FID with 8 steps (with w = 1.5), surpassing DiT and SiT (Ma et al., 2024; which is incorporated by reference herein in its entirety) using the same architecture except for injecting time 5 (see Example 9). Notably, the methods disclosed herein also achieve better 8-step FID than the 10-step visual autoregressive modeling (VAR) (Tian et al., 2024a; which is incorporated by reference herein in its entirety) of comparable size. At 16 steps, IMM also achieves 1.90 FID outperforming VAR’s 2B variant (see Example 9). However, different from VAR, IMM grants flexibility of variable number of inference steps and the large improvement in FID from 1 to 8 steps additionally demonstrates IMM’s efficient inference-time scaling capability. Lastly, the IMM model similarly surpass Shortcut models’ (Frans et al., 2024; which is incorporated by reference herein in its entirety) best performance with only 8 steps. Inference details are described in Section 7.3 and Example 9.2.WSGRNo. 67054-701.6017.2. IMM Training is Stable

[0125] Methods and systems herein can provide trained generative Al models that are stable in performance across various settings. In some embodiments, the trained generative Al model comprises a stable model trained in one stage.

[0126] IMM is stable and achieves improved performance across a range of parameterization choices.

[0127] Positional vs. Fourier embedding. A technical problem with CMs (Song et al., 2023; which is incorporated by reference herein in its entirety) is its training instability when using Fourier embedding with scale 16, which may force reliance on positional embeddings for stability.FIGs. 4A-4B show that IMM does not face this problem. For Fourier embedding, the standard NCSN++ may be used (Song et al., 2020b; which is incorporated by reference herein in its entirety), and architecture and embedding scale may be set to 16. For positional embeddings, DDPM++ can be adopted (Song et al., 2020b; which is incorporated by reference herein in its entirety). Both embedding types can converge reliably, and samples from the Fourier embedding model are included in FIGs. 4A-4B.

[0128] Particle number. Particle number M for estimating MMD may be an important parameter for empirical success (Gretton et al., 2012; Li et al., 2015; each of which is incorporated by reference herein in its entirety), where the estimate may be more accurate with larger M. In this case, increasing M can slow down convergence because a fixed batch size B is used, in which the samples are grouped into B / M groups of M, where each group shares the same t. The larger M may mean that fewer f s are sampled. On the other hand, using small numbers of particles, e.g., M= 2, may lead to training instability and performance degradation, especially on a large scale, e.g., with DiT architectures. As shown in FIG. 5, there may exist an optimal spot where a few particles effectively help with training stability while further increasing M slows down convergence. In ImageNet256><256, training may collapse when M= 1 ( e.g., as used in CM) and M= 2, and lowest FID may be achieved under the same computation budget with M = 4. M < 4 may not allow sufficient mixing between particles and larger M means fewer f s are sampled for each step, thus slowing convergence. It may be useful to use a large enough M for stability, but not too large for slowed convergence.

[0129] Noise embedding cnoise(·). FIG. 9A shows the log absolute mean difference of t and r(s,f) in the positional embedding space. Increasing c can increase distinguishability of nearby distributions. FIG. 9B shows that similar convergence on ImageNet-256×256 across different c can be observed, demonstrating the insensitivity of the framework with respect to noise function.7.3 SamplingWSGRNo. 67054-701.601

[0130] Methods and systems herein can provide efficient sampling methodologies for training a generative Al model.

[0131] In some embodiments, the method comprises repetitively synthesizing a third data or more data at successive time steps, based on data synthesized at a previous time step.. In some embodiments, the successive time steps are between the target time step and the first time step. In some embodiments, the successive time steps are between the target time step and the second time step. In some embodiments, the successive time steps follow a chronological order. For example, the second time step may be between the target time step and the first time step, the third time step may be between the target time step and the second time step, etc. In some embodiments, the successive time steps comprise at least or at most about 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 20, 25, 30, 35, 40, 45, or 50 time steps.

[0132] Different sampling settings may be investigated for best performance. One-step sampling can be performed by simple pushforward from T to e (concrete values in Example 9.2). On CIFAR-10, 2 steps are used and intermediate time ti is set such that η1 / η2= 1.4, a choice that works well empirically. On ImageNet-256×256, more than 2 steps are used and, for simplicity, (1) uniform decrement in t and (2) elucidating the design Space of diffusion-based generative models (EDM) (Karras et al., 2024; which is incorporated by reference herein in its entirety) schedule (detailed in Example 9.2) are investigated. FID of various sampler settings are shown in FIG. 6 with guidance weight w = 1.5. Pushforward samplers with uniform schedule can work the best on ImageNet256><256 and can be used as the default setting for multi-step generation. Additionally, pushforward combined with restart samplers can achieve superior results.Table 3. FID results with different flow schedules and network parameterization.id / cos id / FM sEDM / cos sEDM / FM eFM CIFAR-10 3.77 3.45 2.39 2.10 2.53 ImageNet- 46.44 47.32 27.33 28.67 27.01 256x2567.4 Scaling Behavior

[0133] Methods and systems herein can provide generative Al models that scale in performance with training compute. Similar to diffusion models, IMM can scale with training and inference compute, as well as model size on ImageNet256><256. FIGs. 7A-7C show FID vs. training and inference compute in GFLOPs, and strong correlation is found between compute used and performance. Samples are visualized in FIG. 8 with increasing model size, e.g., DiT-S, DiT-B, DiT-L, or DiT-XL, and increasing inference steps, e.g., 1, 2, 4, or 8 steps. The sample quality increases along both axes, as larger transformers with more inference steps can capture moreWSGRNo. 67054-701.601complex distributions. This may explain that more computing can sometimes yield different visual content from the same initial noise as shown in the visual results.7.5 Ablation Studies

[0134] Methods and systems herein can provide ablation studies that can provide information about the effects of various design choices on model performance. Ablation studies may be performed with probabilistic architectures, e.g., denoising diffusion probabilistic models (DDPM)++ architecture for CIFAR-10 and DiT-B for ImageNet-256×256. FID comparisons use 2-step samplers by default.

[0135] Flow schedules and parameterization. Various combinations of network parameterization and flow schedules can be investigated: Simple-EDM + cosine (sEDM / cos), Simple-EDM + OT-FM (sEDM / FM), Euler-FM + OT-FM (eFM), Identity + cosine (id / cos), Identity + OT-FM (id / FM). Identity parameterization may fall behind other types of parameterization, which may show similar performance across datasets (see Table 3). On smaller scale (e.g., on CIFAR-10), sEDM / FM may work the best, but on larger scale (e.g., on ImageNet-256×256), eFM may work the best. This may indicate that OT-FM schedule and Euler parameterization may be more scalable than other choices.

[0136] Mapping function r(s,t). The choices for ablation include (1) constant decrement in rjt, (2) constant decrement in t, (3) constant decrement in= log (α2 / σ'2), (4) constant increment in 1 / z t (see Example 3.6). For fair comparison, the decrement gap can be chosen so that the minimum t - r(s,t) is ~ 10-3and the same network parameterization can be used. FID progression in FIGs. 11A-11C shows that (1) consistently outperforms other choices. The mapping gap can be additionally ablated using M = 4 in (1). The constant decrement can be in the form of (ηmax−ηmin) / 2kfor an appropriately chosen k. FIG. 10 shows that the performance can be relatively stable across k ∈ {11,12,13} but can experience instability for k = 14. This suggests that, for a given particle number, there may exist a largest k for stable optimization.Table 4. Ablation of weight w(s,t) on ImageNet-256×256.FID-50kw(s,t) = l 40.19+ ELBO weight 96.43+ at33.44+ 1 / (αt2+ σt2) 27.43

[0137] Weighting function. In Table 4, the weighting factors are ablated in three groups: (1) the VDM ELBO factors |α(b − λt) (2) weighting at(i.e., when a = 1), and (3) weighting1 / (αt2+ σt2). It may be useful to use α jointly with weighting, e.g., evidence lower bound (ELBO)WSGRNo. 67054-701.601weighting, because it can convert v-pred network to a e-pred parameterization (see Example 3.9), consistent with diffusion ELBO-objective. Factor (σt2+ αt2) upweighting middle time-steps can further boost performance, a helpful practice that can also be useful for FM training (Esser et al., 2024; which is incorporated by reference herein in its entirety). Additional study of the exponent a is performed in Example 9 and it was found that a = 2 may emphasize optimizing the loss when t is small while a = 1 may distribute weights more equally to larger t. As a result, a = 2 may achieve higher quality multi-step generation than a = 1.8 Conclusion

[0138] The present disclosure provides methods and systems for inductive moment matching, a framework that learns a few-step generative Al model from scratch. It can train by leveraging interpolants, e.g., self-consistent interpolants to interpolate between data and prior and by matching all moments of its own distribution interpolated to be closer to that of data. The methods disclosed herein can provide convergence in distribution and can generalize other generative AI methods. The methods disclosed herein can achieve state-of-the-art performance across benchmarks while achieving orders of magnitude faster inference. The methods disclosed herein can provide a new perspective on training few-step models from scratch and inspire a new generation of generative Al models.

[0139] Thus, the present disclosure can advance research in diffusion models and generative Al, which can enable new creative possibilities and democratize content creation. Potential benefits include expanding artistic expression, assisting content creators, and generating synthetic data for research.References

[0140] Albergo, M. S. and Vanden-Eijnden, E. Building normalizing flows with stochastic interpolants. arXiv preprint arXiv:2209.15571, 2022; which is incorporated by reference herein in its entirety.

[0141] Albergo, M. S., Boffi, N. M., and Vanden-Eijnden, E. Stochastic interpolants: A unifying framework for flows and diffusions. arXiv preprint arXiv:2303.08797, 2023; which is incorporated by reference herein in its entirety.

[0142] Auffray, Y. and Barbillon, P. Conditionally positive definite kernels: theoretical contribution, application to interpolation and approximation. PhD thesis, INRIA, 2009; which is incorporated by reference herein in its entirety.

[0143] Bao, F., Nie, S., Xue, K., Cao, Y., Li, C., Su, H., and Zhu, J. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pp. 22669-22679, 2023; which is incorporated by reference herein in its entirety.WSGRNo. 67054-701.601

[0144] Berthelot, D., Autef, A., Lin, J., Yap, D. A., Zhai, S., Hu, S., Zheng, D., Talbott, W., and Gu, E. Tract: Denoising diffusion models with transitive closure time-distillation. arXiv preprint arXiv: 2303.04248, 2023; which is incorporated by reference herein in its entirety.

[0145] Blattmann, A., Dockhom, T., Kulal, S., Mendelevitch, D., Kilian, M., Lorenz, D., Levi, Y., English, Z., Voleti, V., Letts, A., et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023; which is incorporated by reference herein in its entirety.

[0146] Brock, A. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1809.11096, 2018; which is incorporated by reference herein in its entirety.

[0147] Chang, H., Zhang, H., Jiang, L., Liu, C., and Freeman, W. T. Maskgit: Masked generative image transformer. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 11315-11325, 2022; which is incorporated by reference herein in its entirety.

[0148] Chen, J., Yu, J., Ge, C., Yao, L., Xie, E., Wu, Y., Wang, Z., Kwok, J., Luo, P., Lu, H., et al. Pixart-a: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv: 2310.00426, 2023; which is incorporated by reference herein in its entirety.

[0149] Chen, N., Zhang, Y., Zen, H., Weiss, R. J., Norouzi, M., and Chan, W. Wavegrad: Estimating gradients for waveform generation. arXiv preprint arXiv:2009.00713, 2020; which is incorporated by reference herein in its entirety.

[0150] Dhariwal, P. and Nichol, A. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021; which is incorporated by reference herein in its entirety.

[0151] Esser, P., Rombach, R., and Ommer, B. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pp. 12873-12883, 2021; which is incorporated by reference herein in its entirety.

[0152] Esser, P., Kulal, S., Blattmann, A., Entezari, R., Muller, J., Saini, " H., Levi, Y., Lorenz, D., Sauer, A., Boesel, F., et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024; which is incorporated by reference herein in its entirety.

[0153] Frans, K., Hafner, D., Levine, S., and Abbeel, P. One step diffusion via shortcut models. arXiv preprint arXiv:2410.12557, 2024; which is incorporated by reference herein in its entirety.

[0154] Geng, Z., Pokle, A., Luo, W., Lin, J., and Kolter, J. Z. Consistency models made easy. arXiv preprint arXiv:2406.14548, 2024; which is incorporated by reference herein in its entirety.

[0155] Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., WardeFarley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial networks. Communications of the ACM, 63(11): 139-144, 2020; which is incorporated by reference herein in its entirety.WSGRNo. 67054-701.601

[0156] Gretton, A., Borgwardt, K. M., Rasch, M. J., Scholkopf, B., and Smola, A. A kernel two- sample test. The Journal of Machine Learning Research, 13(1):723–773, 2012; which is incorporated by reference herein in its entirety.

[0157] Heek, J., Hoogeboom, E., and Salimans, T. Multistep consistency models. arXiv preprint arXiv:2403.06807, 2024; which is incorporated by reference herein in its entirety.

[0158] Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017; which is incorporated by reference herein in its entirety.

[0159] Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022; which is incorporated by reference herein in its entirety.

[0160] Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020; which is incorporated by reference herein in its entirety.

[0161] Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, D. P., Poole, B., Norouzi, M., Fleet, D. J., et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022a; which is incorporated by reference herein in its entirety.

[0162] Ho, J., Saharia, C., Chan, W., Fleet, D. J., Norouzi, M., and Salimans, T. Cascaded diffusion models for high fidelity image generation. Journal of Machine Learning Research, 23(47): 1-33, 2022b; which is incorporated by reference herein in its entirety.

[0163] Hoogeboom, E., Heek, J., and Salimans, T. simple diffusion: Endto-end diffusion for high resolution images. In International Conference on Machine Learning, pp. 13213-13232. PMLR, 2023; which is incorporated by reference herein in its entirety.

[0164] Kang, M., Zhu, J.-Y., Zhang, R., Park, J., Shechtman, E., Paris, S., and Park, T. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 10124-10134, 2023; which is incorporated by reference herein in its entirety.

[0165] Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., and Aila, T. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pp. 8110-8119, 2020; which is incorporated by reference herein in its entirety.

[0166] Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusionbased generative models. Advances in neural information processing systems, 35:26565-26577, 2022; which is incorporated by reference herein in its entirety.WSGRNo. 67054-701.601

[0167] Karras, T., Aittala, M., Lehtinen, J., Hellsten, J., Aila, T., and Laine, S. Analyzing and improving the training dynamics of diffusion models. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 24174-24184, 2024; which is incorporated by reference herein in its entirety.

[0168] Kim, D., Lai, C.-H., Liao, W.-H., Murata, N., Takida, Y., Uesaka, T., He, Y., Mitsufuji, Y., and Ermon, S. Consistency trajectory models: Learning probability flow ode trajectory of diffusion. arXiv preprint arXiv:2310.02279, 2023; which is incorporated by reference herein in its entirety.

[0169] Kingma, D. and Gao, R. Understanding diffusion objectives as the elbo with simple data augmentation. Advances in Neural Information Processing Systems, 36, 2024; which is incorporated by reference herein in its entirety.

[0170] Kingma, D., Salimans, T., Poole, B., and Ho, J. Variational diffusion models. Advances in neural information processing systems, 34:21696-21707, 2021; which is incorporated by reference herein in its entirety.

[0171] Kong, Z., Ping, W., Huang, J., Zhao, K., and Catanzaro, B. Diffwave: A versatile diffusion model for audio synthesis. arXiv preprint arXiv: 2009.09761, 2020; which is incorporated by reference herein in its entirety.

[0172] Li, C.-L., Chang, W.-C., Cheng, Y., Yang, Y., and Poczos, B.´ Mmd gan: Towards deeper understanding of moment matching network. Advances in neural information processing systems, 30, 2017; which is incorporated by reference herein in its entirety.

[0173] Li, T., Tian, Y., Li, H., Deng, M., and He, K. Autoregressive image generation without vector quantization. arXiv preprint arXiv:2406.11838, 2024; which is incorporated by reference herein in its entirety.

[0174] Li, Y., Swersky, K., and Zemel, R. Generative moment matching networks. In International conference on machine learning, pp. 1718-1727. PMLR, 2015; which is incorporated by reference herein in its entirety.

[0175] Lipman, Y., Chen, R. T., Ben-Hamu, H., Nickel, M., and Le, M. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022; which is incorporated by reference herein in its entirety.

[0176] Liu, H., Chen, Z., Yuan, Y., Mei, X., Liu, X., Mandic, D., Wang, W., and Plumbley, M. D. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv: 2301.12503, 2023; which is incorporated by reference herein in its entirety.

[0177] Liu, X., Gong, C., and Liu, Q. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv: 2209.03003, 2022; which is incorporated by reference herein in its entirety.WSGRNo. 67054-701.601

[0178] Lu, C. and Song, Y. Simplifying, stabilizing and scaling continuous-time consistency models. arXiv preprint arXiv:2410.11081, 2024; which is incorporated by reference herein in its entirety.

[0179] Lu, C., Zhou, Y., Bao, F., Chen, J., Li, C., and Zhu, J. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022; which is incorporated by reference herein in its entirety.

[0180] Luhman, E. and Luhman, T. Knowledge distillation in iterative generative models for improved sampling speed. arXiv preprint arXiv: 2101.02388, 2021; which is incorporated by reference herein in its entirety.

[0181] Luo, W., Hu, T., Zhang, S., Sun, J., Li, Z., and Zhang, Z. Diffinstruct: A universal approach for transferring knowledge from pre-trained diffusion models. Advances in Neural Information Processing Systems, 36, 2024a; which is incorporated by reference herein in its entirety.

[0182] Luo, W., Huang, Z., Geng, Z., Kolter, J. Z., and Qi, G.-j. One-step diffusion distillation through score implicit matching. arXiv preprint arXiv:2410.16794, 2024b; which is incorporated by reference herein in its entirety.

[0183] Ma, N., Goldstein, M., Albergo, M. S., Boffi, N. M., VandenEijnden, E., and Xie, S. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. arXiv preprint arXiv: 2401.08740, 2024; which is incorporated by reference herein in its entirety.

[0184] Meng, C., Rombach, R., Gao, R., Kingma, D., Ermon, S., Ho, J., and Salimans, T. On distillation of guided diffusion models. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 14297-14306, 2023; which is incorporated by reference herein in its entirety.

[0185] Muller, A. Integral probability metrics and their generating classes " of functions. Advances in applied probability, 29(2): 429-443, 1997; which is incorporated by reference herein in its entirety.

[0186] Nichol, A. Q. and Dhariwal, P. Improved denoising diffusion probabilistic models. In International conference on machine learning, pp. 8162-8171. PMLR, 2021; which is incorporated by reference herein in its entirety.

[0187] OpenAI. Video generation models as world simulators, https: / / openai.com / sora / , 2024; which is incorporated by reference herein in its entirety.

[0188] Peebles, W. and Xie, S. Scalable diffusion models with transformers. In Proceedings of the IEEE / CVF International Conference on Computer Vision, pp. 4195-4205, 2023; which is incorporated by reference herein in its entirety.WSGRNo. 67054-701.601

[0189] Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhom, T., Muller, J., Penna, J., and Rombach, R. Sdxl: Improving latent " diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023; which is incorporated by reference herein in its entirety.

[0190] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, pp. 10684-10695, 2022; which is incorporated by reference herein in its entirety.

[0191] Saharia, C., Chan, W., Saxena, S., Li, L., Whang, J., Denton, E. L., Ghasemipour, K., Gontijo Lopes, R., Karagol Ayan, B., Salimans, T., et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36494, 2022; which is incorporated by reference herein in its entirety.

[0192] Salimans, T. and Ho, J. Progressive distillation for fast sampling of diffusion models. arXiv preprint arXiv:2202.00512, 2022; which is incorporated by reference herein in its entirety.

[0193] Salimans, T., Mensink, T., Heek, J., and Hoogeboom, E. Multistep distillation of diffusion models via moment matching. arXiv preprint arXiv:2406.04103, 2024; which is incorporated by reference herein in its entirety.

[0194] Sauer, A., Lorenz, D., Blattmann, A., and Rombach, R. Adversarial diffusion distillation. In European Conference on Computer Vision, pp. 87-103. Springer, 2025; which is incorporated by reference herein in its entirety.

[0195] Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256-2265. PMLR, 2015; which is incorporated by reference herein in its entirety.

[0196] Song, J., Meng, C., and Ermon, S. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a; which is incorporated by reference herein in its entirety.

[0197] Song, Y. and Dhariwal, P. Improved techniques for training consistency models. arXiv preprint arXiv:2310.14189, 2023; which is incorporated by reference herein in its entirety.

[0198] Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Scorebased generative modeling through stochastic differential equations. arXiv preprint arXiv: 2011.13456, 2020b; which is incorporated by reference herein in its entirety.

[0199] Song, Y., Dhariwal, P., Chen, M., and Sutskever, I. Consistency models. arXiv preprint arXiv: 2303.01469, 2023; which is incorporated by reference herein in its entirety.

[0200] Steinwart, I. and Christmann, A. Support vector machines. Springer Science & Business Media, 2008; which is incorporated by reference herein in its entirety.WSGRNo. 67054-701.601

[0201] Tee, J. T. J., Zhang, K., Yoon, H. S., Gowda, D. N., Kim, C., and Yoo, C. D. Physics informed distillation for diffusion models. arXiv preprint arXiv:2411.08378, 2024; which is incorporated by reference herein in its entirety.

[0202] Tian, K., Jiang, Y., Yuan, Z., Peng, B., and Wang, L. Visual autoregressive modeling: Scalable image generation via nextscale prediction. arXiv preprint arXiv: 2404.02905, 2024a, which is incorporated by reference herein in its entirety.

[0203] Tian, Y., Tu, Z., Chen, H., Hu, J., Xu, C., and Wang, Y. U-dits: Downsample tokens in u- shaped diffusion transformers. arXiv preprint arXiv: 2405.02730, 2024b; which is incorporated by reference herein in its entirety.

[0204] Xiao, Z., Kreis, K., and Vahdat, A. Tackling the generative learning trilemma with denoising diffusion gans. arXiv preprint arXiv:2112.07804, 2027; which is incorporated by reference herein in its entirety.

[0205] Xu, Y., Deng, M., Cheng, X., Tian, Y., Liu, Z., and Jaakkola, T. Restart sampling for improving generative processes. Advances in Neural Information Processing Systems, 36:76806- 76838, 2023; which is incorporated by reference herein in its entirety.

[0206] Yin, T., Gharbi, M., Zhang, R., Shechtman, E., Durand, F., Freeman, W. T., and Park, T. One-step diffusion with distribution matching distillation. In Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 6613- 6623, 2024; which is incorporated by reference herein in its entirety.

[0207] Zhang, R., Isola, P., Efros, A. A., Shechtman, E., and Wang, O. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018; which is incorporated by reference herein in its entirety.

[0208] Zheng, H., Nie, W., Vahdat, A., Azizzadenesheli, K., and Anandkumar, A. Fast sampling of diffusion models via operator learning. In International conference on machine learning, pp.42390- 42402. PMLR, 2023; which is incorporated by reference herein in its entirety.

[0209] Zhou, M., Zheng, H., Wang, Z., Yin, M., and Huang, H. Score identity distillation: Exponentially fast distillation of pretrained diffusion models for one-step generation. In Forty-first International Conference on Machine Learning, 2024; which is incorporated by reference herein in its entirety.Examples

[0210] While various examples of the present disclosure have been shown and described herein, such examples are provided by way of example only. Numerous variations, changes, orWSGRNo. 67054-701.601substitutions may occur without departing from the present disclosure. It should be understood that various alternatives to the examples described herein may be employed.Example 1 — Background: Properties of Stochastic Interpolants

[0211] Some relevant properties of stochastic interpolants are described for exposition.

[0212] Boundary satisfaction. For an interpolant distribution qt (xt|x, e) defined in Albergo et al. (2023, which is incorporated by reference herein in its entirety), and the marginal qt (xt) as defined in Eq. (2), < / i(xi) = / ?(xi) and t / o(xo) = < / (xo) can be checked so that xi = c and xo = x.(Xi ]x, £)< / (x)p(€)dx<te (15) ~ • €)g(x)p(t:)dxde (16) J I g(x)p(x:i)dx (17) p(xi) (18) t7o(x(j|xte)<j(x)p(£)<ix<l€ (19) 3’(x(; — x)< / (x)p(e)dxd« (20) 11 (Kxc)Xe)(ix(21)<?(xo) (22)

[0213] Joint distribution. The joint distribution of x and xt is written as&(x. xt) = j €)r / (x)p(e)d€ (23)

[0214] Independence of joint at t = 1< (x, X; ) y ^(Xilx, 6)< (x)p(£)de (24) ™ I <5(X3- €)g(x)p(€)de (25) = g(x)p(xj ) (26) ~ c (x)p(e) (27) in which case xi = e.Example 2 — Theorems and Derivations

[0215] 2.1 Divergence Minimizer

[0216] Lemma 3. Assuming marginal-preserving interpolant and metric D(*, •), a minimizer 0*. (x^Xf ) of Eq. (7) exists, e.g.,z= qt (x\xt), and the minimum is 0.

[0217] Proof. qt(x\xt) is directly substituted into the objective to check. First,~ yy4(x|xt)< / t(xjdxtdx (28)::::y I qs|t(xs|x.xf)gt(x]xj^(xt)dxtdx (29)(]s( X. ) (30)WSGRNo. 67054-701.601where (a) is due to definition of marginal preservation. So the objective becomes [w($,t)D(qs(x.ls),qs(x.ls))] - 0 (31)

[0218] In general, the minimizer qt (x|xt) exists. However, this does not show that the minimizer is unique. In fact, the minimizer is not unique in general because a deterministic minimizer can also exist under certain assumptions on the interpolant (see Example 2.6).

[0219] Failure Case without Marginal Preservation. The marginal-preservation property of the interpolant qs\t (xs|x, xt) is important for the naive objective in Eq. (7) to attain 0 loss (Lemma 3). Consider the failure case below where the constructed interpolant is a generalized interpolant but not necessarily marginal-preserving. Then it is shown that there exists a t such that's'can never reach qs(xs) regardless of 6.

[0220] Proposition 2 (Example Failure Case). Let~1’, and suppose an interpolant Is|t (x, xt) =and%i* rV1 -7*1* then.> for ap o < s < t < 1 regardless of the learned distributionx^!given any metric D(», •).

[0221] Proof. This example first implies the learning targetgs(xs) Ij gs(xJx- e)^(x) / >(e)dxde (32) ™ yyCKXs—{(1— 5)x+ ))ft(x)d'(e — l)dxde (33)- <5(xs— <s) (34) is a delta distribution. However, it is shown that if any t < 1 is selected and ' ' ' can never be a delta distribution.?)^(XS)::: jj (x Jx. (x |xt)qt(xt}dx(dx (35)I A'((l — |)x + |xt, yy(l - ^•) / )p*it(x|xt)d(xt- t)dx{dx (36) = y Af((l - |)x + s, |^(1 - )z)p“!t(x|x(- t)dx (37)

[0222] Now, it is shown the model distribution has non-zero variance under these choices of t and ■f (x 15. Expectations are over or conditional interpolant qs\t (xs|x, xt) for all equations below.WSGRNo. 67054-701.601(38) (xixt)qi(x()dxdxj — E [xa(39) |x, x{] ( X JXf )< / t(xt)dxdxt(40) x Pa|t(x! xtj. i x. (dxdxj (41) p‘'l. (x |xt )qt. ( x()dxdxi(42) |xxt] p*|t(x{xt)<ft(xt)dxdx(+ pffstt(x |x, )f2>f(x, )dxdxt(43)where (a) can be simplified asVar(xs|x, xt) + f E [xsjx, xd2— E ^xai) > 0 (44)

[0223] because Var(x)x, xt) > 0 for all 0 < 5 < t < 1 due to its non-zero Gaussian noise. Therefore, Var(x ) > 0, implying * do-can never be a delta function regardless of model!-. A valid metric D(*, •) over probability1•sand qs(xs) impliesD(g#(xs),^jt(xs)) ™ () fA((xs) ~ <)s(xs)which meansp*!t(xg) ^(x4) D(<js(xs): / 4|dx^)) >0□

[0224] 2.2 Boundary Satisfaction of Model Distribution13^ (

[0225] The operator outputdo satisfies boundary condition.

[0226] Lemma 4 (Boundary Condition). For all 5 G [0,l] and all 6, the following boundary condition holds.gs(x.,) - i^.?(xa) (45)

[0227] Proof.P«Us(x«) = I y «»^(x<lx,xs)p^(x|x,)^a(x4)dxadx■■■■■■ ( / a(xs) y pa;a(xixa)dx::::'l«(x.s)□

[0228] 2.3 Definition of Well-Conditioned r(s, )

[0229] For simplicity, the mapping function r(s, i) is well-conditioned ifr(.s, () — inax(«, t — A(i)) (46)WSGRNo. 67054-701.601where A( ) > e > 0 is a positive function such that r(s, f) is increasing for t > s + co(5) where co(s) is the largest t that is mapped to 5. Formally, co(s) = sup { / : / '(.s, ) = }. For / > + co(s), the inverse w.r.t. t exists, i.e. r s, •) and r'1(5, / '(.s, / )) = t. All practical implementations follow this general form, and are detailed in Example 3.6.

[0230] 2.4 Main Theorem

[0231] Theorem 1. Assuming r(s, t) is well-conditioned, the interpolant is marginal -preserving, and is a minimizer of Eq. (8) for each n with infinite data and network capacity, for all t [0, 1], s [0, t],lim - 0. (9)

[0232] Proof. It is proven by induction on sequence number n. First, / '(.s, f) is well-conditioned by following the definition in Eq. (46). Furthermore, for notational convenience, let rn1(5, •):= r1(5, r1(5, r’1(s,...))) be n nested application of r’1(5, •) on the second argument. Additionally, / 'o'1(5, = t.

[0233] Base case: n =1. Given any > 0, r(s, u) = s for all 5 < u < co(s), implying MMD2(pJs(x;,)!pJe(x<.)) MMD^xJ.pJjx.,)) - 0 (47) for u < co(s) where (a) is implied by Lemma 4 and ( >) is implied by Lemma 3

[0234] Inductive assumption: n - 1. Assume~ for alls s“ - ft*

[0235] The target distributionjnEq. (8) is inspected if optimized on,s-f;oU0 On this interval, r(s, •) can be applied to the inequality to get =»,•(«.#) <;■(*.«) <: - rK.‘.3(s-A->i.<>)since r(5)•) is increasing. And by inductive assumption= ‘ / dM for ss?•»,«} s >-„LSA {frig implies minimizing”> "7; 7 ( *> 141 ' 5:| U%' ' jon* < u < IA MA isequivalent to minimizing(48)for » < « A AOJIA) Lemma 3 implies that its minimum achieves A[u(x»> - 'XA

[0236] Lastly, takingn~00impliesrf A:s■■■1and thus the induction covers the entire [s, 1] interval given each s. Therefore,~ por an 0 < 5 < / < 1.

[0237] 2.5 Self-Consistency Implies Marginal Preservation

[0238] Without assuming marginal preservation, it is important to define the marginal distribution of X5 under generalized interpolants qs\t (xs|x, xt) asqsfxe) ~ 11 <;i;;(x.4x,xt)( / t(x|xi)%(xt)(ixedx (49)WSGRNo. 67054-701.601and it is shown that with self-consistent interpolants, this distribution is invariant of t, i.e. qs\t (xs) = qs( s).

[0239] Lemma 5. If the interpolant qs\t (xs\x, xi) is self-consistent, the marginal distribution qs\t (xs) as defined in Eq. (49) satisfies qs(xs) = qs\t (xs) for all t[s, 1],

[0240] Proof. For t G [5, 1],q„:f(xsj x, xt) ) qt. ( x j xt) qt(xt)dxtdx (50) **' ■, **■ „,v vv, f < MxdX: <i<i(XM<0 d,ed 1x^dx (51)J Qt (x<! )<j(x) I 5t(xt|x, e)p(e)de<lxtdx (52)?K;t(x»lx' e)dxfI dedx (53)(xjx,c)xt j dedx (54) («):)dedx (55) x.. x. e)g(x)p(e)dedx (56)(57) where (ci) uses definition of self-consistent interpolants and (b) uses definition of the generalized interpolant.

[0241] It is shown in Example 3.1 that DDIM is an example self-consistent interpolant. Furthermore, DDPM posterior (Ho et al., 2020; Kingma et al., 2021; each of which is incorporated by reference herein in its entirety) is also self-consistent (see Lemma 6).

[0242] 2.6 Existence of Deterministic Minimizer

[0243] The formal statement for the deterministic minimizer is presented.

[0244] Proposition 3. if for all t e[0, 1], * e [o.f, 7.,,, = 0. jsinvertible w.r.t. x, and there exists Gi < oc such that (x-e) I! < 1!x > ell, then there exists a function: R'?such that «((x#) ■■■■■■ yy <7.sit(xjx,xf)5(x - ft5|t(x4))t / t(x()dx<lxf. (58)

[0245] Proof.1x51 v’x') be the inverse of \t w.r.t. x such thatxanj F|t'L|t(y>x0,x() =yoraj| x,xt,y e Since there exists Ci < such that \\It\i (x, c) || < Ci||x — <f|| for all t e [0,1], the PF-ODE of the original interpolant It\i (x, e) = It (x, e) exists for all t e [0,1] (Albergo et al., 2023, which is incorporated by reference herein in its entirety). Then, for all ( - [0,1], 5 ^[0, t, let— Xf -F / E-’X. C - J 14 (. € ) Xy d«, (59)•A ] which pushes forward the measure qt (xf) to qs(xs). Define:WSGRNo. 67054-701.601fesjt(xf) — IsUi(7is;j.(xt),x / ). (60)

[0246] Then, sinces°’x‘) =,5,'Xs“x‘)) wherex~d'(x- M*d). Therefore, X., / s|t(hs[t(xf),xt) = (61) whose marginal follows qs(xs) due to it being the result of PF-ODE trajectories starting from

[0247] Concretely, DDIM interpolant satisfies all of the deterministic assumption, the regularity condition, and the invertibility assumption because it is a linear function of x and xt. Therefore, any diffusion or FM schedule with DDIM interpolant will enjoy a deterministic minimizerExample 3 — Analysis of Simplified Parameterization

[0248] 3.1 DDIM Interpolant

[0249] It is checked that DDIM interpolant is self-consistent. By definition,'• / « x() = 5(xg- DDIM(xf, x, t,.$)) jtjschecked that for all 5 < r < t,y (x_, |x,xr)« / ,.|t(xr^x,xt)dxr— y 5(x„ — DDIM(xr, x, r. s))< J(xr— DDIM(xf,x. t, r))dxr= 5(xs- DDIM(DDIM(xz, x, t, r). x, r, s) ) whereDDIM(DDIM(x()x, £, r), x. r. s) — agx+ (<7s / <r,.)([arx + (CTr / < T«)(xe — «tx)j — aTx)~ ltsX + (<7. / ffr)(o'r / o't)(Xj — C X)::: O:SX + ( < T,S / <7t) ( f — < X(X)= DDIM(xt)x. t,s)

[0250] Therefore, 3 (xs— DDIM(DDIM(xf, x, t, r), x, r, 5)) = 3 (xs— DDIM(xf, x, t, s)). So DDIM is self-consistent.

[0251] It also implies a Gaussian forward processasin diffusion models. By definition,flt.i(xt|x) ~ «&(xtjx) ~ J r / t(xt|x, e)p(e)deso that xt is a deterministic transform given x andXf~ DDIM(ε, x, t, 1) = αtx + which implies-*

[0252] 3.2 Reusing xtfor xr

[0253] It is proposed propose that instead of sampling xrvia forward flowG>x+, xt is reused such that xr= DDIM(xt, x, r, i) to reduce variance. In fact, for any self-consistent interpolant, one can reuse xt via xr~ qr\t (xr|x, xt) and xrwill follow qr(x ) marginally. It is checked™ I ™ / / (xr|x,x4) / % (xfjx, e)g(x)p(e)dedxclxjWSGRNo. 67054-701.601where (ci) is due to Lemma 5. Sampling x, xt first then xr~ qr\t (xr|x, xt) respects the marginal distribution qr(xi).

[0254] 3.3 Simplified Objective

[0255] The simplified objective is derived. Given MMD defined in Eq. (1), the objective is written as£MMSD(0) = E.,. Jw(s.t) ] (62) - Es.«[w(M) |EX„X ■) - ki fZ (Xj, -)l||2] (63):::E, Jw(s,t) \EX(.xJfe{f^4x< L •) - - k(f^(x'r), -)] }] (64):::Es>t[w(s.t) Ex^.xj.xj. ■)> + {k(f^r(xr),-), k(f^r«),•)) (65)= Ex„x,.x;!x;...stpa’(*\t) |*(A9t(xt). / *t(xJ)) + k(flr(x,), / *,. (x'r)) (66) - 4; (xr))] ]where is in RKHS, (a is due to the correlation between xrand xt by re-using xt.

[0256] 3.4 Empirical Estimation

[0257] As proposed in Gretton et al. (2012, which is incorporated by reference herein in its entirety), MMD is typically estimated with V-statistics by instantiating a matrix of size M x M such that a batch of B x samples, ' " ', is separated into groups of M (assume B is divisible by M) particleswhere each group share a (s1, r1, f) sample. The Monte Carlo estimate becomes\,f TMMSDW - 22 E E EEE)) (6?)

[0258] Computational efficiency. First it is noted that regardless of AT, w only 2 model forward passes are required - one with and one without stop gradient, since the model takes in all B instances together within the batch and produce outputs for the entire batch. For the calculation of the loss, although the need for M particles may imply inefficient computation, the cost of this matrix computation is negligible in practice compared to the complexity of model forward pass. Suppose a forward pass for a single instance is1, then the total computation for computation loss for a batch of B instances is) Deep neural networks often hasso O{BB. } dominates the computation.

[0259] 3.5 Simplified Parameterization

[0260] / (x-t is derived for each parameterization, which now generally follows the formWSGRNo. 67054-701.601

[0261] Identity. This is simply DDIM with x-prediction network.(ft’s ' I. Cn«ise( #)■>C:noist4-J ) *at 7rTt

[0262] Simple-EDM. / 2 I / 9k9V a? f af \ tjj \ / 07 -r <77....... -i f-sisi' -- s'! jX. - f ' ' S&wjfAAvxwfn 1

[0263] When noise schedule is cosine,. ' ' / .

[0264] And similar to Lu & Song (2024, which is incorporated by reference herein in its entirety), it can be shown that predicting Xs = DDIM(xt, x, s, t) with ^'2 loss is equivalent to u-prediction with cosine schedule.where+ o? \ — osn;e)(<:i’f + of) + <7sa'i — ast^crto«<i-f — aatXf<7t\ I l.... x •(, f I aaOt - asa-.tJ \ <rti.o7 + erf ) <*; + a; J \ / u? + of \ — o., Q:t)(uf + of) — (asai — {<xs#t — o'sO't)tt A — (Tatz 1 \ af + of + erf) J &t€ - crt-x.

[0265] This reduces to u-target if cosine schedule is used, and it deviates from v-target if FM schedule is used instead.

[0266] Euler-FM. Assume OT-FM schedule.WSGRNo. 67054-701.601((1 — s) — -(1 — fl) xt— ta^Gg“ Xj - (t - fi}< J<iGg

[0267] This results in Euler ODE from xt to Xs. It is also shown that the network output reduces to u-prediction if matched with Xs = DDIM(xt, x, s, f). To see this,- £((1 - «) - |(1 - t))x + |xt) || = - (fl - -)x +wherer_ ( _ L_ — )x + (- — 11 ((1 — t)x 4 telt " ' t (1- C’t J\ C— iS) t t= ( -..1) ((< - s)x + (s - f)e)€ — X

[0268] which is n-target under OT-FM schedule. This parameterization naturally allows zero-SNR sampling and satisfies boundary condition at 5 = 0, similar to Simple-EDM above. This is not true for Identity parametrization using Ge as it satisfies boundary condition only at 5 > 0.

[0269] 3.6 Mapping Function r(s,t)

[0270] The concrete choices for r(s, f) are discussed below. A constant decrement e > 0 is used in different spaces.

[0271] Constant decrement in 7(<'}This is the choice that is found to work betterthan other choices in practice. First, let its inverse beJ,r(. S, t) HiaX (.$, Yf1(jf / ft) — <:))

[0272] Choose * ~for some k.« 160 and « 0may bechosen, k = { 10,..., 15} works well enough depending on datasets.

[0273] Constant decrement in t.WSGRNo. 67054-701.601r($, t) max (s, t — c)

[0274] Choosef::::.(tj log-SNRt— 2k>g(a <7

[0275] Constant decrement in

[0276] Let its inverse be X'1(•), then) ~~ max (s, A”'1(A(t) ™ c)))

[0277] Choose * “. This choice comes close to the first choice, but this is refrained from because r(s, i) becomes close to t both when t « 0 and / ~ l instead of just t « 1. This gives more chances for training instability than the first choice.

[0278] Constant increment in 1 / 77(f).r ™ 0. / nt?. •

[0279] Choose

[0280] 3.7 Time Distribution p(s, t)

[0281] In all cases choose) and PHO -- for some £> 0 and T< 1. The decision for time distribution is coupled with r(s, f). The constraints on p(s, f) for each r(s, f) choice are listed below.

[0282] Constant decrement in 77 (t). T < 1 needs to be chosen because, for example, assuming OT-FM schedule, r]t = t / (l - f), one can observe that constant decrement in 77, when t « 1 results in r(s, f) that is too close to t due to 771’ s exploding gradient around 1. T< 1 needs to be defined such that r(s, T) is not too close to T for 5 reasonably far away. With i^nax ~ 160, T= 0.994 can be chosen for OT-FM and T= 0.996 for VP-diffusion.

[0283] Constant decrement in t. No constraints needed. T= 1, e = 0.

[0284] Constant decrement inOne can similarly observe exploding gradient causing r(s, i) to be too close to t at both t « 0 and t « 1, so, e.g. 0.001 can be chosen, in addition to choosing T = 0.994 for OT-FM and T= 0.996 for VP-diffusion.

[0285] Constant increment in l / r|t. This experience exploding gradient for t « 0, so€is required, e.g. 0.005. And T = l.

[0286] 3.8 Kernel Function

[0287] For the Laplace kernel k(x,y) = exp(-a>(, t) max(||x — y ||2, c) / D), let51e a reasonably small constant, e.g. 10'8. Looking at its gradient w.r.t. x,WSGRNo. 67054-701.601VTe- Jj Tnax(-^x^y^’e'^Dl O, otherwise

[0288] one can notice that the gradient is self-normalized to be a unit vector, which is helpful in practice. In comparison, the gradient of RBF kernel of the / <«, * )> fQrm(68)whose magnitude can vary a lot depending on how far x is from y.

[0289] For >(, t), it is helpful to write out the ’Moss between the arguments. For simplicity,denote- 7—7 « cskip(js. r)x', -I- r)G(x'5«, r) cKkip(s, )roKsi. S. ij >

[0290] Setfor the overall weighting to be 1. This allows invariance of magnitude of kernels w.r.t. t.

[0291] 3.9 Weighting Function w(s,t)

[0292] To review VDM (Kingma et al., 2021, which is incorporated by reference herein in its entirety), the negative ELBO loss for diffusion model is1 r / d \., 2] £a.so((ri “ 77^x,e,f I “GTAJ - € h (69)2 [\ di / J where is the noise-prediction network and t = log-SNRt. The weighted-ELBO loss proposed in Kingma & Gao (2024, which is incorporated by reference herein in its entirety) introduces an additional weighting function H monotonically increasing in t (monotonically decreasing in log-SNRt) understood as a form of data augmentation. Specifically, they use sigmoid as the function such that the weighted ELBO is written as£W-ELBO(#) — 4®x.c,t - At) I — -j-At I ||c0(x(,t) — Lp (70)Z \ Ut /

[0293] where G(.) is sigmoid function.

[0294] The at is tailored towards the Simple-EDM and Euler-FM parameterization as shown in Example 3.5 that the networks < dGe amounts to v-prediction in cosine and OT-FM schedules.WSGRNo. 67054-701.601Notice that ELBO diffusion loss matches e instead of v. Inspecting the gradient of Laplace kernel, (again, for simplicity letx& _ - w(«,i) | < (x«) - (x / D&0eD y / f Jx*) ™ f^r(x.r)|^ 00' 'M'lt). - (X<)- Gff{xt, f ) — GrargeiDe:for some constantIt is shown that gradients i- is guided by vectorAssumingG«^- *■* is v-prediction, as is the case for Simple-EDM parameterization with cosine schedule and Euler-FM parameterization with OT-FM schedule, v- to e-prediction can be reparametrized with ee as the new parameterization. Arguments to network are omitted for simplicity.

[0295] It is shown below that for both cases «* "•*-»«« ”for some constants etarget and For Simple-EDM, it is known that x-prediction from v-prediction parameterization (Salimans & Ho, 2022, which is incorporated by reference herein in its entirety), ' °'x*, and it is also known that "" -prediction from e-prediction,t>”! Xft ««•Xf ~~ afEg - - ™ (vx(™ atGy (71 J ftt xj ■■■■ cv^jq.... (72) (1 - + af&iGe ~ <7^ (73) <7i x(-F ™ (74) 6(5 ™ 4- - aiGg (75) - et^gCS•“ F atG& - F (76)€$ — £(aigei ~ CXj^Gf) ™ Gtargcl) (77)

[0296] For Euler-FM, x-prediction from v-prediction parameterization is known,and x-prediction from e -prediction is also known,x<!”,'x‘ “1WSGRNo. 67054-701.601* - xt- tG&(78) i z- JQ - te^ — (1 - t)xf~ / (I - t}G<) (79) tjQ + t(l — t}G$ ™ fefj (80) €# ™ Xf-*- (1 ” t)G$ (81 ) £& ~ Xt -*■ atG$ (82) 4^ c0- ™ xt4- atGfJ- (xt+ (S3)<™s> €# — etarget tf (Gf? — (84)

[0297] In both cases,-can rewri ten to (£0(Xt, s, t) - etarget) by multiplying a factor at, and the guidance vector now matches that of the ELBO-diffusion loss. Therefore, one is motivated to incorporate at into w(s, f) as proposed.

[0298] Lastly, the division of a + G is inspired by the increased weighting for middle timesteps (Esser et al., 2024, which is incorporated by reference herein in its entirety) for Flow Matching training. This is purely an empirical decision.Example 4 — Training Algorithm

[0299] Lastly, the division of a2+ G is inspired by the increased weighting for middle timesteps (Esser et al., 2024, which is incorporated by reference herein in its entirety) for Flow Matching training. This is purely an empirical decision.

[0300] Algorithm 3 MMSD TrainingInput: model f, data distribution q(x) and label distribution < / (c|x) (if label is used), prior distribution, time distribution p(f) and p(s\f), DDIM interpolator DDIM(xf, x, s, f) and its flow coefficients at, a, mapping function r(s, t), kernel function k(*, •), weighting function w(s, t), batch size B, particle number AT, label dropout probability pf0Output: learned modelJ sAInitializen<-while model not converged doSample a batch of data, label, and prior, and split into B / M groups,For each group, sample1,nd?- ' (* 4 ) for eachz- T^Sresuitsjn atuple(Optional) Randomly drop each label c(y)to be null token 0 with probability pWSGRNo. 67054-701.601Qn+i <— optimizer step by minimizing using model (see Eq. (67)) (optionally inputting c(!- into network)end whileExample 5 — Classifier-Free Guidance

[0301] Readers are referred to Example 3.5 for analysis of each parameterization. Most notably, the network Ge in both (1) Simple- EDM with cosine diffusion schedule and (2) Euler-FM with OT-FM schedule are equivalent to v-prediction parameterization in diffusion (Salimans & Ho, 2022, which is incorporated by reference herein in its entirety) and FM (Lipman et al., 2022, which is incorporated by reference herein in its entirety). When conditioned on label c during sampling, it is customary to use classifier-free guidance to reweight this v -prediction network viaG& (t-yii (t)x.z, Qiaise('^):<-:neise(t)" C) (.^5)— i-i-'Gti * Moisei 1)$ c) 4* (1 0)with guidance weight w so that the classifier-free guidedis1 ™ ^-skip ('F -f 1)^0 (, Cja.. CjKiisels):C) (k6.1 Example 6 — Sampling Algorithms

[0302] Pushforward sampling. Assume a series of N time stepswith T= tN > t -i > • • • > t2 > ti > to = e for the maximum time T and minimum time c. Denoteas data standard deviation.

[0303] Algorithm 4 Pushforward SamplingInput: model / 6, time steps, prior distribution ''?, (optional) guidance weight w Output:Samplefor i = TV,..., 1 do(Optional) w <— 1 if A=1 / / can optionally discard unconditional branch for N= 1end for

[0304] Restart sampling. Different from pushforward sampling, N time stepsdo not need to be strictly decreasing for all time steps, e.g. T = tN> tv-1 > • • • > t2> ti > to = £ (assuming T > e). Different from pushforward sampling, restart sampling first denoise a clean sample before resampling a noise to be added to this clean sample. Then a clean sample is predicted again. The process is iterated for N steps.

[0305] Algorithm 5 Restart SamplingWSGRNo. 67054-701.601Input: model fθ, time steps {ti}Ni=0, prior distribution!DDIM interpolant coefficients at and Gt, (optional) guidance weight wOutput:Samplefor i = N, " ■ ' 1 do(Optional) w <— 1 if TV =1 / / can optionally discard unconditional branch for N = 1if i '■ * 1 then£ JV(0, jl)044-1 £,,..x. lx F) / / or more generallyelse- xend ifend forExample 7 — Consistency with Additional Models

[0306] 7.1 Consistency Models

[0307] Consistency models explicitly match PF-ODE trajectories using a networkthat directly outputs a sample given any xt ~ qt (xt). The network explicitly uses EDM parameterization. 1 1 1» • ft IV fO _ V 1. 1 -^’X.-X-i •• (X;.;to satisfy boundary conditionvand trains via loss2where xris a deterministic function of xt from an ODE solver.

[0308] It is shown that CM loss is a special case of the simplified MMSD objective.Lemma 1. When x< -- xj, x,- ~~ x(, k{x, f] ~ — ||ar — i / iiiC. wid s > 0 is « small constant, Eg. (12) reduces to CM loss EX{. X1* |«’{ / ) ii£7t'(x< • 1) ” So- (x,.,r)||^ 4* ('for a validrlt) < t and some constant G Proof. Since xfxj, x,. — x'r, we have / ®;(xj) -- fff xf) and — f.^,.(x'.)- So k(ff:l(xt}, ff, (xt)) —(xrj, ~ 1 by definition. Since fcf ’, y } “ ifs — it is easy to see Eq. (12) reduces toExt,* - ZL(Xr)||2] + C (87)where C = 2 and w(s, f) is a weighting function. If 5 is a small positive constant, have «se><x«’t)where5 is dropped as input.If itself satisfies boundary condition at s = 0, s = 0 can be directly taken in which caseAnd under these assumptions, the loss becomesEXt.x.t - 5#~.(xr,r)|n + O, (88)WSGRNo. 67054-701.601which is simply a CM lossℓ2using distance.

[0309] However, one can notice that from a moment-matching perspective, this loss is problematic in two aspects. First, it assumes single particle estimate, which now ignores the entropy repulsion term in MMD that arises only during multi-particle estimation. This can contribute to mode collapse and training instability of CM. Second, the choice of energy kernel only matches the first moment, which is insufficient for matching two complex distributions! Kernels that match higher moments should be used in practice. In fact, it is shown in the following Lemma that the pseudo-huber loss proposed in Song & Dhariwal (2023, which is incorporated by reference herein in its entirety) matches higher moments as a kernel.

[0310] Lemma 2. Negative pseudo-huber loss &c(x, y) = c — \!’■ ' for c > 0 is a conditionally positive definite kernel that matches all moments of x and y where weights on higher moments depend on c.

[0311] Proof. First check that negative pseudo-huber loss1■■ js aconditionally positive definite kernel (Auffray & Barbillon, 2009, which is incorporated by reference herein in itsentirety). By definition, k(x, y) is conditionally positive definite if for xi, • • •,xn~ ~ and ci, • • •, Cn ~ * with

[0312] It is known that negative Li distance "is conditionally positive definite. This isproven below for completion. Due to triangle inequality, ' '. Then- 0where (a) is due toi. Now sincevfor all c > 0, have> 0 (93)

[0313] So negative pseudo-huber loss is a valid conditionally positive definite kernel.

[0314] Next, pseudo-huber loss’s effect on higher-order moments is analyzed by directly Taylor expandingvz=0WSGRNo. 67054-701.601— H42„ _|UHiS+- — ilzll84. O(l|dl9) (94) 2c ’’ Sc5'1!!16c5”4128c7 4 1- yf ~ ^k - y^ + ^!k - < ■ ■ 2k8+ - kh (95)(96)where - is substituted Each higher order x — 3for k > 2 expands to a polynomial containing up to k-th moments, i.e., 4...4 h fo, 4,... k), thus the implicit feature map contains all higher moments where c contributes to the weightings in front of each term.

[0315] Furthermore, the finite difference (between r(s, i) and t) MMSD objective to the differential limit is extended by taking r(s, f) <— t in Example 8. This results in a new objective that similarly subsumes continuous-time CM (Song et al., 2023; Lu & Song, 2024, which is incorporated by reference herein in its entirety) as a single-particle special case.

[0316] 7.2 Diffusion GAN and Adversarial Consistency Distillation

[0317] Diffusion GAN (Xiao et al., 2021, which is incorporated by reference herein in its entirety) parameterizes its generative distribution asp^H(xs|x^) / q^i^s |<?£)(x,f?z)?x / -)p(z)dzwhere Ge is a neural network, / ?(z) is standard Gaussian distribution, and qs|t(xs|x,xt) is the DDPM posterior(97) h-Q ™ - yXi + a,41 - -)x cry ■1 >j « or -r-;. & Q::::'LG —aj a;

[0318] Note that DDPM posterior is a stochastic interpolant, and more importantly, it is self-consistent, which is shown in the Lemma below.

[0319] Lemma 6. For all 0 < 5 < t < 1, DDPM posterior distribution from t to 5 as defined in Eq. (97) is a seif-consistent Gaussian interpolant betweenxandxt.f. Let x,, <•>» ^(xr|x, x4) and ^Sf.(x:Jx, x,,), we show that x» follows fekx4x- **)•x;™ - s-Xf 4- 0.41 - — sr)x 4 (7r* / 1 - xr-x6i arar tx* erf y a*; / ta; cr^, / ajM X« ™~™-Xr-4- 1 ™ ~~~ ~~pj!. X "s~ (7 <;-i / 1 "'■ ~“4'' Ct" (Tf VWSGRNo. 67054-701.601-, Hi. / !where are i.i.d. Gaussian noise. Directly expanding«>-a? [tjcr- of ay,. 1,, aft,, j., <4 ay x4- ~Xt+ O:,- ( 1 - — PC -r a,, I 1 - - €i + «„ ( 1 - 7 -=• )X + <?♦» / 1 - T. Co ( 10()s ay ay y a* af a; <7; y a; ay (101) «;tr^ at a~ artri f at a'? i al ai - 2X; 4- aJl - ^'ix -i - -4 / 1 - — 7— €j “ — -^€7 (102) aaay ay ay • axa,. y ay ay y ay ay in) a(a(103)asa where (a) is due to the fact that sum of two independent Gaussian variables with variance a2and b2is also Gaussian with variance a2+ b2, and C3 ~ N(0, 1) is another independent Gaussian noise. The calculation of the variance is shown:a; <7;,.2. a: a;— p -H (1 -•22•4 a; af a; 072 _22 2 '

[0320] This shows xsfollows ^|t(xs|x, xt) and completes the proof.

[0321] This shows another possible design of the interpolant that can be used for self-distillation, and diffusion GAN’s formulation generally complies with the design of the generative distribution, except that it learns this conditional distribution of x given xt directly while a marginal distribution is learned. When they directly learn the conditional distribution by matchingwithmo(jeiisforced to learn < / t(x|xt) and there only exists one minimizer. However, in this case, the model can learn multiple different solutions because the marginals are matched instead.

[0322] GAN loss and MMD loss There is also similarity between GAN loss used in Xiao et al. (2021, which is incorporated by reference herein in its entirety); Sauer et al. (2025, which is incorporated by reference herein in its entirety) and MMD loss. MMD is an integral probability metric over a set of functions F in the following form= sup |X P(X)(X) ••••••• ^,'y) / (y) |where a supremum is taken on this set of functions. This naturally gives rise to an adversarial optimization algorithm if F is defined as the set of neural networks. However, MMD bypasses this by selecting F as the RKHS where the optimal f can be analytically found. This eliminates the adversarial objective and gives a stable minimization objective in practice. However, this is not to say that RKHS is the best function set. With the right optimizers and training scheme, theWSGRNo. 67054-701.601adversarial objective may achieve better empirical performance, but this also makes the algorithm difficult to scale to large datasets.

[0323] 7.3 Generative Moment Matching Network

[0324] It is trivial to check that GMMN is a special parameterization. Fix t =1, and due to boundary condition, r(s, t) = s = 0 implies training target< / slXs' the data distribution. Additionally, is a simple pushforward of prior p(c) through networkwhere drop dependency on t and 5 since they are constant.Example 8 — Differential Moment Matching Self-Distillation

[0325] Similar to the continuous-time CMs presented in (Lu & Song, 2024, which is incorporated by reference herein in its entirety), the MMD self-distillation objective can be taken to the differential limit. Consider the simplified loss and parameterization in Eq. (12), the RBF kernel is used as the kernel of choice for simplicity.

[0326] Theorem 2 (Differential Moment Matching Self-Distillation). Letbe a twice continuously differentiable function with bounded first and second derivatives, let k(», •) be RBF.x.x'kernel with unit bandwidth,xl ~~ x. = DDIMiXi,x, t, r) afwlx' = DDLM(x^,x', (he following objective. / £,(<)). can be analytically derived as(105)

[0327] Proof. Firstly, the limit can be exchanged with the expectation due to dominated convergence theorem where the integrand consists of kernel functions which can be assumed to be upper bounded by 7, e.g. RBF kernels are upper bounded by 7, and thus integrable. It then suffices to check the limit of the integrand. Before that, let us review the first and second-order Taylor expansion of* ' ■ wekta.ocja"consants to be expanded around. The Taylor expansion to second-order is noted down below for notational convenience.WSGRNo. 67054-701.601* __f? i!11''g: / -... b11 i'!s... <j;^ <p... 4,...p_ |:> g / 2(i... fafa' a - &):- / ) (a: - ®) *S{ y -:,-‘2.'2. «[ ’ (j / &) 4' "' &)?SIv“** li Z2((b «)(? J gj ' / ) (} / &) »eh1" yij Z2a(x:;;; a<y ~ fae“|:« •• *>!: / i.... g" «»"•4S.... f (JJ;...aj...e"4*>”«S >'- (&...0J ' (y... fa.••!•■ —{ff. fel:( (b. a)(i). «) '. 7) (y. b} 1 'y.( jj aj •, -y.•;• -(&. 6):e”:“s((fe. a) (fe. ©) '. / ) (3 / . &)4- (a- G):S3~ ii*“ai8; 2(I — (a &} (a -- ft):) (y -- b)

[0328] Putting it together, the above results imply,4.........e-p. M2 / 2....e- h^-^P / :2(x a) ' e~’^~"s!■'J(I (a- 6) (a &) ' ) (y ■■■■ &isince it is easy to check that the remaining terms cancel.

[0329] Substituting}; == 0:==-^{x’}-y 6=lim (W6)(107)(108)

[0330] Similarly,. b) =:; hm - — -. / S,4X, JJ = - - (109)

[0331] Therefore,can be derived as / (s'Xr- (110) di. (fU (111)HZ- (112)

[0332] 8.1 Pseudo-ObjectiveWSGRNo. 67054-701.601

[0333] Due to the stop-gradient operation, a pseudo-objective can similarly be found whose gradient matches the gradient of0 jnijmitof7I”*

[0334] Theorem 3. Letbe a twice continuously differentiable function with bounded first and second derivatives, k(»,») be RBF kernel with unit bandwidth,x, x ^(x), x4x*;x' ). ’St pc' )- x, i. r) andX*"x r>’* ^he gradient of the following pseudo-objective- Imi (<)) 0H)can be used to optimize 0 and can be analytically derived as(114)•• iff (*«). v^W df f (xt) ■*r A,f txr1§df as i-

[0335] Proof. Similar to the derivation of 05j)?kt **:x:ft havelim - — • — ••••••• a); / 2(f fa ) (y bi(t. V; ’ ’ ' " 'where,, 1z,df.^ (x?) 1 {Xj; hill - - (it:. ft } — -:- Ihu - - -(w. bi, f....... ’ ■'r..^t frj ' ilttiP",

[0336] Note that is now parameterized by ' instead of 0 because the gradient is already taken w.r.t. / f((x() outside of the brackets, so (x — d) and (y — V) merely require evaluation at current Q with no gradient information, whichsatisfies. The objective can be derived asWSGRNo. 67054-701.601f..Q,.„ 1 ’ df* (xj).-.... < T \ J — \.. i A < (Xt) / ; t (xj — ",, \f» t (X() f. t i Vs^ix, H- iU i.t <t -• ■ If o -,, J-, JWA W LT., j T \,f\ -... - [A,f{xt). ■x. q

[0337] 8.2 Connection with Continuous-Time CMs

[0338] Observing Eq. (105) and Eq. (110), it can be seen that whenx* ”x*and Xr x> s being a small positive constant, thenj •-•-• -l-a,sJwhere since 5 is fixed the dependency on 5 is discarded as input. Then, Eq. (105) reduces to(115)which is the same as differential consistency loss (Song et al., 2023; Geng et al., 2024; each of which is incorporated by reference herein in its entirety). And Eq. (110) reduces to(116)which is the pseudo-objective for continuous-time CMs (Song et al., 2023; Lu & Song, 2024; each of which is incorporated by reference herein in its entirety) (minus a weighting function of choice).Example 9 - Experiment Settings

[0339] The best runs are summarized in Table 5. Specifically, for ImageNet-256×256, a latent space paradigm is adopted for computational efficiency. For its autoencoder, EDM2 is followed (Karras et al., 2024, which is incorporated by reference herein in its entirety) and all images from ImageNet are pre-encoded into latents without flipping, and the channel-wise mean and std for normalization is calculated. Stable Diffusion VAE is used and the latents are rescaled by channel mean [0.86488, —0.27787343, 0.21616915, 0.3738409] and channel std [4.85503674, 5.31922414, 3.93725398, 3.9870003], After this normalization transformation, the latents are further multiplied by 0.5 so that the latents roughly have std 0.5. For DiT architecture of different sizes, the same hyperparameters are used for all experiments.

[0340] Choices for T and c By default assuming mapping function r (s, f) is used by constant decrement in z / t, / max ~ 160 is kept. This implies that for time distribution of the form U (e, 7), T = 0.996 is set for cosine diffusion and T= 0.994 for OT-FM. For e, it is set differently for pixelspace and latent-space model, e = 0 is set by default.Table 5. Experimental settings for different architectures and datasets.WSGRNo. 67054-701.601CIFAR-10 ImageNet-256x256 Parameterization SettingArchitecture DDPM++ DiT-S DiT-B DiT-L DiT-XL GFlops 21.28 6.06 23.01 80.71 118.64 Params (M) 55 33 130 458 675 Cnoise ( / ) 1000 / 1000 / 1000 / 1000 / 1000 / Flow Trajectory OT-FM OT-FM OT-FM OT-FM OT-FM Euler- ge (xt,s,f) Simple-EDM Euler-FM Euler-FM Euler-FM FM cd 0.5 0.5 0.5 0.5 0.5 Training iter 400K 1.2M 1.2M 1.2M 1.2M Training SettingDropout 0.2 0. 0. 0. 0. Optimizer RAdam Adam Adam Adam Adam Optimizer e IO’8IO’8IO’8IO’8IO’8fii 0.9 0.9 0.9 0.9 0.9 p20.999 0.999 0.999 0.999 0.999 Learning Rate 0.0001 0.0001 0.0001 0.0001 0.0001 Weight Decay 0 0 0 0 0 Batch Size 4096 4096 4096 4096 4096 M 8 4 Kernel Laplace Laplace Laplace Laplace Laplace r(5, / ), -i,nW(0, 0.994), M0.0.994 p(t) Mb, 0.994) * ' ' «({), 0.994) M 0,0.994) b 5 4 4 4 4 EMA Rate 0.9999 0.9999 0.9999 0.9999 0.9999 x-flip True False False False FalseInference Setting Pushforwa Pushforwar Pushforwa Pushforw Sampler Type Restart rd d rd ard Number of Steps 2 8 8 8 8 Schedule Type / 1 = 1.35 Uniform Uniform Uniform Uniform FID-50K (w = 0) 2.05....FID-50K (w = 1.0, i.e. - 42.28 26.02 9.33 7.25 no guidance)FID-50K (w = 1.5) - 20.36 9.69 2.80 2.13

[0341] Exceptions occur when other choices of r (5, f) are ablated, e.g. constant decrement in At in which case c = 0.001 is set to prevent r (5, f) for being too close to t when t is small.

[0342] Injecting time 5. The design for additionally injecting 5 exactly follows the time injection of t. Positional time embedding of.s fed through 2-layer MLP (same as for f) is extracted before adding this new embedding to the embedding of t after MLP. The summed embedding is then fed through all the Transformer blocks as in standard DiT architecture.

[0343] Improved CT baseline. For ImageNet-256×256, iCT baseline is implemented by using the improved parameterization with Simple-EDM and OT-FM schedule. The proposed pseudo-huber loss is used for training but find training often collapses using the same r (s, i) schedule.WSGRNo. 67054-701.601The gap is carefully tuned to achieve reasonable performance without collapse and the results are presented in Table 2.

[0344] 9.2 Inference Settings

[0345] Inference schedules. For all one-step inference, directly start from e ~ N (0,at time T to time e through pushforward sampling. For all 2-step methods, set the intermediate timestep ti such that ]ti = 1.35; this choice is arbitrary but works well. For N> 4 steps explore two types of time schedules: (1) uniform decrement in / with r / o < r]i < zv whereL = T + 7’j H17}N ' and (2) EDM (Karras et al., 2022, which is incorporated by reference herein in its entirety) time schedule. EDM schedule specifies rjo < rji < zv where

[0346] The schedule is slightly modified so that z o = zmin is the endpoint instead of 771 = zmin and TJO = 0 as originally proposed, since the TJO can be set to 0 without numerical issue.

[0347] The time schedule type used for the best runs and their results is also specified in Table 5.

[0348] 9.3 Scaling Settings

[0349] Model GFLOPs Numbers from DiT (Peebles & Xie, 2023, which is incorporated by reference herein in its entirety) are reused for each model architecture.

[0350] Training compute. Following Peebles & Xie (2023, which is incorporated by reference herein in its entirety), the formula model GFLOPs batch size training steps 4 is used for training compute where, different from DiT, constant is 4 because for each iteration there are 2 forward pass and 1 backward pass, which is estimated as twice the forward compute.

[0351] Inference compute. Inference compute is calculated via model GFLOPs number of steps.

[0352] Additional visualizations are shown in FIGs. 12-14.Example 10 — Additional Demonstration of Technical Improvements

[0353] Tables 6-8 provide additional demonstrations of technical improvements over other methods described herein.Table 6CIFAR-10 FID1-step _ 3,202-step 1.98Table 7WSGRNo. 67054-701.601ImageNet-256x256 FID1 -step (w= 1.25) 7.772-step (w=1.25) 5.334-step (w=1.25) 3.668-step (w=1.25) 2.771 -step (w= 1.5) 8.052-step (w=1.5) 3.994-step (w=1.5) 2.518-step (w=1.5) 1.99Table 8ImageNet-256x256 FID10-step (w=1.5) 1.9816-step (w=1.5) 1.9032-step (w=1.5) 1.89Computing systems

[0354] In another aspect, disclosed here is a system comprising at least one processor and instructions executable by the at least one processor to cause the at least one processor to perform operations comprising: (a) obtaining (i) a first dataset comprising an observed data sample from a data distribution of observed data and (ii) a second dataset comprising a prior sample from a prior distribution; (b) synthesizing first data at a first time step based on the observed data sample and the prior sample; (c) synthesizing second data at a second time step based on the observed data sample and the first data, wherein the second time step is between a target time step and the first time step; (d) generating, using a neural network, a first output and a second output at the target time step, wherein (i) the first output is obtained using the first data with the target time step and the first time step as conditioning inputs and (ii) the second output is obtained using the second data with the target time step and the second time step as conditioning inputs; and (e) obtaining a trained generative Al model by updating parameters of the neural network, wherein the updating comprises at least computing a loss between a first distribution of the first output and a second distribution of the second output.

[0355] Referring to FIG. 15, a block diagram is shown depicting an exemplary machine that includes a computer system 1500 (e.g., a processing or computing system) within which a set of instructions can execute for causing a device to perform or execute any one or more of the aspects and / or methodologies for static code scheduling of the present disclosure. The components in FIG. 15 are examples only and do not limit the scope of use or functionality of any hardware,WSGRNo. 67054-701.601software, embedded logic component, or a combination of two or more such components implementing particular embodiments.

[0356] Computer system 1500 may include one or more processors 1501, a memory 1503, and a storage 1508 that communicate with each other, and with other components, via a bus 1540. The bus 1540 may also link a display 1532, one or more input devices 1533 (which may, for example, include a keypad, a keyboard, a mouse, a stylus, etc.), one or more output devices 1534, one or more storage devices 1535, and various tangible storage media 1536. All of these elements may interface directly or via one or more interfaces or adaptors to the bus 1540. For instance, the various tangible storage media 1536 can interface with the bus 1540 via storage medium interface 1526. Computer system 1500 may have any suitable physical form, including but not limited to one or more integrated circuits (ICs), printed circuit boards (PCBs), mobile handheld devices (such as mobile telephones or PDAs), laptop or notebook computers, distributed computer systems, computing grids, or servers.

[0357] Computer system 1500 includes one or more processor(s) 1501 (e.g., central processing units (CPUs) or general purpose graphics processing units (GPGPUs)) that carry out functions. Processor(s) 1501 optionally contains a cache memory unit 1502 for temporary local storage of instructions, data, or computer addresses. Processor(s) 1501 are configured to assist in execution of computer readable instructions. Computer system 1500 may provide functionality for the components depicted in FIG. 15 as a result of the processor(s) 1501 executing non-transitory, processor-executable instructions embodied in one or more tangible computer-readable storage media, such as memory 1503, storage 1508, storage devices 1535, and / or storage medium 1536. The computer-readable media may store software that implements particular embodiments, and processor(s) 1501 may execute the software. Memory 1503 may read the software from one or more other computer-readable media (such as mass storage device(s) 1535, 1536) or from one or more other sources through a suitable interface, such as network interface 1520. The software may cause processor(s) 1501 to carry out one or more processes or one or more steps of one or more processes described or illustrated herein. Carrying out such processes or steps may include defining data structures stored in memory 1503 and modifying the data structures as directed by the software.

[0358] The memory 1503 may include various components (e.g., machine readable media) including, but not limited to, a random access memory component (e.g., RAM 1504) (e.g., static RAM (SRAM), dynamic RAM (DRAM), ferroelectric random access memory (FRAM), phasechange random access memory (PRAM), etc.), a read-only memory component (e.g., ROM 1505), and any combinations thereof. ROM 1505 may act to communicate data and instructions unidirectionally to processor(s) 1501, and RAM 1504 may act to communicate data andWSGRNo. 67054-701.601instructions bidirectionally with processor(s) 1501. ROM 1505 and RAM 1504 may include any suitable tangible computer-readable media described below. In one example, a basic input / output system 1506 (BIOS), including basic routines that help to transfer information between elements within computer system 1500, such as during start-up, may be stored in the memory 1503.

[0359] Fixed storage 1508 is connected bidirectionally to processor(s) 1501, optionally through storage control unit 1507. Fixed storage 1508 provides additional data storage capacity and may also include any suitable tangible computer-readable media described herein. Storage 1508 may be used to store operating system 1509, executable(s) 1510, data 1511, applications 1512 (application programs), and the like. Storage 1508 can also include an optical disk drive, a solid-state memory device (e.g., flash-based systems), or a combination of any of the above. Information in storage 1508 may, in appropriate cases, be incorporated as virtual memory in memory 1503.

[0360] In one example, storage device(s) 1535 may be removably interfaced with computer system 1500 (e.g., via an external port connector (not shown)) via a storage device interface 1525. Particularly, storage device(s) 1535 and an associated machine-readable medium may provide non-volatile and / or volatile storage of machine-readable instructions, data structures, program modules, and / or other data for the computer system 1500. In one example, software may reside, completely or partially, within a machine-readable medium on storage device(s) 1535. In another example, software may reside, completely or partially, within processor(s) 1501.

[0361] Bus 1540 connects a wide variety of subsystems. Herein, reference to a bus may encompass one or more digital signal lines serving a common function, where appropriate. Bus 1540 may be any of several types of bus structures including, but not limited to, a memory bus, a memory controller, a peripheral bus, a local bus, and any combinations thereof, using any of a variety of bus architectures. As an example and not by way of limitation, such architectures include an Industry Standard Architecture (ISA) bus, an Enhanced ISA (EISA) bus, a Micro Channel Architecture (MCA) bus, a Video Electronics Standards Association local bus (VLB), a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, an Accelerated Graphics Port (AGP) bus, HyperTransport (HTX) bus, serial advanced technology attachment (SATA) bus, and any combinations thereof.

[0362] Computer system 1500 may also include an input device 1533. In one example, a user of computer system 1500 may enter commands and / or other information into computer system 1500 via input device(s) 1533. Examples of an input device(s) 1533 include, but are not limited to, an alpha-numeric input device (e.g., a keyboard), a pointing device (e.g., a mouse or touchpad), a touchpad, a touch screen, a multi-touch screen, a joystick, a stylus, a gamepad, an audio input device (e.g., a microphone, a voice response system, etc.), an optical scanner, a video or still image capture device (e.g., a camera), and any combinations thereof. In some embodiments, the inputWSGRNo. 67054-701.601device is a Kinect®, Leap Motion®, or the like. Input device(s) 1533 may be interfaced to bus 1540 via any of a variety of input interfaces 1523 (e.g., input interface 1523) including, but not limited to, serial, parallel, game port, USB, FIREWIRE, THUNDERBOLT, or any combination of the above.

[0363] In particular embodiments, when computer system 1500 is connected to network 1530, computer system 1500 may communicate with other devices, specifically mobile devices and enterprise systems, distributed computing systems, cloud storage systems, cloud computing systems, and the like, connected to network 1530. Communications to and from computer system 1500 may be sent through network interface 1520. For example, network interface 1520 may receive incoming communications (such as requests or responses from other devices) in the form of one or more packets (such as Internet Protocol (IP) packets) from network 1530, and computer system 1500 may store the incoming communications in memory 1503 for processing. Computer system 1500 may similarly store outgoing communications (such as requests or responses to other devices) in the form of one or more packets in memory 1503 and communicated to network 1530 from network interface 1520. Processor(s) 1501 may access these communication packets stored in memory 1503 for processing.

[0364] Examples of the network interface 1520 include, but are not limited to, a network interface card, a modem, and any combination thereof. Examples of a network 1530 or network segment 1530 include, but are not limited to, a distributed computing system, a cloud computing system, a wide area network (WAN) (e.g., the Internet, an enterprise network), a local area network (LAN) (e.g., a network associated with an office, a building, a campus or other relatively small geographic space), a telephone network, a direct connection between two computing devices, a peer-to-peer network, and any combinations thereof. A network, such as network 1530, may employ a wired and / or a wireless mode of communication. In general, any network topology may be used.

[0365] Information and data can be displayed through a display 1532. Examples of a display 1532 include, but are not limited to, a cathode ray tube (CRT), a liquid crystal display (LCD), a thin film transistor liquid crystal display (TFT-LCD), an organic liquid crystal display (OLED) such as a passive-matrix OLED (PMOLED) or active-matrix OLED (AMOLED) display, a plasma display, and any combinations thereof. The display 1532 can interface to the processor(s) 1501, memory 1503, and fixed storage 1508, as well as other devices, such as input device(s) 1533, via the bus 1540. The display 1532 is linked to the bus 1540 via a video interface 1522, and transport of data between the display 1532 and the bus 1540 can be controlled via the graphics control 1521. In some embodiments, the display is a video projector. In some embodiments, the display is a head-mounted display (HMD) such as a VR headset. In further embodiments, suitable VR headsets include, by way of non-limiting examples, HTC Vive®, Oculus Rift®, Samsung GearWSGRNo. 67054-701.601VR®, Microsoft HoloLens®, Razer OSVR®, FOVE VR®, Zeiss VR One®, Avegant Glyph®, Freefly VR® headset, and the like. In still further embodiments, the display is a combination of devices such as those disclosed herein.

[0366] In addition to a display 1532, computer system 1500 may include one or more other peripheral output devices 1534 including, but not limited to, an audio speaker, a printer, a storage device, and any combinations thereof. Such peripheral output devices may be connected to the bus 1540 via an output interface 1524. Examples of an output interface 1524 include, but are not limited to, a serial port, a parallel connection, a USB port, a FIREWIRE port, a THUNDERBOLT port, and any combinations thereof.

[0367] In addition or as an alternative, computer system 1500 may provide functionality as a result of logic hardwired or otherwise embodied in a circuit, which may operate in place of or together with software to execute one or more processes or one or more steps of one or more processes described or illustrated herein. Reference to software in this present disclosure may encompass logic, and reference to logic may encompass software. Moreover, reference to a computer-readable medium may encompass a circuit (such as an IC) storing software for execution, a circuit embodying logic for execution, or both, where appropriate. The present disclosure encompasses any suitable combination of hardware, software, or both.

[0368] Various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality.

[0369] The various illustrative logical blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented or performed with a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but in the alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.

[0370] The steps of a method or algorithm described in connection with the embodiments disclosed herein may be embodied directly in hardware, in a software module executed by one or more processor(s), or in a combination of the two. A software module may reside in RAMWSGRNo. 67054-701.601memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium. An exemplary storage medium is coupled to the processor such the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.

[0371] In accordance with the description herein, suitable computing devices include, by way of non-limiting examples, server computers, desktop computers, laptop computers, notebook computers, sub-notebook computers, netbook computers, netpad computers, set-top computers, media streaming devices, handheld computers, Internet appliances, mobile smartphones, tablet computers, personal digital assistants, video game consoles, and vehicles. Select televisions, video players, and digital music players with optional computer network connectivity are suitable for use in the system described herein. Suitable tablet computers, in various embodiments, include those with booklet, slate, and convertible configurations.

[0372] In some embodiments, the computing device includes an operating system configured to perform executable instructions. The operating system is, for example, software, including programs and data, which manages the device’s hardware and provides services for execution of applications. Suitable server operating systems include, by way of non-limiting examples, FreeBSD®, OpenBSD®, NetBSD®, Linux®, Apple® Mac OS X Server®, Oracle Solaris®, Windows Server®, and Novell NetWare®. Suitable personal computer operating systems include, by way of non-limiting examples, Microsoft Windows®, Apple Mac® OS X, UNIX®, and UNIX-like operating systems such as GNU / Linux®. In some embodiments, the operating system is provided by cloud computing. Suitable mobile smartphone operating systems include, by way of non-limiting examples, Nokia Symbian® OS, Apple® iOS, Research In Motion BlackBerry® OS, Google® Android®, Microsoft® Windows Phone® OS, Microsoft® Windows Mobile OS, Linux®, and Palm® WebOS. Suitable media streaming device operating systems include, by way of nonlimiting examples, Apple TV®, Roku®, Boxee®, Google TV®, Google Chromecast®, Amazon Fire®, and Samsung® HomeSync®. Suitable video game console operating systems include, by way of non-limiting examples, Sony® PS3®, Sony® PS4®, Microsoft® Xbox 360®, Microsoft Xbox One®, Nintendo Wii®, Nintendo Wii U®, and Ouya®. Suitable virtual reality headset systems include, by way of non-limiting example, Meta Oculus®.Non-transitory computer readable storage mediums

[0373] In some embodiments, the platforms, systems, media, and methods disclosed herein include one or more non-transitory computer readable storage media encoded with a programWSGRNo. 67054-701.601including instructions executable by the operating system of an optionally networked computing device. In further embodiments, a computer readable storage medium is a tangible component of a computing device. In still further embodiments, a computer readable storage medium is optionally removable from a computing device. In some embodiments, a computer readable storage medium includes, by way of non-limiting examples, CD-ROMs, DVDs, flash memory devices, solid state memory, magnetic disk drives, magnetic tape drives, optical disk drives, distributed computing systems including cloud computing systems and services, and the like. In some cases, the program and instructions are permanently, substantially permanently, semipermanently, or non-transitorily encoded on the media.Computer programs

[0374] In another aspect, disclosed herein is a computer program product, the computer program product comprising at least one non-transitory computer-readable medium having computer-readable program code portions embodied therein, the computer-readable program code portions comprising: an executable portion configured to obtain (i) a first dataset comprising an observed data sample from a data distribution of observed data and (ii) a second dataset comprising a prior sample from a prior distribution; an executable portion configured to synthesize first data at a first time step based on the observed data sample and the prior sample; an executable portion configured to synthesize second data at a second time step based on the observed data sample and the first data, wherein the second time step is between a target time step and the first time step; an executable portion configured to generate, using a neural network, a first output and a second output at the target time step, wherein (i) the first output is obtained using the first data with the target time step and the first time step as conditioning inputs and (ii) the second output is obtained using the second data with the target time step and the second time step as conditioning inputs; and an executable portion configured to obtain a trained generative AI model by updating parameters of the neural network, wherein the updating comprises at least computing a loss between a first distribution of the first output and a second distribution of the second output.

[0375] In some embodiments, the platforms, systems, media, and methods disclosed herein include at least one computer program, or use of the same. A computer program includes a sequence of instructions, executable by one or more processor(s) of the computing device’s CPU, written to perform a specified task. Computer readable instructions may be implemented as program modules, such as functions, objects, Application Programming Interfaces (APIs), computing data structures, and the like, that perform particular tasks or implement particular abstract data types. In light of the present disclosure provided herein, a computer program may be written in various versions of various languages.WSGRNo. 67054-701.601

[0376] The functionality of the computer readable instructions may be combined or distributed as desired in various environments. In some embodiments, a computer program comprises one sequence of instructions. In some embodiments, a computer program comprises a plurality of sequences of instructions. In some embodiments, a computer program is provided from one location. In other embodiments, a computer program is provided from a plurality of locations. In various embodiments, a computer program includes one or more software modules. In various embodiments, a computer program includes, in part or in whole, one or more web applications, one or more mobile applications, one or more standalone applications, one or more web browser plug-ins, extensions, add-ins, or add-ons, or combinations thereof.Web applications

[0377] In some embodiments, a computer program includes a web application. In light of the present disclosure provided herein, a web application, in various embodiments, utilizes one or more software frameworks and one or more database systems. In some embodiments, a web application is created upon a software framework such as Microsoft®. NET or Ruby on Rails® (RoR). In some embodiments, a web application utilizes one or more database systems including, by way of non-limiting examples, relational, non-relational, object oriented, associative, and XML database systems. In further embodiments, suitable relational database systems include, by way of non-limiting examples, Microsoft® structured query language (SQL) Server, mySQL™, and Oracle®. A web application, in various embodiments, is written in one or more versions of one or more languages. A web application may be written in one or more markup languages, presentation definition languages, client-side scripting languages, server-side coding languages, database query languages, or combinations thereof. In some embodiments, a web application is written to some extent in a markup language such as Hypertext Markup Language (HTML), Extensible Hypertext Markup Language (XHTML), or extensible Markup Language (XML). In some embodiments, a web application is written to some extent in a presentation definition language such as Cascading Style Sheets (CSS). In some embodiments, a web application is written to some extent in a client-side scripting language such as Asynchronous Javascript and XML® (AJAX), Flash Actionscript, Javascript®, or Silverlight®. In some embodiments, a web application is written to some extent in a server-side coding language such as Active Server Pages® (ASP), ColdFusion®, Perl®, Java®, JavaServer Pages® (JSP), Hypertext Preprocessor® (PHP), Python®, Ruby®, Tel®, Smalltalk®, WebDNA®, or Groovy®. In some embodiments, a web application is written to some extent in a database query language such as Structured Query Language (SQL). In some embodiments, a web application integrates enterprise server products such as IBM Lotus Domino®. In some embodiments, a web application includes a media player element. In various further embodiments, a media player element utilizes one or more of many suitable multimediaWSGRNo. 67054-701.601technologies including, by way of non-limiting examples, Adobe® Flash®, HTML 5, Apple® QuickTime®, Microsoft Silverlight®, Java®, and Unity®.

[0378] Referring to FIG. 16, in a particular embodiment, an application provision system comprises one or more databases 1600 accessed by a database management system (DBMS) 1610. Suitable DBMSs include Firebird®, MySQL®, NoSQL®, PostgreSQL®, SQLite®, Oracle Database®, Microsoft SQL Server®, IBM DB2®, IBM Informix®, SAP Sybase®, SAP Sybase®, Teradata®, PostGIS®, Apache® Hive, Apache® Impala, time-series databases, graph databases, key-value storage, and the like. In this embodiment, the application provision system further comprises one or more application severs 1620 (such as Java® servers,. NET® servers, PHP® servers, and the like) and one or more web servers 1630 (such as Apache®, IIS®, GWS® and the like). The web server(s) optionally expose one or more web services via app application programming interfaces (APIs) 1640. Via a network, such as the Internet, the system provides browser-based and / or mobile native user interfaces. In some cases, a DBMS may be a relational DBMS.

[0379] Referring to FIG. 17, in a particular embodiment, an application provision system alternatively has a distributed, cloud-based architecture 1700 and comprises elastically load balanced, auto-scaling web server resources 1710 and application server resources 1720 as well synchronously replicated databases 1730.Mobile applications

[0380] In some embodiments, a computer program includes a mobile application provided to a mobile computing device. In some embodiments, the mobile application is provided to a mobile computing device at the time it is manufactured. In other embodiments, the mobile application is provided to a mobile computing device via the computer network described herein.

[0381] In view of the present disclosure provided herein, a mobile application is created by techniques using hardware, languages, and development environments. Mobile applications are written in several languages. Suitable programming languages include, by way of non-limiting examples, C, C++, C#, Objective-C, Java®, Javascript®, Pascal®, Object Pascal®, Python™, Ruby®, VB. NET®, WML®, and XHTML / HTML with or without CSS, or combinations thereof.

[0382] Suitable mobile application development environments are available from several sources. Commercially available development environments include, by way of non-limiting examples, AirplaySDK®, alcheMo®, Appcelerator®, Celsius®, Bedrock®, Flash Lite®,. NET Compact Framework®, Rhomobile®, and WorkLight Mobile Platform®. Other development environments are available without cost including, by way of non-limiting examples, Lazarus®, MobiFlex®, MoSync®, and Phonegap®. Also, mobile device manufacturers distribute software developer kits including, by way of non-limiting examples, iPhone® and iPad® (iOS) SDK, Android® SDK,WSGRNo. 67054-701.601BlackBerry® SDK, BREW SDK, Palm® OS SDK, Symbian® SDK, webOS® SDK, and Windows® Mobile SDK.

[0383] Several commercial sources are available for distribution of mobile applications including, by way of non-limiting examples, Apple® App Store, Google® Play, Chrome® WebStore, BlackBerry® App World, App Store® for Palm devices, App Catalog® for webOS, Windows® Marketplace for Mobile, Ovi Store for Nokia® devices, Samsung® Apps, and Nintendo® DSi Shop.Standalone applications

[0384] In some embodiments, a computer program includes a standalone application, which is a program that is run as an independent computer process, not an add-on to an existing process, e.g., not a plug-in. Standalone applications are often compiled. A compiler is a computer program(s) that transforms source code written in a programming language into binary object code such as assembly language or machine code. Suitable compiled programming languages include, by way of non-limiting examples, C, C++, Objective-C®, COBOL®, Delphi®, Eiffel®, Java®, Lisp®, Python®, Visual Basic®, and VB. NET®, or combinations thereof. Compilation is often performed, at least in part, to create an executable program. In some embodiments, a computer program includes one or more executable compiled applications. Additionally, microservices related to Python® and JavaScript® may be used.Web browser plug-ins

[0385] In some embodiments, the computer program includes a web browser plug-in (e.g., web extension, etc.). In computing, a plug-in is one or more software components that add specific functionality to a larger software application. Makers of software applications support plug-ins to enable third-party developers to create abilities which extend an application, to support easily adding new features, and to reduce the size of an application. When supported, plug-ins enable customizing the functionality of a software application. For example, plug-ins are commonly used in web browsers to play video, generate interactivity, scan for viruses, and display particular file types. Several web browser plug-ins may include Adobe Flash Player®, Microsoft Silverlight®, and Apple QuickTime®. In some embodiments, the toolbar comprises one or more web browser extensions, add-ins, or add-ons. In some embodiments, the toolbar comprises one or more explorer bars, tool bands, or desk bands.

[0386] In view of the present disclosure provided herein, several plug-in frameworks are available that enable development of plug-ins in various programming languages, including, by way of nonlimiting examples, C++, Delphi®, Java®, PHP®, Python®, and VB. NET®, or combinations thereof.

[0387] Web browsers (also called Internet browsers) are software applications, designed for use with network-connected computing devices, for retrieving, presenting, and traversing information resources on the World Wide Web. Suitable web browsers include, by way of non-limitingWSGRNo. 67054-701.601examples, Microsoft Internet Explorer®, Mozilla Firefox®, Google Chrome®, Apple Safari®, Opera Software Opera®, and KDE Konqueror®. In some embodiments, the web browser is a mobile web browser. Mobile web browsers (also called microbrowsers, mini-browsers, and wireless browsers) are designed for use on mobile computing devices including, by way of nonlimiting examples, handheld computers, tablet computers, netbook computers, subnotebook computers, smartphones, music players, personal digital assistants (PDAs), and handheld video game systems. Suitable mobile web browsers include, by way of non-limiting examples, Google Android® browser, RIM BlackBerry® Browser, Apple Safari®, Palm Blazer®, Palm WebOS® Browser, Mozilla Firefox® for mobile, Microsoft Internet Explorer Mobile®, Amazon Kindle Basic Web®, Nokia Browser®, Opera Software Opera Mobile®, and Sony PSP® browser.Software modules

[0388] In some embodiments, the platforms, systems, media, and methods disclosed herein include software, server, and / or database modules, or use of the same. In view of the present disclosure provided herein, software modules are created by techniques using machines, software, and languages. The software modules disclosed herein are implemented in a multitude of ways. In various embodiments, a software module comprises a file, a section of code, a programming object, a programming structure, or combinations thereof. In further various embodiments, a software module comprises a plurality of files, a plurality of sections of code, a plurality of programming objects, a plurality of programming structures, or combinations thereof. In various embodiments, the one or more software modules comprise, by way of non-limiting examples, a web application, a mobile application, and a standalone application. In some embodiments, software modules are in one computer program or application. In other embodiments, software modules are in more than one computer program or application. In some embodiments, software modules are hosted on one machine. In other embodiments, software modules are hosted on more than one machine. In further embodiments, software modules are hosted on a distributed computing platform such as a cloud computing platform. In some embodiments, software modules are hosted on one or more machines in one location. In other embodiments, software modules are hosted on one or more machines in more than one location.Databases

[0389] In some embodiments, the platforms, systems, media, and methods disclosed herein include one or more databases (DB), or use of the same. In view of the present disclosure provided herein, many databases are suitable for storage and retrieval data. In various embodiments, suitable databases include, by way of non-limiting examples, relational databases, non-relational databases, object oriented databases, object databases, entity-relationship model databases, associative databases, XML databases, time-series databases, graph databases, and the like.WSGRNo. 67054-701.601Further non-limiting examples include SQL, PostgreSQL®, MySQL®, Oracle®, DB2®, and Sybase. In some embodiments, a database is internet-based. In further embodiments, a database is web-based. In still further embodiments, a database is cloud computing-based. In a particular embodiment, a database is a distributed database. In other embodiments, a database is based on one or more local computer storage devices.Terms and Definitions

[0390] Unless otherwise defined, all technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this present disclosure belongs.

[0391] As used herein, the singular forms “a,” “an,” and “the” include plural references unless the context clearly dictates otherwise. Any reference to “or” herein is intended to encompass “and / or” unless otherwise stated.

[0392] As used herein, the term “about” in some cases refers to an amount that is approximately the stated amount.

[0393] As used herein, the term “about” refers to an amount that is near the stated amount by 10%, 5%, or 1%, including increments therein.

[0394] As used herein, the term “about” in reference to a percentage refers to an amount that is greater or less the stated percentage by 10%, 5%, or 1%, including increments therein.

[0395] As used herein, the phrases “at least one”, “one or more”, and “and / or” are open-ended expressions that are both conjunctive and disjunctive in operation. For example, each of the expressions “at least one of A, B and C”, “at least one of A, B, or C”, “one or more of A, B, and C”, “one or more of A, B, or C” and “A, B, and / or C” means A alone, B alone, C alone, A and B together, A and C together, B and C together, or A, B and C together.

[0396] While preferred embodiments of the present disclosure have been shown and described herein, such embodiments are provided by way of example only. It is not intended that the present disclosure be limited by the specific examples provided within the specification. While the present disclosure has been described with reference to the aforementioned specification, the descriptions and illustrations of the embodiments herein are not meant to be construed in a limiting sense. Numerous variations, changes, and substitutions may occur without departing from the present disclosure. Furthermore, it shall be understood that all aspects of the present disclosure are not limited to the specific depictions, configurations, or relative proportions set forth herein which depend upon a variety of conditions and variables. It should be understood that various alternatives to the embodiments of the present disclosure described herein may be employed in practicing the present disclosure. It is therefore contemplated that the present disclosure shall also cover any such alternatives, modifications, variations, or equivalents. It is intended that the following claimsWSGRNo. 67054-701.601define the scope of the present disclosure and that systems, methods and structures within thescope of these claims and their equivalents be covered thereby.

Claims

WSGRNo. 67054-701.601CLAIMS WHAT IS CLAIMED IS:

1. A method for training a generative artificial intelligence (AI) model, the method comprising:(a) obtaining (i) a first dataset comprising an observed data sample from a data distribution of observed data and (ii) a second dataset comprising a prior sample from a prior distribution;(b) synthesizing first data at a first time step based on the observed data sample and the prior sample;(c) synthesizing second data at a second time step based on the observed data sample and the first data, wherein the second time step is between a target time step and the first time step;(d) generating, using a neural network, a first output and a second output at the target time step, wherein (i) the first output is obtained using the first data with the target time step and the first time step as conditioning inputs and (ii) the second output is obtained using the second data with the target time step and the second time step as conditioning inputs; and (e) obtaining a trained generative AI model by updating parameters of the neural network, wherein the updating comprises at least computing a loss between a first distribution of the first output and a second distribution of the second output.

2. The method of claim 1, further comprising generating the first dataset and the second dataset.

3. The method of claim 1, wherein the observed data comprises data obtained from image data, video data, audio data, text data, sensor data, geospatial data, or metadata.

4. The method of claim 1, wherein the observed data comprises continuous data, structured data, or unstructured data.

5. The method of claim 1, wherein the prior sample comprises random noise.

6. The method of claim 1, wherein the prior distribution comprises a Gaussian distribution, a uniform random distribution, or a Dirichlet distribution.

7. The method of claim 1, wherein synthesizing the first data comprises sampling from a first marginal interpolating distribution parametrized by the observed data sample and the prior sample.

8. The method of claim 7, wherein the first marginal interpolating distribution is determined using a denoising diffusion implicit model.

9. The method of claim 1, wherein synthesizing the second data comprises sampling from a second marginal interpolating distribution parametrized by the observed data sample and the first data.WSGRNo. 67054-701.60110. The method of claim 9, wherein the second marginal interpolating distribution is determined using a denoising diffusion implicit model.

11. The method of claim 1, wherein the first output comprises a first marginal preserving interpolant between the first data and the observed data.

12. The method of claim 11, wherein the first marginal preserving interpolant preserves a marginal distribution of the first output for a plurality of time steps.

13. The method of claim 1, wherein the second output comprises a second marginal preserving interpolant between the second data and the observed data.

14. The method of claim 13, wherein the second marginal preserving interpolation preserves a marginal distribution of the second first output for a plurality of time steps.

15. The method of claim 1, wherein the first time step is sampled from a first time distribution.

16. The method of claim 15, wherein the target time step is sampled from a second time distribution, parametrized by the first time step.

17. The method of claim 16, wherein the second time step is determined using a mapping function parametrized by the first time step and the target time step.

18. The method of claim 1, wherein the computed loss is determined using maximum mean discrepancy.

19. The method of claim 18, wherein the maximum mean discrepancy is determined using a kernel function.

20. The method of claim 1, further comprising applying a stop gradient operation on the second output when computing the loss.

21. The method of claim 1, further comprising prompting the trained generative Al model with at least a request for a type of data and a category of data.

22. The method of claim 21, further comprising generating, using the trained generative AI model, new data based at least on the prompting.

23. The method of claim 22, wherein the new data is determined to be statistically similar to the observed data.

24. The method of claim 22, wherein the new data is determined to be statistically similar to the observed data with a Fréchet inception distance (FID) of less than about 2.5.

25. The method of claim 22, further comprising automatically rendering the new data on a graphical user interface (GUI), wherein the GUI comprises dynamically selectable objects or elements to modify the new data or append additional data to the new data in real time.

26. The method of claim 1, wherein the trained generative AI model comprises a stable model trained in one stage.WSGRNo. 67054-701.60127. The method of claim 1, further comprising repetitively synthesizing a third data or more data at successive time steps, based on data synthesized at a previous time step.

28. The method of claim 1, wherein the first dataset comprises a first training dataset, wherein the second dataset comprises a second training dataset, and wherein each dataset is used to obtain the trained generative Al model.

29. A method comprising:(a) training a generative artificial intelligence (AI) model in one stage, wherein the model is stable; and(b) generating, using the trained generative AI model, new data, wherein the new data comprises high fidelity data.

30. A method comprising:(a) obtaining a trained generative artificial intelligence (AI) model according to any one of claims 1 - 28;(b) prompting the trained generative AI model with at least a request for a type of data and a category of data; and(c) generating, using the trained generative AI model, new data based at least on the prompting in (b).

31. A system comprising at least one processor and instructions executable by the at least one processor to cause the at least one processor to perform operations comprising:(a) obtaining (i) a first dataset comprising an observed data sample from a data distribution of observed data and (ii) a second dataset comprising a prior sample from a prior distribution;(b) synthesizing first data at a first time step based on the observed data sample and the prior sample;(c) synthesizing second data at a second time step based on the observed data sample and the first data, wherein the second time step is between a target time step and the first time step;(d) generating, using a neural network, a first output and a second output at the target time step, wherein (i) the first output is obtained using the first data with the target time step and the first time step as conditioning inputs and (ii) the second output is obtained using the second data with the target time step and the second time step as conditioning inputs; and (e) obtaining a trained generative AI model by updating parameters of the neural network, wherein the updating comprises at least computing a loss between a first distribution of the first output and a second distribution of the second output.WSGRNo. 67054-701.60132. The system of claim 31, further comprising generating the first dataset and the second dataset.

33. The system of claim 31, wherein the observed data comprises data obtained from image data, video data, audio data, text data, sensor data, geospatial data, or metadata.

34. The system of claim 31, wherein the observed data comprises continuous data, structured data, or unstructured data.

35. The system of claim 31, wherein the prior sample comprises random noise.

36. The system of claim 31, wherein the prior distribution comprises a Gaussian distribution, a uniform random distribution, or a Dirichlet distribution.

37. The system of claim 31, wherein synthesizing the first data comprises sampling from a first marginal interpolating distribution parametrized by the observed data sample and the prior sample.

38. The system of claim 37, wherein the first marginal interpolating distribution is determined using a denoising diffusion implicit model.

39. The system of claim 31, wherein synthesizing the second data comprises sampling from a second marginal interpolating distribution parametrized by the observed data sample and the first data.

40. The system of claim 39, wherein the second marginal interpolating distribution is determined using a denoising diffusion implicit model.

41. The system of claim 31, wherein the first output comprises a first marginal preserving interpolant between the first data and the observed data.

42. The system of claim 41, wherein the first marginal preserving interpolant preserves a marginal distribution of the first output for a plurality of time steps.

43. The system of claim 31, wherein the second output comprises a second marginal preserving interpolant between the second data and the observed data.

44. The system of claim 43, wherein the second marginal preserving interpolation preserves a marginal distribution of the second first output for a plurality of time steps.

45. The system of claim 31, wherein the first time step is sampled from a first time distribution.

46. The system of claim 45, wherein the target time step is sampled from a second time distribution, parametrized by the first time step.

47. The system of claim 46, wherein the second time step is determined using a mapping function parametrized by the first time step and the target time step.

48. The system of claim 31, wherein the computed loss is determined using maximum mean discrepancy.WSGRNo. 67054-701.60149. The system of claim 48, wherein the maximum mean discrepancy is determined using a kernel function.

50. The system of claim 31, further comprising applying a stop gradient operation on the second output when computing the loss.

51. The system of claim 31, further comprising prompting the trained generative AI model with at least a request for a type of data and a category of data.

52. The system of claim 51, further comprising generating, using the trained generative AI model, new data based at least on the prompting.

53. The system of claim 52, wherein the new data is determined to be statistically similar to the observed data.

54. The system of claim 52, wherein the new data is determined to be statistically similar to the observed data with a Fréchet inception distance (FID) of less than about 2.5.

55. The system of claim 52, further comprising automatically rendering the new data on a graphical user interface (GUI), wherein the GUI comprises dynamically selectable objects or elements to modify the new data or append additional data to the new data in real time.

56. The system of claim 31, wherein the trained generative AI model comprises a stable model trained in one stage.

57. The system of claim 31, further comprising repetitively synthesizing a third data or more data at successive time steps, based on data synthesized at a previous time step.

58. The system of claim 31, wherein the first dataset comprises a first training dataset, wherein the second dataset comprises a second training dataset, and wherein each dataset is used to obtain the trained generative Al model.

59. A computer program product, the computer program product comprising at least one non-transitory computer-readable medium having computer-readable program code portions embodied therein, the computer-readable program code portions comprising:an executable portion configured to obtain (i) a first dataset comprising an observed data sample from a data distribution of observed data and (ii) a second dataset comprising a prior sample from a prior distribution;an executable portion configured to synthesize first data at a first time step based on the observed data sample and the prior sample;an executable portion configured to synthesize second data at a second time step based on the observed data sample and the first data, wherein the second time step is between a target time step and the first time step;an executable portion configured to generate, using a neural network, a first output and a second output at the target time step, wherein (i) the first output is obtained using the firstWSGRNo. 67054-701.601data with the target time step and the first time step as conditioning inputs and (ii) the second output is obtained using the second data with the target time step and the second time step as conditioning inputs; andan executable portion configured to obtain a trained generative AI model by updating parameters of the neural network, wherein the updating comprises at least computing a loss between a first distribution of the first output and a second distribution of the second output.

Images