Dynamic data pipeline for training generative neural networks

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for fine-tuning a generative neural network. For example, the system can fine-tune the generative neural network to more effectively generate data items that have a target property.

Description

[0001] Atorney Docket No. 45288-0573WO1

[0002] DYNAMIC DATA PIPELINE FOR TRAINING GENERATIVE NEURAL NETWORKS

[0003] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 63 / 743,197, filed on January 8, 2025. The disclosure of the prior application is considered part of and is incorporated by reference in its entirety in the disclosure of this application.

[0004] BACKGROUND

[0005] This specification relates to training neural networks to generate data items. For example, the data items can include text data, image data, video data, audio data, or the like.

[0006] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more intermediate layers in addition to a final layer. The output of each intermediate layer is used as input to the next layer in the network, i.e., the next intermediate layer or the final layer. Each layer of the network generates an output from a received input in accordance with current value inputs of a respective set of parameters.

[0007] SUMMARY

[0008] This specification describes a training system implemented as computer programs on one or more computers in one or more locations that trains a generative neural network to generate an output data item conditioned on a conditioning input.

[0009] More specifically, this specification describes how a training system can fine-tune the generative neural network, e.g., a diffusion neural network, to improve the performance of the generative neural network in accurately generating output data items in response to conditioning inputs that specify respective target values for a particular target property. For example, the data items can include image data, video data, or audio data.

[0010] Particular embodiments of the subject matter described in this specification can be implemented so as to realize one or more of the following advantages.

[0011] Training data for continuously improving the performance of an already trained generative neural network can be generated automatically by making use of the described techniques. By identifying conditioning inputs which the generative neural network did not perform well on when generating new data items, e.g., because the generated data items doAtorney Docket No. 45288-0573WO1

[0012] not accurately reflect the target values for the target properties specified in the conditioning inputs, and then generating training data based on these identified conditioning inputs for the further training of the generative neural network, the described techniques provide an effective solution to the alignment problem of generative neural networks.

[0013] In the technical field of generative machine learning, “alignment” refers to the process of adjusting the model’s generation process to better match human preferences or a specific target distribution, thereby ensuring that the output data items generated by the model are more closely aligned with what is considered desirable or relevant based on given criteria, e.g., that are specified in a conditioning input.

[0014] Generating new data items, e.g., images or other data items having higher dimensionalities, can be computationally expensive. An inadequately aligned generative neural network often requires consuming a significant amount of computing resource to generate a large number of candidate data items before arriving at a final data item that matches human preferences, e.g., that has the desired content or that satisfies some objective criteria.

[0015] In contrast, the generative neural network trained using the techniques described in this specification can avoid this by having an improved performance that enables it to generate data items that each better match human preferences, technical expectations, or a specific target distribution, thereby decreasing computing resources usage, e.g., processing power and memory consumption, at inference time.

[0016] As a particular example, a system can use the described techniques to continuously adjust the parameters of the generative neural network whenever semantic consistency issues hinder the performance of the generative neural network. For example, a semantic consistency issue occurs when data items generated by the generative neural network from a same conditioning input reflect varying or inconsistent values for a same target property, or when data items generated by the generative neural network from a first conditioning input inadvertently reflect a second value for a target property that is not specified in the first conditioning input but rather, in a second, different conditioning input. In response, the system can identify conditioning inputs that cause the semantic consistency issues and then specifically train the generative neural network on training examples automatically generated based on those identified conditioning inputs, such that the values of the parameters of theAtorney Docket No. 45288-0573WO1

[0017] generative neural network can be adjusted to improve semantic consistency while reducing entanglement across the generated data items.

[0018] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

[0019] BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 shows an example training system and an example data generation system. FIG. 2 is a diagram of an example data flow illustrating operations performed by a training system.

[0020] FIG. 3 is a flow diagram of an example process for fine-tuning a generative neural network.

[0021] Like reference numbers and designations in the various drawings indicate like elements.

[0022] DETAILED DESCRIPTION FIG. 1 shows an example training system 100 and an example data generation system 150.

[0023] The training system 100 and the data generation system 150 are examples of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

[0024] The training system 100 trains a generative neural network 120.

[0025] After the training, the data generation system 150 can generate new output data items 104 conditioned on new conditioning inputs 101 using the generative neural network 120.

[0026] That is, the training system 100 outputs data specifying the trained generative neural network 120, e.g., data specifying trained values of the parameters of the generative neural network 120, to the data generation system 150 — and the data generation system 150 uses the generative neural network 120 to perform conditional data generation.

[0027] Additionally or alternatively, the training system 100 outputs data specifying the trained generative neural network 120, e.g., data specifying trained values of the parameters of the generative neural network 120, for storage at one or more storage devices.Atorney Docket No. 45288-0573WO1

[0028] This specification generally describes the generative neural network 120 being a diffusion neural network, and the data generation process executed using the generative neural network 120 being a reverse diffusion process.

[0029] More generally, however, the generative neural network 120 can be any appropriate generative neural network that can map a conditioning input to an output data item, e.g., a denoising neural network, an auto-regressive generative neural network, a non-auto-regressive masked token generation neural network, a normalizing flows model, the generator of a generative adversarial neural network, and so on.

[0030] That is, the diffusion neural network mentioned in this specification should be understood to refer to any type of generative neural network that can be used to denoise a noisy representation of a data item using any of a variety of denoising frameworks, e.g., reverse diffusion, unmasking, flow matching, multi-step consistency, and so on, and the reverse diffusion process can refer to any appropriate denoising process that can map the noisy representation of a data item to a final data item.

[0031] More specifically, the training system 100 can fine-tune the generative neural network 120, e.g., the diffusion neural network, to improve the performance of the generative neural network 120 in accurately generating output data items 104 in response to conditioning inputs 101 that specify respective target values for a particular target property.

[0032] That is, the training system 100 fine-tunes, i.e., further trains, a pre-trained generative neural network 110 so that the generative neural network 120 can accurately generate an output data item 104 that has a target value of a particular target property that is specified in the conditioning input 101.

[0033] For example, when the output data items are images, the target property can be rendered content within the image and the target value of the target property specifies a particular item of content to be rendered within the output image. Thus, the training system 100 trains the generative neural network 120 to accurately generate output images that accurately depict specific items of content that are described by the conditioning input 101.

[0034] As one example of this, the target property can be a rendered graphic and the target value of the target property specifies a particular graphic to be rendered within the output image. For example, the particular graphic can include a depiction of an object. CommonAtorney Docket No. 45288-0573WO1

[0035] examples of objects include landmarks, landscape or location features, vehicles, tools, food, clothing, devices, animals, human, to name just a few.

[0036] As another example of this, the target property can be rendered text and the target value of the target property specifies a particular sequence of text in some natural language to be rendered within the output image. Thus, the training system 100 trains the generative neural network to accurately generate output images that include accurately rendered text, i.e., legible text that matches text specified in the conditioning input 101.

[0037] Other examples of conditioning inputs and data items are described below.

[0038] Thus, as described above, the training system 100 performs “fine-tuning,” i.e., further training, of the pre-trained generative neural network 110 to improve the performance of the neural network in accurately generating output dada items that have values of a particular property that match a value for the property that is specified in the conditioning inputs.

[0039] In other words, prior to being trained as described in this specification, the training system 100 or another training system has already trained the diffusion neural network.

[0040] In general the diffusion neural network can have been trained using any diffusion model objective. As one example, the diffusion neural network can have been trained on a set of training data items on a diffusion score matching objective or a variant thereof.

[0041] As a result of this training, the diffusion neural network can generate high-quality data items, e.g., high-quality images, videos, or audios, but may have difficulty in accurately and consistently aligning the final output data items with the corresponding conditioning inputs when the conditioning inputs request that the output data items have specific values for the target properties.

[0042] For example, when the output data items are images, the diffusion neural network may be able to generate high-quality images with good aesthetics, but may not be able to consistently accurately render text that is specified by the conditioning input, e.g., may generate text that is illegible or that does not match exactly the text that is specified in the conditioning input. This limits the ability of the data generation system 150 to apply the diffusion neural network to use cases that frequently require generating such output images, e.g., that require generating images with accurately rendered text.

[0043] As another example, the diffusion neural network may be able to generate high-quality images with good aesthetics, but may not be able to consistently accurately maintainAtorney Docket No. 45288-0573WO1

[0044] consistency across the images that are generated based on conditioning inputs specifying the same or similar target values for the same target property. This negatively impacts user experience with the system that deploys the diffusion neural network to generate images in response to user requests.

[0045] For example, the images generated by the diffusion neural network from the same or similar conditioning inputs that specify a same target value of a target property may nevertheless reflect varying or inconsistent values for the same target property.

[0046] As a particular example of this, not all generated images may show graphic content that matches a graphic type corresponding to the text included in the conditioning inputs. A graphic type is defined by graphic attributes such as color tones, layouts, and so on.

[0047] As another example, the images generated by the diffusion neural network from the first conditioning inputs that specify a first value for a target property may inadvertently reflect a second value for the target property that is not specified in the first conditioning inputs but rather, in second conditioning inputs.

[0048] As a particular example of this, the generated images may show graphic content that matches a graphic type corresponding to the text excluded from the conditioning inputs while lacking the graphic content that matches a graphic type corresponding to the text actually included in the conditioning inputs.

[0049] The diffusion neural network can be any appropriate diffusion neural network that is configured to receive an input that includes a current (noisy) representation of a data item and a conditioning input and to generate a denoising output.

[0050] In some implementations, the diffusion neural network performs a reverse diffusion process in output space, e.g., pixel space when the data items are images. In this example, when the data items are images, the data items (“representations”) operated on and generated by the diffusion neural network have values for each pixel that specify color values, e.g., RGB values or another color encoding scheme.

[0051] Examples of such diffusion neural networks include Imagen, as described in arXiv:2205.11487.

[0052] In some other implementations, the diffusion neural network performs a reverse diffusion process in latent space, e.g., in a latent space that is lower-dimensional than the output space. That is, the data items (“representations”) operated on by the diffusion neuralAtorney Docket No. 45288-0573WO1

[0053] network are latent representations and the values in the representations are learned, latent values, e.g., rather than color values when the data items are images.

[0054] Examples of such diffusion neural networks include MobileDiffusion, as described in arxiv:2311.16567.

[0055] In these implementations, during training, the diffusion neural network can be associated with an encoder to encode training data items into the latent space and, after training and to generate new output data items, a decoder neural network that receives an input that includes a latent representation of a data item and decodes the latent representation to reconstruct the data item.

[0056] The diffusion neural network can have any appropriate architecture that allows the neural network to map a diffusion input that includes an input data item that has the same dimensionality as the output data item to a denoising output that also has the same dimensionality as the output data item.

[0057] For example, when the output data item is an audio signal or an image, the diffusion neural network can be a convolutional neural network, e.g., a U-Net or other architecture that maps one input of a given dimensionality to an output of the same dimensionality.

[0058] As another example, the diffusion neural network can be a Transformer neural network that processes the diffusion input through a set of self-attention layers to generate the denoising output.

[0059] The diffusion neural network can be conditioned on the conditioning input in any of a variety of ways.

[0060] As one example, a system (either the training system 100 or the data generation system 150) can use an encoder neural network to generate one or more embeddings that represent the conditioning input and the diffusion neural network can include one or more cross-attention layers that each cross-attend into the one or more embeddings.

[0061] An embedding, as used in this specification, is an ordered collection of numerical values, e.g., a vector of floating point values or other types of values.

[0062] For example, when the conditioning input is text, the system can use a text encoder neural network, e.g., a Transformer neural network, to generate a fixed or variable number of text embeddings that represent the conditioning input.Atorney Docket No. 45288-0573WO1

[0063] When the conditioning input is an image, the system can use an image encoder neural network, e.g., a convolutional neural network or a vision Transformer neural network, to generate a set of embeddings that represent the image.

[0064] When the conditioning input is audio, the system can use, e.g., an audio encoder neural network, e.g., an audio encoder neural network that has been trained jointly with a decoder neural network as part of a neural audio codec, to generate one or more embeddings that encode the audio.

[0065] When the conditioning input is a scalar value, the system can use, e.g., an embedding matrix to map the scalar value or a one-hot representation of the scalar value to an embedding.

[0066] In some cases, the conditioning input includes multiple different types of inputs, e.g., two or more of text, images, bound values, or context embeddings.

[0067] In some of these cases, the system can generate one or more initial embeddings for each of the different types of inputs, i.e., using an appropriate encoder neural network as described above, and then process the initial embeddings for all of the different types of inputs using a Transformer encoder neural network to update each of the initial embeddings to generate a set of final embeddings. The one or more cross-attention layers within the diffusion neural network can then cross-attend into the set of final embeddings.

[0068] In others of these cases, different cross-attention layers within the diffusion neural network can cross-attend into embeddings of different types of conditioning inputs.

[0069] In yet others of these cases, the system can concatenate the initial embeddings of the different types of inputs along the sequence dimension and then the one or more crossattention layers can cross-attend into the concatenated set of final embeddings.

[0070] As another example, the diffusion neural network can include one or more other types of neural network layers that are conditioned on the one or more embeddings. Examples of such layers include Feature-wise Linear Modulation (FiLM) layers, layers with conditional gated activation functions, and so on.

[0071] The diffusion input at any given updating iteration can also include data defining a noise level for the iteration. Generally, each updating iteration has a corresponding time step t and the noise level for the iteration depends on the time step. For example, the noise level can be a decreasing function of the time step t. Examples of such functions include a linearAtorney Docket No. 45288-0573WO1

[0072] function, a cosine function, and a sigmoid function. In these cases, data identifying the noise level, the time step, or both can be embedded using an appropriate neural network, e.g., a multi-layer perceptron (MLP) and used to condition the diffusion neural network as described above for the conditioning input.

[0073] The training system 100 fine-tunes the generative neural network 120 to adjust the values of at least some of the parameters of the generative neural network 120 on a training dataset 160 based on optimizing a preference learning objective.

[0074] For example, the preference learning objective can be a supervised objective that, for each training example that includes multiple data items, is based on which data item in the training example is preferred among the multiple data items as a response to a conditioning input.

[0075] In particular, the training dataset 160 is an automatically generated training dataset that includes a plurality of training examples generated by the training system 100 in an automatic manner based on historic conditioning inputs and by using a first generative neural network 130 and a second generative neural network 140. The first and second generative neural networks 130, 140 can each be the same as or different from the generative neural network 120 being trained.

[0076] Some examples of data items and conditioning inputs now follow.

[0077] Generally, the conditioning input characterizes one or more desired properties for the data item, i.e., characterizes one or more properties that the final data item generated by the generative neural network 120 should have.

[0078] As mentioned above the generative neural network 120 can be configured to generate any of a variety of output data items conditioned on any of a variety of conditioning inputs.

[0079] For example, the generative neural network 120 can be configured to generate audio data, e.g., a waveform of audio or a spectrogram, e.g., a mel-spectrogram or a spectrogram where the frequencies are in a different scale, of the audio.

[0080] In this example, the conditioning input can be text or features of text that the audio should represent, i.e., so that the generative neural network 120 serves as a text-to-speech machine learning model that converts text or features of the text to audio data for an utterance of the text being spoken.Atorney Docket No. 45288-0573WO1

[0081] As another example, the conditioning input can identify a desired speaker for the audio, i.e., so that the generative neural network 120 generates audio data that represents speech by the desired speaker.

[0082] As another example, the conditioning input can characterize properties of a song or other piece of music, e.g., lyrics, genre, and so on, so that the generative neural network 120 generates a piece of music that has the properties characterized by the conditioning input.

[0083] As another example, the conditioning input can specify a classification for the audio data into a class from a set of possible classes, so that the generative neural network 120 generates audio data that belongs to the class. For example, the classes can represent types of musical instruments or other audio emitting devices, i.e., so that the generative neural network 120 generates audio that is emitted by the corresponding class, or types of animals, i.e., so that the generative neural network 120 generates audio that represents noises generated by the corresponding animal, and so on.

[0084] As another particular example, the data item can be an image, such that the generative neural network 120 can perform conditional image generation by generating the intensity values of the pixels of the image. In general the conditioning input can specify one or more characteristics for the image.

[0085] In this particular example, the conditioning input can be a sequence of text and the output data item can be an image that describes the text, i.e., the conditioning input can be a caption for the output image.

[0086] As yet another particular example, the conditioning input can be an object detection input that specifies one or more bounding boxes and, optionally, a respective type of object that should be depicted in each bounding box.

[0087] As yet another particular example, the conditioning input can specify an object class from a plurality of object classes to which an object depicted in the output image should belong.

[0088] As another example, the conditioning input can specify one or more images.

[0089] For example, the conditioning input can specify an image at a first resolution and the output data item can include the image at a second, higher resolution.

[0090] For example, the conditioning input can specify an image and the output data item can comprise a de-noised, enhanced, stylized, or otherwise edited version of the image.Atorney Docket No. 45288-0573WO1

[0091] As yet another particular example, the conditioning input can specify an image including a target entity for detection, e.g. a tumor, and the output data item can comprise the image without the target entity, e.g., to facilitate detection of the target entity by comparing the images.

[0092] As yet another particular example, the conditioning input can be a segmentation that assigns each of a plurality of pixels of the output image to a category from a set of categories, e.g., that assigns to each pixel a respective one of the category.

[0093] As yet another example, the conditioning input can be a different type of structured input, e.g., a mesh or a graph that specifies properties of the image to be generated.

[0094] More generally, the conditioning input can include one or more different types of inputs of one or more different modalities, e.g., only text, only one or more images, both text and one or more images, and so on.

[0095] As yet another example, the output data item can be a video. Again the conditioning input can specify one or more characteristics for the video.

[0096] As a particular example, the conditioning input can include text and the output data item can be a video described by the text.

[0097] As yet another particular example, the conditioning input can include one or more images and the output data item can be a video that completes the one or images, e.g., video starting from the one or more images.

[0098] More generally, the task of generating the output data item can be any task that outputs continuous data conditioned on a conditioning input. For example, the output can be an output of a different sensor, e.g., a lidar point cloud, a radar point cloud, an electrocardiogram reading, and so on, and the conditioning input can represent the type of data that should be measured by the sensor. Where a discrete output is desired this can be obtained, e.g., by thresholding the outputs generated by the diffusion neural network.

[0099] In some applications, the output data item can be used in a control task to control an action of a mechanical agent acting in a real-world environment to perform a mechanical task. For example, the output data item can be processed by a policy neural network of the agent to select one or more actions to be performed by the agent as part of the task. The agent may then perform the one or more actions. The output data item (e.g., image) can, for example, characterize a state of the real-world environment that is predicted to be obtainedAtorney Docket No. 45288-0573WO1

[0100] by the agent performing the one or more actions. The conditioning input can, e g., specify a state of the real-world environment and the one or more actions. As another example the conditioning input can specify a state of the real-world environment and the output data item can be used to select one or more actions to be performed by the mechanical agent to perform a task (i.e. the diffusion neural network can represent an action selection policy).

[0101] FIG. 2 is a diagram of an example data flow 200 illustrating operations performed by the training system 100 to generate training examples for inclusion in the training dataset 160. Operations described below can be repeated any number of times to generate any number of training examples.

[0102] The training system 100 obtains data that identifies a subset of historic conditioning inputs stored in a conditioning input log 115.

[0103] The conditioning input log 115 can be any single data store or any combination of data stores that stores conditioning inputs previously submitted by various users into the data generation system 150 or another data generation system.

[0104] In some implementations, the conditioning input log 115 also stores the data items generated by the first generative neural network 130, which can be the same as or different from the generative neural network 120 being trained, from these conditioning inputs. In some implementations, the conditioning input log 115 also stores data identifying those data items rather than the data items themselves.

[0105] The identified subset of historic conditioning inputs includes those that the first generative neural network 130 did not perform well on when generating corresponding data items.

[0106] FIG. 2 thus illustrates that the training system 100 obtains data that identifies one or more first conditioning inputs 131 and one or more second conditioning inputs 132 from the conditioning input log 115. How the system can obtain such data will be described below with reference to step 302 of FIG. 3.

[0107] Each first conditioning input 131 specifies a first target value of a target property for a data item. Each second conditioning input 132 specifies a second target value of the target property for a data item. The first target value is different from the second target value.

[0108] For example, when the output data items are images, the target property can be rendered content (e g., a vehicle) within the image, and the first target value of the targetAtorney Docket No. 45288-0573WO1

[0109] property specifies a first item of content (e.g., a blue vehicle) to be rendered within the image, while and the second target value of the target property specifies a second item of content (e.g., a red vehicle) to be rendered within the image.

[0110] Upon receiving each of the first and second conditioning inputs, the first generative neural network 130 has generated one or more corresponding data items based on the conditioning input, e.g., the one or more data items 133 based on each first conditioning input 131, and the one or more data items 134 based on each second conditioning input 132.

[0111] Not having performed well when generating corresponding data items means that, the data items 133, 134 that have been generated by the first generative neural network 130 fail to satisfy one or more similarity criteria, e.g., one or more semantic similarity criteria, with respect to the first and second conditioning inputs 131.

[0112] The one or more similarity criteria (e.g., semantic similarity criteria) require a first similarity between one or more first conditioning inputs 131 and one or more data items 133 generated based on the one or more first conditioning inputs 131 to be greater than a second similarity between the one or more second conditioning inputs 132 and the one or more data items 133 generated based on the one or more first conditioning inputs 131.

[0113] That is, the one or more similarity criteria specifies that the data items 133 generated based on the one or more first conditioning inputs 131 should be closer to, e.g., semantically closer to, the one or more first conditioning inputs 131 than the one or more second conditioning inputs 132, and not the other way around.

[0114] Thus, in FIG. 2, a first similarity between the one or more first conditioning inputs 131 and one or more data items 133 that have been generated based on the one or more first conditioning inputs 131 is less than a second similarity between the one or more second conditioning inputs 132 and the one or more data items 133 that have been generated based on the one or more first conditioning inputs 131.

[0115] For example, this could happen when the one or more data items 133 do not reflect the first target value specified in the first conditioning inputs 131 for the target property; instead, they reflect the second target value specified in the second conditioning inputs 132 for the target property.

[0116] That is, the data items 133, 134 generated by the first generative neural network 130 are entangled. In generative models, entanglement refers to a state where multiple distinctAtorney Docket No. 45288-0573WO1

[0117] values for the same target property are bundled together in the model’s internal representation embedded in its parameter values. This is problematic in many scenarios because it fundamentally limits its capability to accurately perform conditional data generation.

[0118] There are many ways in which the similarity between a conditioning input and a data item can be determined.

[0119] For example, when the conditioning input is text and the output data items are images or videos, the similarity can be determined as a text-to-image similarity. Such a similarity could be based on text embeddings and image embeddings generated by (an intermediate or final layer of) a multi-modal neural network.

[0120] Examples of the text-to-image similarity include the ALIGN similarity as described in Jia, Chao, et al. Scaling up visual and vision-language representation learning with noisy text supervision. International conference on machine learning. PMLR, 2021, the Siglip similarity as described in Zhai, Xiaohua, et al. Sigmoid loss for language image pre-training. Proceedings of the IEEE / CVF international conference on computer vision. 2023, and the Siglip 2 similarity as described in Tschannen, Michael, et al. Siglip 2: Multilingual visionlanguage encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786 (2025).

[0121] As another example, when the conditioning input specifies content to be rendered in an image and the output data items are images, the similarity can be determined based on performing object detection (when the content is an object) optical character recognition (OCR) (when the content is a text sequence) on each image to detect objects or text in the image and then determine whether the detected obj ect or text matches the obj ect or text sequence specified in the conditioning input.

[0122] As a particular example, when the conditioning input specifies a text sequence to be rendered in an image and the output data items are images, the similarity can be determined as an edit distance between the detected text and the specified text sequence.

[0123] The training system 100 can use any appropriate object detection or OCR technique, e.g., one that uses a neural network or one that performs object detection or OCR using statistical image analysis techniques.Atorney Docket No. 45288-0573WO1

[0124] As another example, when the conditioning input is an image and the output data items are images or videos, the similarity can be determined as an image-to-image similarity. Such a similarity could be based on image embeddings generated by (an intermediate or final layer of) an image processing neural network, which can be a convolutional neural network or a vision Transformer (ViT), for example.

[0125] Examples of the image-to-image similarity include perceptual similarity, e.g., learned perceptual image patch similarity (LPIPS), and style similarity.

[0126] As another example, when the conditioning input is text and the output data items are audios, the similarity can be determined as a text-to-audio similarity. Such a similarity could be based on text embeddings and audio embeddings generated by (an intermediate or final layer of) a multi-modal neural network.

[0127] Examples of the text-to-audio similarity include the MuLan similarity as described in Huang, Qingqing, et al. Mulan: A joint embedding of music audio and natural language. arXiv preprint ar Xiv:2208.12415 (2022).

[0128] The training system 100 generates one or more first data items 143 based on the one or more first conditioning inputs 131. Each first data item 143 accurately reflects the first target value of the target property specified in a first conditioning input 131. Each first data item 143 thus represents a modification, e.g., an improvement or correction, to the data item 133 previously generated by the first generative neural network 130 based on the same first conditioning input 131.

[0129] As a result, a similarity (e.g., a semantic similarity) between the one or more first conditioning inputs 131 and the one or more first data items 143 is greater than the first similarity between the one or more first conditioning inputs 131 and the one or more data item 133 and, in some cases, greater than the second similarity between the one or more second conditioning inputs 132 and the one or more data items 133 that have been generated based on the one or more first conditioning inputs 131.

[0130] There are many ways in which the one or more first data items 143 can be generated. For example, the training system 100 can do this by modifying the previously generated data items 133 to replace the incorrectly generated second target value for the target property with the first target value.Atorney Docket No. 45288-0573WO1

[0131] For example, when the conditioning input specifies a target graphic to be rendered in an image and the output data items are images, the training system 100 can perform inpainting between the target graphic and a modified first output data item that excludes a portion of the first output data item where the incorrectly rendered target graphic appears.

[0132] As a particular example, the training system 100 can perform image in-painting by providing, to an image in-painting model, an input that includes the target graphic, the first output data item, and a mask or other data that identifies the portion of the first output data item where the incorrectly rendered target graphic appears.

[0133] As an analogous example, the training system 100 can perform video in-painting by using a video in-painting model when the output data items are videos, or audio in-filling by using an audio in-filling model when the output data items are audios.

[0134] As another example, the training system 100 can do this by incorporating model performance enhancement technique when performing conditional data item generation using the second generative neural network 140 to improve the likelihood that the first data items 143 reflect the first target value of the target property relative to the data items 133 previously generated by the generative neural network 120.

[0135] The second generative neural network 140 can be the generative neural network 120 that is being fine-tuned or can be a different, already-trained generative neural network, e.g., a diffusion neural network or a multi-modal auto-regressive neural network.

[0136] An example of the model performance enhancement technique is the instructiontuning technique.

[0137] Instruction-tuning involves training the second generative neural network 140 through supervised fine-tuning (SFT) on a plurality of pairs that each include a conditioning input and an output data item, where each output data item accurately reflects the first target value of the target property specified in the corresponding conditioning input, and then using the trained, i.e., instruction tuned, second generative neural network 140 to generate the one or more first data items 143 based on processing the one or more first conditioning inputs 131.

[0138] Another example of the model performance enhancement technique is few-shot prompting.Atorney Docket No. 45288-0573WO1

[0139] Few-shot prompting involves providing, to the second generative neural network 140, an input that includes a few, e.g., 2-5, examples, in addition to the first conditioning input 131, then processing the input using the second generative neural network 140 to generate the one or more first data items 143.

[0140] Each example includes a pair of conditioning input and output data item, where each output data item accurately reflects the first target value of the target property specified in the corresponding conditioning input.

[0141] As a particular example, the training system 100 can generate an input that includes a data item modification instruction derived from the first conditioning input 131, a data item 133 generated by the first generative neural network 130 based on the first conditioning input 131, and a few examples, and process the input using the second generative neural network 140 to generate a first data item 143 that is a modified version of the data item 133.

[0142] That is, the system can prompt the second generative neural network 140 to cause it to modify the one or more previously generated data items 133 to generate one or more one or more first data items 143 that have the first target value for the target property.

[0143] Likewise, the training system 100 generates one or more second data items 144 based on the one or more second conditioning inputs 132. Each second data item 144 accurately reflects the second target value of the target property specified in a second conditioning input 132. The training system 100 can generate the one or more second data items 144 in ways similar to how the one on more first data items 132 can be generated, as discussed above.

[0144] Each second data item 144 thus represents a modification, e.g., an improvement or correction, to the data item 134 previously generated by the first generative neural network 130 based on the same second conditioning input 132.

[0145] As a result, a similarity (e.g., a semantic similarity) between the one or more second conditioning inputs 132 and the one or more second data items 144 is greater than a similarity between the one or more second conditioning inputs 132 and the one or more data items 134 and, in some cases, greater than a similarity between the one or more first conditioning inputs 131 and the one or more data items 134 that have been generated based on the one or more second conditioning inputs 132.

[0146] The training system 100 generates one or more first training examples 150.Atorney Docket No. 45288-0573WO1

[0147] Each first training example 150 includes a first conditioning input 131 , a first data item 143, a second data item 144, and preference data 146 indicating that the first data item 143 is preferred over the second data item 144 given the first conditioning input 131, i.e., because the first data item 143 has the first target value of the target property while the second data item 144 does not.

[0148] In some implementations, each first training example 150 can include data derived from the first conditioning input 131 in place of the first conditioning input 131. For example, each first training example 150 can include a truncated, expanded, or otherwise modified version of the first conditioning input 131 that specifies the same, first target value of the same, target property for the data item.

[0149] Optionally, the training system 100 also generate one or more second training examples.

[0150] Each second training example includes a second conditioning input 132 or data derived from the second conditioning input 132, a first data item 143, and a second data item 144, and preference data indicating that the second data item 144 is preferred over the first data item 143 given the second conditioning input 132, i.e., because the second data item 144 has the second target value of the target property while the first data item 143 does not.

[0151] The training system 100 stores the one or more first training examples 150 and, when generated, the one or more second training examples in the training dataset 160.

[0152] The preference data 146 facilitates the use of a supervised objective function to finetune the generative neural network 120 even in cases where no ground-truth preference data is available to the training system 100.

[0153] Leveraging the automatically generated preference data and the supervised objective, the training system 100 can then train the generative neural network 120 from being merely capable of generating data items to being aligned with specific human preferences or technical expectations, and in particular, can train the generative neural network 120 to generate disentangled data items with an improved objective quality measure, all while consuming fewer computing resources (e.g., fewer processing resources, storage resources, memory resources, network resources, and so on) than conventional systems that rely on human annotations or an unsupervised or self-supervised objective function.Atorney Docket No. 45288-0573WO1

[0154] FIG. 3 is a flow diagram of an example process 300 for fine-tuning a generative neural network. For convenience, the process 300 will be described as being performed by a system of one or more computers located in one or more locations. For example, a training system, e.g., the training system 100 depicted in FIG. 1, appropriately programmed in accordance with this specification, can perform the process 300.

[0155] The system obtains data indicating that data items generated by a first generative neural network from one or more first conditioning inputs and one or more second conditioning inputs fail to satisfy one or more similarity criteria (step 302), e g., one or more semantic similarity criteria. The system will generally obtain different data that references different pairs of conditioning inputs at different iterations of step 302.

[0156] Each first conditioning input specifies a first target value of a target property for a data item. Each second conditioning input specifies a second target value of the target property for a data item. The first target value is different from the second target value.

[0157] As described above, the first generative neural network can be the generative neural network that is being fine-tuned or can be a different, already-trained generative neural network, e.g., a diffusion neural network or a multi-modal auto-regressive neural network.

[0158] The one or more similarity criteria (e.g., semantic similarity criteria) require a first similarity (e.g., a first similarity) between the one or more first conditioning inputs and data items generated based on the one or more first conditioning inputs to be greater than a second similarity (e g., a second similarity )between the one or more second conditioning inputs and the data items generated based on the one or more first conditioning inputs.

[0159] There are many ways in which the system can obtain such data.

[0160] In some implementations, the system performs an extensive pairwise comparison across the data items generated by the first generative neural network from historic conditioning inputs stored in the conditioning input log based on a similarity (e.g., a semantic similarity), which can be any one of the similarities mentioned above with reference to FIG.

[0161] 2 or another suitable similarity (e.g., any suitable semantic similarity), to select the data items that fail to satisfy the one or more similarity criteria (e.g. the one or more semantic similarity criteria).

[0162] As a particular example, the system can perform an exhaustive pairwise comparison by, for each historic conditioning input, comparing a similarity between a data itemAtorney Docket No. 45288-0573WO1

[0163] generated based on the historic conditioning input and the historic conditioning input, to a similarity between the data item and each other historic conditioning input stored in the conditioning input log.

[0164] In some implementations, the system leverages data received through a feedback loop to perform a more focused, i.e., less extensive, pairwise comparison across the data items generated by the first generative neural network from historic conditioning inputs stored in the conditioning input log.

[0165] For example, the system can receive, from a user who submitted the first and second conditioning inputs, user feedback data indicating that the data item generated based on the first conditioning input is, in fact, semantically closer to the second conditioning input and rather than the first. That is, the user feedback data indicates that the data item is entangled in view of the first and second conditioning inputs.

[0166] As another example, the system can receive, from an evaluator model, automated feedback data indicating that the data item generated based on the first conditioning input is, in fact, closer to (e.g., semantically closer to) the second conditioning input and rather than the first. For example, the evaluator model can be a multi-modal neural network that has been trained on training data that includes training data items associated with human annotation data.

[0167] As yet another example, the system can receive an input that identifies a target property that, when included in conditioning inputs, will result in the generation of entangled data items by the first generative neural network. The system can then identify the first and second conditioning inputs by using the identified target property as a search query to search the conditioning input log.

[0168] In any example, the system could compare a similarity (e.g., a semantic similarity) between a data item generated based on a first conditioning input and the first conditioning input, to a similarity (e.g., a semantic similarity) between the data item and a second conditioning input only when feedback data indicating entanglement in the data item is received.

[0169] Leveraging the feedback data enables the system to conserve computing resources since an exhaustive pairwise comparison can be avoided.Atorney Docket No. 45288-0573WO1

[0170] The system generates one or more first data items based on each of the one or more first conditioning inputs (step 304). Each first data item accurately reflects the first target value of the target property specified in the first conditioning input.

[0171] The system generates one or more second data items based on each of the one or more second conditioning inputs (step 306). Each second data item accurately reflects the second target value of the target property specified in the second conditioning input.

[0172] A few example ways in which the one or more first and second data items can be generated are described above with reference to FIG. 2.

[0173] The system generates one or more first training examples (step 308). Each first training example includes a first conditioning input (i.e., one of the first conditioning inputs referenced in the data obtained at step 302) or data derived from the first conditioning input, a first data item (i.e., one of the first data items generated at step 304), a second data item (i.e., one of the second data items generated at step 306), and preference data indicating that the first data item is preferred over the second data item, i.e., because the first data item has the first target value of the target property while the second data item does not.

[0174] Optionally, the system also generates one or more second training examples. Each second training example includes a second conditioning input (i.e., one of the second conditioning inputs referenced in the data obtained at step 302) or data derived from the second conditioning input, a first data item (i.e., one of the first data items generated at step 304), and a second data item (i.e., one of the second data items generated at step 306), and preference data indicating that the second data item is preferred over the first data item given the second conditioning input, i.e., because the second data item has the second target value of the target property while the first data item does not.

[0175] Steps 302-308 of process 300 can be repeated any number of times before proceeding to step 310.

[0176] The system trains the generative neural network on training data (step 310).

[0177] The training data can include all of the one or more first training examples and the one or more optional, second training examples generated as a result of the repeated iterations of steps 302-308. The training data can also optionally include some or all of the training examples that were used to train the pre-trained generative neural network.Atorney Docket No. 45288-0573WO1

[0178] For example, the system can train the generative neural network to adjust the values of at least some of the parameters of the generative neural network based on optimizing a preference learning objective.

[0179] For example, the preference learning objective can be a supervised objective that, for each training example, is based on which data item in the training example is preferred among the multiple data items, i.e., as indicated by the preference data.

[0180] One example of such an objective is the direct preference optimization (DPO) objective. Another example is the identity preference optimization (IPO) objective. Another example is the odds ratio preference optimization (ORPO) objective.

[0181] After the training, the system or another data generation system can use the fine-tuned generative neural network as the final neural network to be used to generate data items.

[0182] As another example, to preserve the pre-trained capability of the generative neural network while maintaining the improvements resulting from the fine-tuning, the system can generate a final generative neural network by combining the first trained values of the parameters of the generative neural network, i.e., the parameters after the fine-tuning is complete, with pre-trained values of the parameters determined from the pre-training. For example, the system can determine a “model soup” by computing a weighted combination of the first trained values and the pre-trained values.

[0183] In this specification, the term “configured” is used in relation to computing systems and environments, as well as computer program components. A computing system or environment is considered "configured" to perform specific operations or actions when it possesses the necessary software, firmware, hardware, or a combination thereof, enabling it to carry out those operations or actions during operation. For instance, configuring a system might involve installing a software library with specific algorithms, updating firmware with new instructions for handling data, or adding a hardware component for enhanced processing capabilities. Similarly, one or more computer programs are “configured” to perform particular operations or actions when they contain instructions that, upon execution by a computing device or hardware, cause the device to perform those intended operations or actions.

[0184] The embodiments and functional operations described in this specification can be implemented in various forms, including digital electronic circuitry, software, firmware,Atorney Docket No. 45288-0573WO1

[0185] computer hardware (encompassing the disclosed structures and their structural equivalents), or any combination thereof. The subject matter can be realized as one or more computer programs, essentially modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by or to control the operation of a computing device or hardware. The storage medium can be a storage device such as a hard drive or solid-state drive (SSD), a storage medium, a random or serial access memory device, or a combination of these. Additionally or alternatively, the program instructions can be encoded on a transmitted signal, such as a machine-generated electrical, optical, or electromagnetic signal, designed to carry information for transmission to a receiving device or system for execution by a computing device or hardware. Furthermore, implementations may leverage emerging technologies like quantum computing or neuromorphic computing for specific applications, and may be deployed in distributed or cloud-based environments where components reside on different machines or within a cloud infrastructure.

[0186] The term “computing device or hardware” refers to the physical components involved in data processing and encompasses all types of devices and machines used for this purpose. Examples include processors or processing units, computers, multiple processors or computers working together, graphics processing units (GPUs), tensor processing units (TPUs), and specialized processing hardware such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). In addition to hardware, a computing device or hardware may also include code that creates an execution environment for computer programs. This code can take the form of processor firmware, a protocol stack, a database management system, an operating system, or a combination of these elements. Embodiments may particularly benefit from utilizing the parallel processing capabilities of GPUs, in a General-Purpose computing on Graphics Processing Units (GPGPU) context, where code specifically designed for GPU execution, often called kernels or shaders, is employed. Similarly, TPUs excel at running optimized tensor operations crucial for many machine learning algorithms. By leveraging these accelerators and their specialized programming models, the system can achieve significant speedups and efficiency gains for tasks involving artificial intelligence and machine learning, particularly in areas such as computer vision, natural language processing, and robotics.Atorney Docket No. 45288-0573WO1

[0187] A computer program, also referred to as software, an application, a module, a script, code, or simply a program, can be written in any programming language, including compiled or interpreted languages, and declarative or procedural languages. It can be deployed in various forms, such as a standalone program, a module, a component, a subroutine, or any other unit suitable for use within a computing environment. A program may or may not correspond to a single file in a file system and can be stored in various ways. This includes being embedded within a file containing other programs or data (e.g., scripts within a markup language document), residing in a dedicated file, or distributed across multiple coordinated files (e.g., files storing modules, subprograms, or code segments). A computer program can be executed on a single computer or across multiple computers, whether located at a single site or distributed across multiple sites and interconnected through a data communication network. The specific implementation of the computer programs may involve a combination of traditional programming languages and specialized languages or libraries designed for GPGPU programming or TPU utilization, depending on the chosen hardware platform and desired performance characteristics.

[0188] In this specification, the term “engine” broadly refers to a software-based system, subsystem, or process designed to perform one or more specific functions. An engine is typically implemented as one or more software modules or components installed on one or more computers, which can be located at a single site or distributed across multiple locations. In some instances, one or more dedicated computers may be used for a particular engine, while in other cases, multiple engines may operate concurrently on the same one or more computers. Examples of engine functions within the context of Al and machine learning could include data pre-processing and cleaning, feature engineering and extraction, model training and optimization, inference and prediction generation, and post-processing of results. The specific design and implementation of engines will depend on the overall architecture and the distribution of computational tasks across various hardware components, including CPUs, GPUs, TPUs, and other specialized processors.

[0189] The processes and logic flows described in this specification can be executed by one or more programmable computers running one or more computer programs to perform functions by operating on input data and generating output. Additionally, graphics processing units (GPUs) and tensor processing units (TPUs) can be utilized to enable concurrentAtorney Docket No. 45288-0573WO1

[0190] execution of aspects of these processes and logic flows, significantly accelerating performance. This approach offers significant advantages for computationally intensive tasks often found in Al and machine learning applications, such as matrix multiplications, convolutions, and other operations that exhibit a high degree of parallelism. By leveraging the parallel processing capabilities of GPUs and TPUs, significant speedups and efficiency gains compared to relying solely on CPUs can be achieved. Alternatively or in combination with programmable computers and specialized processors, these processes and logic flows can also be implemented using specialized processing hardware, such as field-programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs), for even greater performance or energy efficiency in specific use cases.

[0191] Computers capable of executing a computer program can be based on general-purpose microprocessors, special-purpose microprocessors, or a combination of both. They can also utilize any other type of central processing unit (CPU). Additionally, graphics processing units (GPUs), tensor processing units (TPUs), and other machine learning accelerators can be employed to enhance performance, particularly for tasks involving artificial intelligence and machine learning. These accelerators often work in conjunction with CPUs, handling specialized computations while the CPU manages overall system operations and other tasks. Typically, a CPU receives instructions and data from read-only memory (ROM), random access memory (RAM), or both. Computer elements may include a CPU for executing instructions and one or more memory devices for storing instructions and data. The specific configuration of processing units and memory will depend on factors like the complexity of the Al model, the volume of data being processed, and the desired performance and latency requirements. Embodiments can be implemented on a wide range of computing platforms, from small embedded devices with limited resources to large-scale data center systems with high-performance computing capabilities. The system may include storage devices like hard drives, SSDs, or flash memory for persistent data storage.

[0192] Computer-readable media suitable for storing computer program instructions and data encompass all forms of non-volatile memory, media, and memory devices. Examples include semiconductor memory devices such as read-only memory (ROM), solid-state drives (SSDs), and flash memory devices; hard disk drives (HDDs); optical media; and optical discs such as CDs, DVDs, and Blu-ray discs. The specific type of computer-readable media usedAtorney Docket No. 45288-0573WO1

[0193] will depend on factors such as the size of the data, access speed requirements, cost considerations, and the desired level of portability or permanence.

[0194] To facilitate user interaction, embodiments of the subject matter described in this specification can be implemented on a computing device equipped with a display device, such as a liquid crystal display (LCD) or an organic light-emitting diode (OLED) display, for presenting information to the user. Input can be provided by the user through various means, including a keyboard), touchscreens, voice commands, gesture recognition, or other input modalities depending on the specific device and application. Additional input methods can include acoustic, speech, or tactile input, while feedback to the user can take the form of visual, auditory, or tactile feedback. Furthermore, computers can interact with users by exchanging documents with a user's device or application. This can involve sending web content or data in response to requests or sending and receiving text messages or other forms of messages through mobile devices or messaging platforms. The selection of input and output modalities will depend on the specific application and the desired form of user interaction.

[0195] Machine learning models can be implemented and deployed using machine learning frameworks, such as TensorFlow or JAX. These frameworks offer comprehensive tools and libraries that facilitate the development, training, and deployment of machine learning models.

[0196] Embodiments of the subject matter described in this specification can be implemented within a computing system comprising one or more components, depending on the specific application and requirements. These may include a back-end component, such as a back-end server or cloud-based infrastructure; an optional middleware component, such as a middleware server or application programming interface (API), to facilitate communication and data exchange; and a front-end component, such as a client device with a user interface, a web browser, or an app, through which a user can interact with the implemented subject matter. For instance, the described functionality could be implemented solely on a client device (e.g., for on-device machine learning) or deployed as a combination of front-end and back-end components for more complex applications. These components, when present, can be interconnected using any form or medium of digital data communication, such as a communication network like a local area network (LAN) or a wide area network (WAN)Atorney Docket No. 45288-0573WO1

[0197] including the Internet. The specific system architecture and choice of components will depend on factors such as the scale of the application, the need for real-time processing, data security requirements, and the desired user experience.

[0198] The computing system can include clients and servers that may be geographically separated and interact through a communication network. The specific type of network, such as a local area network (LAN), a wide area network (WAN), or the Internet, will depend on the reach and scale of the application. The client-server relationship is established through computer programs running on the respective computers and designed to communicate with each other using appropriate protocols. These protocols may include HTTP, TCP / IP, or other specialized protocols depending on the nature of the data being exchanged and the security requirements of the system. In certain embodiments, a server transmits data or instructions to a user's device, such as a computer, smartphone, or tablet, acting as a client. The client device can then process the received information, display results to the user, and potentially send data or feedback back to the server for further processing or storage. This allows for dynamic interactions between the user and the system, enabling a wide range of applications and functionalities.

[0199] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0200] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallelAttomev Docket No. 45288-0573WO1

[0201] processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0202] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0203] What is claimed is:

Claims

Attorney Docket No. 45288-0573WO1CLAIMS1. A method performed by one or more computers, the method comprising:obtaining data indicating that data items generated by a first generative neural network from one or more first conditioning inputs and one or more second conditioning inputs fail to satisfy one or more similarity criteria, wherein the one or more similarity criteria require a first similarity between the one or more first conditioning inputs and data items generated based on the one or more first conditioning inputs to be greater than a second similarity between the one or more second conditioning inputs and the data items generated based on the one or more first conditioning inputs;generating, using a second generative neural network and based on each of the one or more first conditioning inputs, one or more first data items;generating, using the second generative neural network and based on each of the one or more second conditioning inputs, one or more second data items;generating one or more first training examples, each first training example comprising a first conditioning input, a first data item, and a second data item and indicating that the first data item is preferred over the second data item; andtraining a third generative neural network on a supervised objective using training data that includes the one or more first training examples.

2. The method of claim 1, wherein the third generative neural network is the first generative neural network.

3. The method of claim 1 or claim 2, wherein the third generative neural network is the second generative neural network.

4. The method of any one of claims 1-3, wherein the first conditioning inputs comprise text, and the data items comprise images.

5. The method of any one of claims 1-4, wherein generating the one or more first data items comprises, for each of the one or more first conditioning inputs:processing, using the second generative neural network, a modification input derived from the first conditioning input and a data item generated by the first generative neuralAttorney Docket No. 45288-0573WO1network based on the first conditioning input, a first data item that is a modified version of the data item, wherein a similarity between the first data item and the first conditioning input is greater than a similarity between the data item and the first conditioning input.

6. The method of any preceding claim, further comprising generating one or more second training examples, each second training example comprising a second conditioning input, a first data item, and a second data item and indicating that the second candidate data item is preferred over the first candidate data item, and wherein the training data includes the one or more first training examples and the one or more second training examples.

7. The method of any preceding claim, wherein obtaining the data comprises:receiving a plurality of conditioning inputs;generating, using the first generative neural network and based on each of the plurality of conditioning inputs, one or more data items; andidentifying, based on the one or more data items that have been generated based on each of the plurality of conditioning inputs, the one or more first conditioning inputs and the one or more second conditioning inputs from the plurality of conditioning inputs.

8. The method of any preceding claim when also dependent on claim 7, wherein identifying the one or more first conditioning inputs and one or more second conditioning inputs from the plurality of conditioning inputs comprises:identifying the one or more first conditioning inputs based on the first similarity between the one or more first conditioning inputs and the data items generated based on the one or more first conditioning inputs.

9. The method of any preceding claim when also dependent on claim 7, wherein identifying the one or more first conditioning inputs and one or more second conditioning inputs from the plurality of conditioning inputs comprises:identifying the one or more first conditioning inputs based on the second similarity between the one or more second conditioning inputs and the data items generated based on the one or more first conditioning inputs.Attorney Docket No. 45288-0573WO110. The method of any one of claim 8-9, wherein identifying the one or more first conditioning inputs and one or more second conditioning inputs from the plurality of conditioning inputs comprises:determining that the second similarity is greater than the first similarity.

11. The method of any preceding claim, wherein the first generative neural network is a diffusion neural network.

12. The method of any preceding claim, wherein the second generative neural network is a diffusion neural network.

13. The method of any preceding claim, wherein the third generative neural network is a diffusion neural network.

14. The method of any preceding claim, wherein the supervised objective is a direct preference optimization (DPO) objective.

15. The method of any preceding claim, wherein the supervised objective is an identity preference optimization (IPO) objective.

16. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one more computers to perform the operations of the respective method of any one of claims 1-15.

17. One or more computer storage media storing instructions that when executed by one or more computers cause the one more computers to perform the operations of the respective method of any one of claims 1-15.

18. One or more computer storage media storing parameters of the third generative neural network of any one of claims 1-17.Attorney Docket No. 45288-0573WO119. A method performed by one or more computers, the method comprising:receiving a conditioning input; andgenerating the data item conditioned on the conditioning input using the third generative neural network of any one of claims 1-17.

Images