Generating data items using generative neural networks conditioned on modified tokens

Abstract

Systems and methods of generating a data item, implemented by a trained neural network system comprising a generative neural network and one or more token modulation neural networks. The one or more token modulation neural networks are configured to process specific conditioning tokens and generate corresponding token modulation values that, when used to modulate the conditioning input of the generative neural network, cause the generative neural network to generate a data item with a particular characteristic associated with the specific conditioning tokens.

Description

ATTORNEY DOCKET NO. 45288-0608WO1 / / GP-403453-00-PCTGENERATING DATA ITEMS USING GENERATIVE NEURAL NETWORKS CONDITIONED ON MODIFIED TOKENSCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority to UK Patent Application No. GB2500749.3, filed 20 January 2025. The disclosure of the prior application is considered part of and is incorporated by reference in the disclosure of this application.BACKGROUND

[0002] This specification relates to processing data using machine learning models.

[0003] Neural networks are machine learning models that employ one or more layers of nonlinear units to predict an output for a received input. Some neural networks include one or more hidden layers in addition to an output layer. The output of each hidden layer is used as input to the next layer in the network, i.e., the next hidden layer or the output layer. Each layer of the network generates an output from a received input in accordance with current values of a respective set of parameters.SUMMARY

[0004] This specification descnbes systems and methods, implemented as computer programs on one or more computers in one or more locations, for generating data items with particular characteristics. The data items can be, e.g., still or moving images, audio, or other data items.

[0005] The methods described herein provide a framework for disentangling and recomposing complex visual characteristics, such as objects, lighting and pose, from source images, enabling the controlled generation of new data items with localized control of multiple characteristics. The methods exploit a per-token modulation space, where modifying the modulation vector for a single token leads to modifications localized to only the concept associated with that token. Using per-token modulations allows characteristics of a data item to be controlled, without affecting the fidelity of other characteristics of the data item.

[0006] In a first aspect there is described a computer-implemented method of generating a data item. The method can be implemented by a (trained) neural network system comprising a (trained) generative neural network and one or more (trained) token modulation neural networks. The generative neural network may be, e.g. a diffusion model neural network, inparticular a latent diffusion model. As used herein a diffusion model neural network includes a neural network to implement related models, such as flow models (e.g. rectified flow models) and consistency models. The generative neural network may utilize a transformer backbone.

[0007] In implementations the one or more token modulation neural networks can be neural networks that are configured to process specific conditioning tokens and generate corresponding token modulation values that, when used to modulate (modify) the conditioning input of the generative neural network, cause the generative neural network to generate a data item with a particular characteristic associated with the specific conditioning tokens.

[0008] The subject matter described in this specification can be implemented in particular embodiments so as to realize one or more of the following advantages.

[0009] The described techniques can be used to find distinct directions in a modulation space of a generative neural network such as a diffusion model. This facilitates modifying the generated data items, e.g. images, so that a particular characteristic can be incorporated, e.g. based on one or more example or source data items with the particular characteristic, from which a modification to a modulation direction can be learned. Furthermore, the modulation space allows for localized modifications of the generated data items, such that the particular characteristic can be incorporated without affecting other aspects of the data item.

[0010] The described techniques also enable multiple different particular characteristics to be combined. They can be used to confer conceptual characteristics beyond objects, such as pose and / or lighting in an image. This is unlike some other techniques which can be more restricted in the type or breadth of modifications that can be made. The described techniques also enable particular characteristics to be independently extracted from source data items, for example source images, without the need for spatial masking or other manual input such as concept mapping.

[0011] The details of one or more embodiments of the subject matter of this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.BRIEF DESCRIPTION OF DRAWINGS

[0012] FIG. 1 illustrates a structure and architecture of an example system for generating a data item with a particular characteristic.

[0013] FIG. 2 is a flow diagram of an example method for generating a data item.

[0014] FIG. 3 is a flow diagram of an example method for training a token modulation neural network.

[0015] FIG. 4 depicts four token modulation neural networks each trained to leam a token modulation value from a concept image.

[0016] FIG. 5 illustrates concept isolation loss.DETAILED DESCRIPTION

[0017] FIG. 1 illustrates a structure and architecture of a system 100 for generating a data item 190. The system comprises a (trained) generative neural network 110, for example a diffusion model neural network. As used herein a diffusion model neural network includes a neural network to implement related models, such as consistency models, flow models etc.

[0018] In some implementations, the generative neural network 110 comprises a stack of transformer blocks. For example where the generative neural network 110 comprises a diffusion model neural network, the diffusion model neural network can comprise a diffusion transformer neural network (DiT, e.g. as described in Peebles and Xie, “Scalable Diffusion Models with Transformers”, arXiv:2212.09748v2, March 2023). In Figure 1, one transformer block is shown.

[0019] The generative neural network 110 can receive an initialized representation of a data item 150 as a sequence of data item tokens 150’ and a characterizing input 140 characterizing one or more desired properties for the data item 190. That is, the characterizing input 140 characterizes one or more properties that the data item 190 generated by the system 100 should have. In Figure 1, the characterizing input 140 comprises the text sequence “a chicken made of mosaic under lights”, indicating that it is desired for the generated data item 190 to represent the text sequence, i.e. depict an image of a chicken made of mosaic under lights. The generative neural network 110 can be provided with this input as a sequence 145 of tokens 140’, 150’.

[0020] The data item tokens 150’ are, in some implementations, noisy data item tokens and this can involve, e.g., sampling the values in the representation from a noise distribution such as a Gaussian distribution. The characterizing input 140 may be encoded as a plurality, e.g. sequence, of conditioning tokens 140’. For example, where the characterizing input 140 comprises text in a natural or computer language this can be done using atokenizing method.e.g. BPE or WordPiece, that converts the text, e.g., words or wordpieces, into token identifiers.

[0021] A conditioning input for the generative neural network 110 is generated from the plurality of conditioning tokens. The conditioning input can comprise an embedding of the conditioning tokens 140', e.g. a pooled or partially pooled representation. As used throughout this specification, an “embedding’" refers to an ordered collection of numerical values, e.g., a vector or matrix of numerical values. The conditioning input can be generated by processing the plurality of conditioning tokens 140’ using a token embedding neural network 115. As an example, when the characterizing input 140 comprises text, a token embedding neural network 115 can comprise one or more (pre-trained) text encoder neural networks. For example, a Transformer neural network can be used to convert the conditioning token identifiers to respective vectors which can then be pooled, or partially pooled (e.g. retaining a representation of the sequence), or both. The token embedding neural network 115 may utilise, for example, contrastive Language-Image Pre-training (CLIP).

[0022] A transformer block typically includes an attention or self-attention neural network layer 125, which may be followed by a feedforward neural network layer 130. In general, a self-attention neural network layer applies a self-attention operation, e.g. a query-key-value (QKV) self-attention operation, to elements of an embedding to update each element of the embedding.

[0023] A transformer block can also include a modulation layer 120. For each transformer block the process involves mapping the conditioning input to a latent modulation space to generate a set of one or more modulation space conditioning parameters y in the latent modulation space. In implementations this can be done by applying a shared conditioning neural network, e.g. MLP (Multi-Layer Perceptron) to a conditioning input, to map the conditioning input to the latent modulation space for each of the transformer blocks.Broadly, the latent modulation space is a (learned) space of an intermediate representation of the conditioning input in which, when modifications are applied to the intermediate representation, the generated output is modified accordingly. The latent modulation space allows for per-channel scale and shift parameters for semantic image manipulation. In the systems and methods described herein, modulation can be applied on a per-token basis. Therefore, the modulation layer 120 is a per-token modulation layer, also referred to as a per-token modulation block.

[0024] The conditioning tokens 140" provided as a conditioning input to the generative neural network 110 and the conditioning tokens 140’ used to generate the set of modulationspace conditioning parameters y may be equivalent (e.g. derived using the same methods and / or same neural networks). Alternatively, they may be derived from different means, for example a first embedding may be used to convert the characterizing input 140 for input to the generative neural network 110 and a second embedding may be used to convert the characterizing input 140 for use in generating the set of modulation space conditioning parameters y.

[0025] The conditioning input includes one or more controlled conditioning tokens which represent one or more particular characteristics of the data item 190 that are to be controlled when generating the data item 190. In some implementations, a user can identify the controlled conditioning token(s). This can involve receiving an input defining the conditioning tokens 140’ that represent the one or more particular characteristics of the data item 190. For example, where the characterizing input 140 comprises text, such as the text sequence “a chicken made of mosaic under lights”, a user can identify that a particular word or words, e.g. ‘"chicken”, “mosaic”, “lights” defines the more particular characteristic(s) of the data item 190 that is to be controlled. The particular characteristic(s) may be obtained from one or more source data items 161, 162, 163 e.g. source images. A user interface can be provided for this purpose, e.g. by enabling a user to highlight or otherw ise select one or more words or tokens. The particular characteristic of the data item 190 that is obtained is defined by a (pre-trained) token modulation neural network 180.

[0026] Each of the controlled conditioning tokens is processed using a respective (pretrained) token modulation neural netw ork 180 to generate a respective token modulation value Ay. The token modulation values Ay can be applied to a per-token modulation space, where the modulation vector is modified for a specific text token. In general, a token modulation value Ay represents a modification to be applied to a conditioning input of the generative neural network 110, in a modulation space, i .e. conditioning space, of the generative neural network 110, so that the generative neural network 110 generates an output data item 190 with the particular characteristic specified by the controlled conditioning token. This modification can involve, for example, adding the token modulation value Ay to the conditioning input or using the token modulation value Ay to scale or shift the conditioning input within the modulation space.

[0027] A token modulation value Ay may be considered a direction vector that adjusts the direction of a token in the modulation space. Each direction vector is personalized to a respective particular characteristic. FIG. 1 depicts the system 100 at inference, where thetoken modulation values Ay (i.e. Achicken, Amosaic, Alight) have been pre-leamed e.g. in a previous training process. Training is described in more detail below.

[0028] The token modulation neural network 180 can process either the controlled conditioning token or an embedding of the controlled conditioning token. The token modulation neural network 180 may be termed a ConceptExtractor neural network. The token modulation value Ay may have a shape that matches a shape of a modulation space of the generative neural network 110.

[0029] In implementations, the modulation space conditioning parameters y are modified by each of the respective token modulation values Ay for the one or more controlled conditioning tokens, to obtain a set of modified modulation space conditioning parameters. In implementations a token modulation value Ay for a controlled conditioning token can be obtained for each of the modulation space conditioning parameters y from a token modulation neural network 180. A modulation space conditioning parameter y can be modified, e.g., by adding or subtracting a token modulation value Ay.

[0030] The transformer block uses the set of modified modulation space conditioning parameters. That is the processing of the elements of the input 145 to the transformer block to generate the elements of the output from the transformer block, is conditioned on the set of modified modulation space conditioning parameters rather than on the set of modulation space conditioning parameters Ay.

[0031] The transformer block may use one or more scale or shift neural network layers (before and / or after the self-attention neural network layer 125). A scale neural network layer applies a scaling factor dependent on one of the modified modulation space conditioning parameters. A shift neural network layer adds or subtracts a value, dependent on one of the modified modulation space conditioning parameters.

[0032] In some implementations an additional correction neural network, e.g. MLP, is applied, per controlled conditioning token and per transformer block. For example this can involve, for each transformer block, and for each of the controlled conditioning tokens, processing the conditioning input using a respective (trained) correction neural network of the transformer block to generate a modulation space correction value. The modulation space correction value can then be applied to one or more of the set of modified modulation space conditioning parameters modulating the transformer block processing.

[0033] FIG. 2 shows an example method 200 for generating a data item 190. The method 200 can be implemented by a neural network system comprising a generative neural network 110and one or more token modulation neural networks 180, for example such as the system 100 of FIG. 1.

[0034] The method 200 involves initializing 210 a representation of the data item 150 as a sequence of data item tokens 150’. In some implementations the data item tokens 150’ are noisy data item tokens and this can involve, e.g., sampling the values in the representation from a noise distribution such as a Gaussian distribution. The initialization may comprise generating noisy latent patches that tile the representation of the data item 150.

[0035] The method 200 involves receiving 220 a characterizing input 140 characterizing one or more desired properties for the data item 190. That is, the characterizing input 140 characterizes one or more properties that the data item 190 generated by the system 100 should have. The characterizing input 140 is encoded as a plurality, e.g. sequence, of conditioning tokens. For example, where the characterizing input 140 comprises text in a natural or computer language this can be done using a tokenizing method, e.g. BPE or WordPiece, that converts the text, e.g., words or wordpieces, into token identifiers.

[0036] The method 200 includes generating 230 a conditioning input for the generative neural network 110. The conditioning input can be generated from the plurality of conditioning tokens or directly from the characterizing input 140. The conditioning input can comprise an embedding of the conditioning tokens, e.g. a pooled or partially pooled representation.

[0037] The conditioning input is provided to the generative neural network 110.

[0038] The method 200 also involves identifying 240 one or more controlled conditioning tokens in the plurality of conditioning tokens. The one or more controlled conditioning tokens represent one or more particular characteristics of the data item 190 that are to be controlled when generating the data item 190.

[0039] As described above, in some implementations a user can identify 240 the controlled conditioning token(s). This can involve receiving an input defining the conditioning tokens that represent the one or more particular characteristics of the data item 190. For example, where the characterizing input 140 comprises text, a user can identify that a particular word or words defines the one or more more particular characteristics of the data item 190 that is to be controlled. The user may highlight or otherwise select one or more words or tokens, for example using a user interface.

[0040] Each of the controlled conditioning tokens is processed 250 using a respective (pretrained) token modulation neural network 180 to generate a respective token modulationvalue Ay. The token modulation neural network 180 can process either the controlled conditioning token or an embedding of the controlled conditioning token.

[0041] In general the characterizing input 140 can comprise one or more of text, an image, audio, or other data. As previously described the characterizing input 140 is encoded as a plurality of conditioning tokens, from which the conditioning data is generated, e.g. as a tensor embedding of the conditioning tokens, generated using a token embedding neural network.

[0042] For example, when the characterizing input 140 comprises text, the system can use a text encoder neural network, e.g., a Transformer neural network, to generate a fixed or variable number of text embeddings for the conditioning input.

[0043] When the characterizing input 140 comprises an image, the system can use an image encoder neural network, e.g., a convolutional neural network or a vision Transformer neural network, to generate a set of embeddings that represent the image.

[0044] When the characterizing input 140 comprises audio, the system can use, e.g., an audio encoder neural network, e.g., an audio encoder neural network that has been trained jointly with a decoder neural network as part of a neural audio codec, to generate one or more embeddings that encode the audio.

[0045] When the characterizing input 140 comprises a scalar value, the system can use, e.g., an embedding matrix to map the scalar value or a one-hot representation of the scalar value to an embedding.

[0046] As some examples a moving image can be generated from a text and / or audio characterizing input; or an audio data item can be generated from a text and / or image characterizing input. As some other examples, where a moving image or audio sequence is generated the characterizing input 140 can, e.g., include a previous video or audio sequence, or an edited version of a video or audio sequence.

[0047] In some cases, the characterizing input 140 comprises multiple different types of inputs, e.g., two or more of text, images, and audio. Then the system can generate one or more initial embeddings for each of the different types of inputs, i.e., using an appropriate encoder neural network as described above, and then process the initial embeddings for all of the different types of inputs, e.g. using a Transformer encoder neural network, to update each of the initial embeddings to generate a set of final embeddings as the conditioning input.

[0048] In some implementations, the methods (e.g. method 200) and systems (e.g. system 100) herein may be used to implement a computer-implemented method of generating a still or moving image. The method involves initializing a representation of the image 150 as asequence of image tokens 150', receiving a characterizing input 140 characterizing one or more desired properties for the image, processing the characterizing input 140 to encode the characterizing input 140 as a plurality of conditioning tokens, generating a conditioning input from the plurality of conditioning tokens, and identifying, from the plurality of conditioning tokens, one or more controlled conditioning tokens. The one or more controlled conditioning tokens represent one or more particular characteristics of the image that are to be controlled. For each of the controlled conditioning token, the controlled conditioning token is processed using a respective token modulation neural network 180 to generate a respective token modulation value Ay.

[0049] The image is generated by processing the representation of the image 150 using a generative neural network 110. The processing is conditioned on the conditioning input modified by the respective token modulation value Ay for each of the controlled conditioning tokens.

[0050] As previously described, some implementations of the system include a user interface, to enable a user to specify one or more training data pairs comprising a source data item 161, 162, 163 with the particular characteristic and the source characterizing input 140, e.g. text, that characterizes the particular characteristic.

[0051] In some implementations the system is configured to extract a particular characteristic (which can be considered as a particular concept) from a first image 161 by training the token modulation neural network 180 to generate the token modulation value Ay for a particular token representing the particular characteristic. A second image 190 can then be generated with the particular characteristic as it is particularly represented in the first image 161. This can be done by using the generated token modulation value Ay representing the particular characteristic in the first image 161 to modify a controlled conditioning token corresponding to the particular token that represents the particular characteristic of the second image 190.

[0052] As an illustrative example, the user can provide an image of a particular turtle on a beach and can specify that the system is to leam a respective token modulation value Ay for the word “turtle” in the phrase “a turtle on the beach”. The user interface can be used to direct the system to leam the token modulation value Ay that converts a generic turtle in an image generated by the system (e.g. for “the turtle is in the sea”) into an image containing the specific turtle previously provided with the source image of the particular turtle. This can be extended to other image attributes, e.g. pose and lighting; and to audio and other data items.

[0053] In another illustrative example, the user can provide an image of a particular human and specify that the system is to learn a respective modulation token value for that particular human. The system can then generate an image of the particular human, while also controlling other characteristics of the generated image such as other objects depicted in the image, the pose of the particular human, the clothes of the particular human etc. Using pertoken modulation as described herein enables personalization of humans, where capturing precise details is crucial, rather than simply depicting a generic human. Using modulations across the entire modulation space (rather than per-token modulation) to modify one concept in an image often results in non-local changes which affect other concepts in the generated image, for example representing a different human.

[0054] Some implementations involve training the token modulation neural network 180 to generate the token modulation value Ay representing the particular characteristic of the data item 190. This can be done by training a neural network system comprising the token modulation neural network 180 and the generative neural network 110, e.g. the diffusion model neural network, on a training data pair.

[0055] FIG. 3 illustrates a portion of a system for generating a data item during training. In particular, the illustrated portion shows four token modulation neural networks 380, which may be equivalent to the token modulation neural networks 180 described above and with reference to FIG. 1 during training. The illustrated portion also shows a per-token modulation layer 320 of a transformer block, which may be. for example, the per-token modulation layer 120 of FIG. 1.

[0056] The system is provided w ith a source data item 350, for example in the form of a source image, and a corresponding source characterizing input 340, for example in the form of a corresponding caption. The source data item 350 and source characterizing input 340 may be referred to as a training pair. The system can leam disentangled representations for each of a set of characteristics featured in the training data pair in an unsupervised manner. For example, in FIG. 3 the source data item 350 comprises an image of a mosaic vase holding flowers, and the caption 340 comprises the phrase “a mosaic vase holding flowers'’ which corresponds to the set of characteristics “mosaic”, “vase”, “holding” and “flowers”. Given the source data item 350 and characterizing input 340, each token modulation neural network 380 can be trained to leam a respective token modulation value Ay associated with a respective one of the set of characteristics. That is, a token modulation value Ay is learned for each of the words “mosaic”, “vase”, “holding” and “flowers” and each token modulation value Ayrepresents a disentangled representation of each of the visual concepts “mosaic”, “vase”, “holding” and “flowers”. The token modulation value Ay represents personalized directions in the modulation space which can be used to modulate the text tokens in the per-token modulation layer 320 enabling the generation of data items 390 with personalized characteristics. The per-token modulation layer 320 may be referred to as a per-token modulation block.

[0057] FIG. 4 shows a flow diagram of the method 400 for training a token modulation neural network. The token modulation neural network may be comparable to, for example, the token modulation neural networks 180, 380 described above.

[0058] The method 400 includes obtaining 410 a training data pair. The training data pair comprises a source data item 350 with the particular characteristic and a source characterizing input 340 that characterizes the particular characteristic. In particular, the source characterizing input 340 is one that, when encoded as a plurality of conditioning tokens, includes a controlled conditioning token that represents the particular characteristic of the source data item 350 that is to be controlled. The training pair may comprise, for example, a picture of a turtle as a source data item 350 and the phrase “a turtle” as a source characterizing input 340, characterizing the characteristic “turtle”. Each training data pair may comprise one or more characteristics, for example a single image (or other data item) may be accompanied by a caption which characterizes multiple characteristics.|059| The method 400 includes processing 420 the source characterizing input 340 to encode the source characterizing input 340 as a plurality of conditioning tokens and generating 430 a conditioning input from the plurality' of conditioning tokens.

[0060] The method 400 includes processing 440 the controlled conditioning token representing the particular characteristic of the source data item 350 using the token modulation neural network 380 to generate the token modulation value Ay for the particular characteristic. Where the source characterizing input 340 defines multiple characteristics, each characteristic can be directed to a different token modulation neural network 380 and used to generate an individual token modulation value Ay for that particular characteristic. Of course, while described as individual token modulation neural networks 380, these may be implemented in a single neural network with appropriate architecture (e.g. modular architecture, single-shot multi-head).

[0061] For illustrative purposes, the method 400 of training the token modulation neural network will be described in combination with a generative neural network 110 in the form ofa diffusion model. However, the principles apply equivalently to other suitable models (e.g. flow-based models, consistency models).

[0062] The method 400 includes processing 450 a generation input that comprises a representation (e.g. a noisy representation) of the source data item 350 and the conditioning input 340, using a diffusion model, to generate a diffusion output. For the processing 450, the conditioning input 340 is modified by the token modulation value Ay for the particular characteristic. The diffusion output may be a denoising output. The noise can be added, e.g. according to a noise schedule that defines a level of noise to be added depending on the update iteration or corresponding time step.

[0063] The method 400 includes maintaining 460 the trainable parameters of the generative neural network 110 (e.g. diffusion model). That is, while training the token modulation neural network 380 for a controlled conditioning token that represents a particular characteristic, the trainable parameters of the generative neural network 110 are held fixed.

[0064] The method 400 includes training 470 the token modulation neural network 380. The training can involve backpropagating gradients of a generative model training objective (e.g. a diffusion model training objective) that depends on the denoising output, through the generative neural network 110 to update the trainable parameters of the token modulation neural network 380. This may use any appropriate gradient descent optimization algorithm, e.g. Adam or another optimization algorithm. This training can be repeated, randomly sampling multiple different levels of noise to add to the noisy representation of the source data item.

[0065] In some implementations the diffusion model training objective depends on a difference between the denoising output and one or both of the source data item and noise added to the source data item 350 to generate the representation of the source data item.

[0066] The training can involve obtaining a plurality of training data pairs each comprising a source data item 350 with one of the particular characteristics and a corresponding source characterizing input 340 that characterizes the particular characteristic. Each corresponding source characterizing input 340 may be independent or may be shared between multiple identical source data items 350 (e.g. in the instance where a single characterizing input 340 describes multiple characteristics associated w ith a single source image). A plurality of the token modulation neural networks 380 can be trained, each for a different controlled conditioning token representing a different particular characteristic of the data item, each to generate the token modulation value Ay representing the respective particular characteristic. Disentanglement of multiple particular characteristics are naturally facilitated by the system’sinherent association between conditioning tokens (e.g. words in a caption) and their respective data item tokens (e.g. image patches).

[0067] The neural network system is trained using any training objective appropriate to the generative neural network 110 and the type(s) of data item it generated, such as a log likelihood loss, a cross entropy loss, and so forth. Where the generative neural network 110 comprises a diffusion model neural net ork, the neural network system can be trained using any diffusion model training objective.

[0068] Broadly, the training objective measures the quality of the output relative to a target output. For example in the case of a diffusion model the diffusion model training objective measures the quality of the denoising output relative to a target output generated from the source data item. Merely as an example diffusion model training objective can comprise a diffusion score matching objective or a variant thereof. Where the diffusion input includes data identifying the updating iteration or time step, this may be chosen randomly.

[0069] In general the training involves keeping trainable parameters, e.g. w eights, of the generative neural network 110, e.g. of the diffusion model neural network fixed (frozen), whilst updating trainable parameters, e.g. weights, of the token modulation neural netw ork 380.

[0070] Where the generative neural netw ork 110 comprises a diffusion model neural network the training can, but need not, involve applying classifier-free guidance, i.e. with two forward passes, one for the conditioned and another for the unconditioned diffusion model. For example a classifier-free guidance distillation technique can be used, e.g. as described in Hsiao et al., arXiv:2406.01954v2, June 2024, in which only one forward pass is used.

[0071] In some instances, the training and data item generation can be performed separately. In other implementations, there may be a combined method (e.g. implemented by one or more computers) of obtaining a trained generative neural network system configured to generate a data item by processing a plurality of conditioning tokens, the conditioning tokens specifying one or more desired properties for the data item. That is, the method may include both training of the generative neural network system (including token generation neural network(s) thereof) and generation of a data item with a particular characteristic.

[0072] A source data item is obtained that has a particular characteristic desired for the data item, represented by a particular conditioning token.

[0073] A token modulation neural network 380 is trained, using the source data item, to process a particular conditioning token to generate a token modulation value Ay thatrepresents a modification to apply to the particular conditioning token in a modulation space of the generative neural network system to obtain the particular characteristic.

[0074] A plurality, e.g. sequence, of conditioning tokens is obtained, specifying one or more desired properties for a data item generated by the generative neural network system, the plurality of conditioning tokens including the particular conditioning token.

[0075] The particular conditioning token is modified by processing the particular conditioning token to generate the token modulation value Ay, and applying the token modulation value Ay to the particular conditioning token to obtain a modified token.

[0076] The plurality of conditioning tokens, including the modified token, is processed using the generative neural network system to generate a data item with the particular characteristic.

[0077] In implementations training the token modulation neural network 380 involves updating trainable parameters of the token modulation neural network 380 whilst holding trainable parameters of the trained generative neural network system fixed. The training can use any training objective suitable for training the generative neural network system to generate a data item (whilst conditioned on conditioning data); in general this can depend on the particular type of generative neural network system with which the technique is being used.

[0078] In implementations that use a correction neural network as described above, the correction neural network for a transformer block can be trained in the same way as for the token modulation neural networks 380. In particular, this can involve processing the conditioning input using the respective correction neural network of the transformer block to generate a modulation space correction value and applying the modulation space correction value to the set of modified modulation space conditioning parameters modulating the transformer block processing. Then the correction neural network of the (each) transformer block can be trained by backpropagating gradients of the diffusion model training objective whilst keeping the trainable parameters of the diffusion model neural network fixed.

[0079] In some implementations, two stages of training occur. In a first stage, coarse aspects of the concept data item are learned. This is done by prioritizing the selection of high noise levels in the optimization of the diffusion loss. In a second stage, lower noise levels are selected resulting in refinement of the token modulation value Ay. In some implementations, where the generative neural network 110 comprises transformer blocks, an additional neural network, for example a MLP, is used to output a block modulation value for each transformerblock. The block modulation values can be combined with the token modulation values Ay, leading to per-token per-block modulation values. Analogous to noise levels, where flow models or consistency models are used the first and second stage may utilize training over different timestep ranges. For example, in the first stage, high timesteps (e.g. between timesteps 800 to 1000) may be used. In the second stage, low timesteps (e.g. below timesteps 800) may be used. The first stage may be implemented for a majority of the training iterations, for example over 50% of the iterations e.g. for 92% of the iterations. The second stage may be used for the remaining training iterations (e.g. 8% of the iterations).

[0080] In some implementations the training can use an additional concept isolation loss for at least some training steps. This can encourage the token modulation neural network 380 to learn directions in the modulation space that, when used to generate a data item with particular characteristics, do not interfere with data items generated without the particular characteristics. In broad terms this can involve training the system on a “mixed” data item in which part of the generated data item is modified and part is not, training the system train only on the region of the generated data item that is not modified.

[0081] FIG. 5 illustrates concept isolation loss, which may be used in training. In particular, FIG. 5 shows a token modulation neural network 580, which may be comparable to the other token modulation neural networks 380, 180 described herein, and generative neural networks 510, which may be comparable to other generative neural networks 110 described herein. These generative neural networks 510 are described, for the purposes of illustration, as diffusion models.

[0082] Concept isolation loss can involve training a token modulation neural network 580 on the diffusion model training objective using a concept isolation training data pair comprising a concept isolation data item 550 and a concept isolation characterizing input 540. The concept isolation data item 550 comprises disjoint regions 551, 552. The concept isolation characterizing input 540 comprises disjoint parts 541, 542 that respectively characterize the disjoint regions 551. For example, as illustrated the concept isolation data item 550 comprises a first region 551 depicting a doll wearing a jacket and a second region 552 depicting a giraffe and children in snow. The concept isolation characterizing input 540 comprises a first disjoint part 541 in the form of the phrase “a doll wearing a jacket” and a second disjoint part 542 in the form of a phrase “a picture divided into two parts. The left part is a doll wearing a jacket. The right part is a giraffe and children in snow”. One of the disjoint parts 541, 542 only includes the conditioning input from one disjoint region 551, 552 of the concept isolation data item 550. Only one of these disjoint parts 541, 542 (e.g., only the captiondescribing the doll wearing a jacket) is provided to the token modulation neural network 580. As such, only the conditioning input y generated from one of the parts of the concept isolation characterizing input 540 is modified by the token modulation value Ay for a particular characteristic.

[0083] A concept isolation training data pair can be obtained starting with a training data pair comprising a source data item 551 with the particular characteristic and a source characterizing input 541 that characterizes the particular characteristic. The concept isolation training data pair comprises a concept isolation data item 550 and a concept isolation characterizing input 540.

[0084] A second source data item 552 characterized by a second source characterizing input (e.g. the phrase "‘a giraffe and children in snow”) can be generated by the diffusion model neural network conditioned on a conditioning input generated from the second source characterizing input.

[0085] The concept isolation data item 550 can be generated with a first region comprising (consisting of) the source data item 350 with the particular characteristic and a second region, disjoint with the first region, comprising (consisting of) the second source data item 552.

[0086] The concept isolation characterizing input 540 is arranged to have a first portion 541 comprising the source characterizing input 340 and a second portion 542, disjoint with the first portion, comprising the second source characterizing input. For example in the case of text, a prompt may be constructed that comprises the first and second portions, optionally with text that describes a format or relative disposition of the two regions of the data item, e.g. “the left region is” the right region is”.

[0087] The concept isolation characterizing input can be processed to encode this as a plurality of concept isolation conditioning tokens, including the controlled conditioning token representing the particular characteristic of the data item. A concept isolation conditioning input can then be generated from the plurality of concept isolation conditioning tokens.

[0088] The conditioning token representing the particular characteristic of the source data item 550 can be processed using the token modulation neural network 580 to generate the token modulation value Ay for the particular characteristic.

[0089] A diffusion input that comprises a representation of the concept isolation data item and the concept isolation conditioning input can be processed, using the diffusion model neural network, to generate the denoising output. The concept isolation conditioning input is modified by the token modulation value Ay for the particular characteristic.

[0090] The token modulation neural network 580 for the controlled conditioning token that represents the particular characteristic can then be trained by backpropagating gradients of a diffusion model training objective to update trainable parameters of the token modulation neural network 580 whilst keeping trainable parameters of the diffusion model neural network fixed.

[0091] The diffusion model training objective is evaluated just for the second region of the concept isolation data item comprising the second source data item 552 (generated by the diffusion model neural network 110).

[0092] The diffusion model training objective can, e.g., depend on a difference between the denoising output and one or both of the concept isolation data item and noise added to the concept isolation data item to generate the representation of the concept isolation data item. The diffusion model training objective is evaluated for a region of the concept isolation data item that excludes the first region of the concept isolation data item comprising the source data item 550.

[0093] Performance of the methods of data item generation described herein are evaluated quantitatively using Concept Preservation (CP), which evaluates how well specific visual characteristics are maintained, and Prompt Fidelity (PF), which measures adherence to a target text prompt. Both metrics CP and PF are ranked individually from 0 to 1 using an appropriate neural network model. In existing methods of data item generation, there often exists a tradeoff between CP and PF. A combined product CP-PF is used as a unified measure of success.

[0094] Performance is evaluated for three tasks: composition (composing an image with particular characteristics learned from concept images), decomposition (extracting several characteristics from a single image) and a full task of decomposition and composition. For each task, the methods described herein are compared to other available models suitable for the relevant tasks. In particular, benchmarking is performed against models which either use further training to fine-tune model parameters (e.g. using low-rank adaptation, LoRA), or which require spatial conditioning masks and / or manual concept mapping to identify or combine multiple characteristics.

[0095] In composition, the methods described herein outperform all alternative models for CP, achieving a CP score of CP = 0.470108 compared to the nearest highest CP score of CP = 0.392912 for a model which requires spatial conditioning masks. While the methods described herein are out-performed in prompt fidelity by a LoRA model, achieving PF = 0.688061 compared to the LoRA model achieving PF = 0.844422, the unified successmeasure for the methods described herein represents a greater overall performance, achieving CP-PF = 0.323463 compared to the LoRA model achieving CP-PF = 0.236642.

[0096] In decomposition, the methods described herein outperform all alternative models for CP, achieving a CP score of CP = 0.669940 compared to the nearest highest CP score of CP = 0.668524 for the LoRa model. While the methods described herein are out-performed in PF by a model which requires spatial conditioning masks, achieving PF = 0.747698 compared to the spatial conditioning mask method achieving PF = 0.791793, the unified success measure for the methods described herein represents a significantly greater overall performance, achieving CP-PF = 0.500431 compared to CP-PF = 0.211787 for the spatial conditioning mask method.

[0097] For the full task, the methods described herein outperform all alternative models for CP, achieving a CP score of CP = 0.553125 compared to the nearest highest CP score of CP = 0.499054 for a spatial conditioning mask method. While the methods described herein are out-performed in PF by a model which requires spatial conditioning masks, achieving PF = 0.553125 compared to the spatial conditioning mask method achieving PF = 0.843395. the unified success measure for the methods described herein represents a significantly greater overall performance, achieving CP-PF = 0.454600 compared to CP-PF = 0.175246 for the spatial conditioning mask method.

[0098] In summary-, the overall performance of the methods described herein represent a significant improvement compared to alternative methods, while also obviating the need for model re-training and manually implementing masking or concept mapping.

[0099] Diffusion model neural networks

[0100] As described above, the generative neural networks 110, 510 referred to herein may comprise a diffusion model neural network, also referred to as a diffusion neural network or diffusion model. As used herein a diffusion model neural network includes a neural network to implement related models, such as consistency models, flow models etc.

[0101] In general, a (trained) diffusion model neural network can be used to generate a representation of a data item across multiple time steps by performing a reverse diffusion process to gradually de-noise an initial data item until a final representation of the data item is reached. The diffusion model neural network can process a current representation of the data item, e.g. to predict a de-noised representation of the data item or an estimate of a noise component of the current representation (an estimate of the noise that has been added to theoriginal representation of the data item to generate the current noisy representation of the data item). The representation of the data item can be the data item.

[0102] In some implementations diffusion model neural network implements a latent diffusion model (LDM). Then the representation of the data item can be a representation of the data item in a latent space, e.g. one that has fewer dimensions than a space of the data item. The de-noising process is performed in the latent space and a final representation from the process can be processed using a decoder neural network to generate a final, output data item.

[0103] In general the diffusion model neural network is arranged to process an input to generate an output of the same dimension. In implementations of the described system the diffusion model neural network also processes conditioning data that guides generation of the output. Some examples of diffusion model neural networks are described later.

[0104] In implementations the representation of the data item is updated at a plurality of updating iterations to generate a final representation of the data item.

[0105] Each updating iteration can involve identifying a current representation of the data item comprising a current sequence of data item tokens as of the updating iteration. At a first iteration this can be the initialized representation of the data item; at each subsequent iteration this can be the current sequence of data item tokens from the previous iteration.

[0106] In implementations, at each iteration the diffusion model neural network processes a diffusion input for the updating iteration to generate a denoising output for the updating iteration. The diffusion input comprises a current representation of the data item and the conditioning input. The conditioning input is modified by the respective token modulation value Ay for each of the controlled conditioning tokens. That is, the conditioning input is modified, in a modulation space of the diffusion model, by the respective token modulation value Ay. In implementations the diffusion input can also include data identify ing the updating iteration, e.g. data identifying a corresponding time point for the updating iteration. In implementations, the modifications from multiple token modulation values Ay can be combined, e.g. summed, in modulation space.

[0107] The denoising output is configured to predict a reduced noised representation of the data item, either directly, or indirectly, e.g. by predicting a noise component of the representation. The current representation of the data item is updated using the denoising output.

[0108] The final, e.g. output data item is generated from the final representation of the data item. The final representation of the data item can be the final data item or, where the diffusion model neural network operates in latent space, the final, generated data item can be decoded from the final representation of the data item by a (trained) decoder neural network.

[0109] The decoder neural network can complement an encoder neural network used to encode data items during training of the diffusion model neural network, e.g. the decoder and encoder neural networks of a trained autoencoder can be used.

[0110] Merely as an example, initializing the representation of the data item may comprise generating a noisy latent representation of the data item, which can then be re-arranged into a sequence of patches that tile the latent representation, to which a positional encoding may be added. The patch encoding can be reversed for the final representation of the data item, and the representation of the data item can then be converted back to a space of the data item, e.g. to the pixel space of an image.

[0111] The diffusion model neural network can be a Transformer neural network, such as a diffusion transformer (DiT), that processes the diffusion input through a set of self-attention layers to generate the denoising output. The representation of the data item can comprise a plurality, e.g. sequence, of tokens, e.g. in a latent space that encodes the data item, which may comprise, e.g. an audio signal or an image. In some implementations the generative neural network 110, 510 comprises a stack of transformer blocks. In some implementations processing the input, e.g. the diffusion input, using the generative neural network, e.g. the diffusion transformer neural network, to generate the output, e.g. the denoising output involves processing the current sequence of data item tokens using a stack of transformer blocks to generate the output, e.g. the denoising output for the updating iteration.

[0112] As another example the diffusion model neural network can be a convolutional neural network, e.g., a U-Net or other architecture that maps one input of a given dimensionality to an output of the same dimensionality.

[0113] In some implementations, the generative neural network 110, 510 comprises a unified multimodal model trained over both discrete and continuous data. Such a model may utilize a single transformer backbone that processes mixed-modal sequences comprising the source data item and the edit instruction. The model may apply a language modeling objective (e.g., next token prediction) to text tokens of the edit instruction and a diffusion objective to continuous representations (e.g., image patches) of the edited data item being generated. The model may employ modality-specific encoding and decoding layers (e.g., linear layers or U-Net blocks) to convert between raw data and the transformer's vector space, potentiallyapplying causal attention to the instruction tokens and bidirectional attention to the image patches within the shared transformer architecture, thereby conditioning the generation of the edited data item on the semantic and structural context provided by the mixed-modal sequence.

[0114] In general the diffusion model neural network generates the denoising output conditioned on the conditioning input. There are many ways in which the conditioning can be performed, e.g. by including extra layers such as scale or shift layers, or Feature-wise Linear Modulation (FiLM) layers, or layers with conditional gated activation functions; or by using cross-attention to cross-attend to the conditioning input; or by including the conditioning input as extra input tokens.

[0115] The diffusion input at any given updating iteration can also include data defining a noise level for the iteration, e.g. defining an updating or time step. Generally, each updating iteration has a corresponding time step t and the noise level for the iteration depends on the time step. For example, the noise level can be a decreasing function of the time step t.Examples of such functions include a linear function, a cosine function, and a sigmoid function. In these cases, data identifying the noise level, the time step, or both can be embedded using an appropriate neural network, e.g., a multi-layer perceptron (MLP) and used to condition the diffusion model neural netw ork as described above for the conditioning input.

[0116] In some implementations the representation of the data item as a sequence of data item tokens can include a positional encoding for each token, e.g. a frequency-based positional embedding such as a sine-cosine embedding. The data defining the noise level for the iteration can also include a positional encoding.

[0117] In some implementations, the diffusion model neural netw ork is one of a sequence of diffusion model neural networks, e.g., a hierarchy or a cascade of diffusion model neural networks, that the system uses to generate a final data item. For example, each diffusion model neural network in the sequence can receive as input the output data item generated by the preceding diffusion model neural network in the sequence and generate an output data item that has an increased resolution, e.g., an increased spatial resolution, an increased temporal resolution, or both, relative to the preceding diffusional neural network in the sequence. In these implementations, all of the neural networks in the sequence can receive the conditioning input or only a proper subset of the diffusion model neural netw orks in the sequence can receive the conditioning input, e.g., only the diffusion model neural networks at one or more earliest positions in the sequence.

[0118] The final representation of the data item can be a representation of the generated data item in a latent space representation of the output space of the generated data item. The output space may comprise values of a generated data item of the appropriate type, e.g., values of image pixels, amplitude values of an audio signal, and so forth. Alternatively the final representation of the data item generated using the diffusion model neural network can be a representation in the output space.

[0119] The final representation of the data item can be decoded into the output space using a decoder neural network, e.g. one that has been pre-trained in an auto-encoder framework. During training, the system can use an encoder neural network, e.g., one that has been pretrained jointly with the decoder in the auto-encoder framework, to encode data items in the output space to generate target outputs for the diffusion model neural network in the latent space.

[0120] The methods and systems herein can be implemented using a one or more computers, and one or more storage devices communicatively coupled to the one or more computers. The storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform the operations of the described methods. The storage devices may be implemented separately to the system, for example in some implementations there are one or more non-transitory computer storage media storing instructions that when executed by one or more computers perform the operations of the described methods.

[0121] Example generated data items

[0122] A data item can define, e.g. pixel values of an image, audio signal values (e.g. instantaneous amplitude values), and so forth. As used herein an image can be a still or moving image, in monochrome or color (including in non-visible wavelengths), in 2D or in 3D; and can includes a LIDAR point cloud (a “pixel” may then be a point of the point cloud).

[0123] A moving image i.e. video can be generated, e.g. by treating the temporal axis as an additional spatial dimension, or by using a rolling diffusion technique as described in Ruhe et al., arXiv: 2402.09470vl, and in various other ways.

[0124] An image, or training image, as described herein may be an image that represents a real world environment. For example a training image may be captured from the real world by a camera or other image sensor. In a training image, or an image generated by a diffusion model trained as described herein, objects represented in the image may comprise physical real-world objects.

[0125] The generated data item can but need not represent an image. As some other examples the generated data item can represent audio data, weather or climate data, or fluid mechanics data. In some implementations the generated data item can encode another type of data, e.g. a generated image can encode an audio or other spectrogram from which the audio or other data can be recovered, e.g. with an inverse time-frequency domain transform.

[0126] As another example, implementations of the system can be used to generate an image representing a predicted state or trajectory of a real -world physical system, such as a robot or vehicle, for use by a control algorithm in controlling the physical system. For example a video sequence may be captured by a camera and then the system used to generate a still or moving image that predicts a future state or configuration of the physical system, optionally conditioned on one or more variables relating to the physical system. This can be used in a model predictive control system to control a mechanical agent such as a robot to perform a particular task, by processing the predicted image using the control system to generate control signals to control the mechanical agent, in accordance with the generated image to perform the task.

[0127] As some other examples, the generated data item may comprise, e.g., sensor data such as data for an EEG or ECG trace; or medical imaging data such as CT or MRI data; or data representing a 3D configuration of a molecule, e.g. as coordinates of atoms of the molecule.

[0128] This specification uses the term “configured” in connection with systems and computer program components. For a system of one or more computers to be configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform the operations or actions. For one or more computer programs to be configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform the operations or actions.

[0129] Embodiments of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly -embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory storage medium for execution by, or to control the operation of, data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them. Alternatively or in addition, the program instructions can be encoded on an artificially -generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to suitable receiver apparatus for execution by a data processing apparatus.

[0130] The term “data processing apparatus7’ refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can also be, or further include, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them. Thus a system, artificial neural network, or trained artificial neural network as descnbed herein, can be implemented in hardware using electronic circuitry, e.g. in a physical box. Similarly computer code as described herein can be code to emulate such hardware or code for a hardware description language.

[0131] A computer program, which may also be referred to or described as a program, software, a software application, an app. a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages; and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a data communication network.

[0132] In this specification the term “engine” is used broadly to refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some cases, one or morecomputers will be dedicated to a particular engine; in other cases, multiple engines can be installed and running on the same computer or computers.

[0133] The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA or an ASIC, or by a combination of special purpose logic circuitry and one or more programmed computers.

[0134] Computers suitable for the execution of a computer program can be based on general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. The central processing unit and the memory can be supplemented by, or incorporated in, special purpose logic circuitry. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. Moreover, a computer can be embedded in another device, e.g., a mobile telephone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a Global Positioning System (GPS) receiver, or a portable storage device, e.g., a universal serial bus (USB) flash drive, to name just a few.

[0135] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory¬ devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks.

[0136] To provide for interaction with a user, embodiments of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory7feedback, e.g., visual feedback, auditory- feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from adevice that is used by the user; for example, by sending web pages to a web browser on a user’s device in response to requests received from the web browser. Also, a computer can interact with a user by sending text messages or other forms of message to a personal device, e.g., a smartphone that is running a messaging application, and receiving responsive messages from the user in return.

[0137] Data processing apparatus for implementing machine learning models can also include, for example, special-purpose hardware accelerator units for processing common and compute-intensive parts of machine learning training or production, i.e., inference, workloads.

[0138] Machine learning models can be implemented and deployed using a machine learning framework, e.g., a TensorFlow framework.

[0139] Embodiments of the subject matter described in this specification can be implemented in a computing system that includes a back-end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front-end component, e.g., a client computer having a graphical user interface, a web browser, or an app through which a user can interact with an implementation of the subject matter described in this specification, or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g.. a communication network. Examples of communication networks include a local area network (LAN) and a wide area network (WAN), e.g., the Internet.

[0140] The computing system can include clients and servers. A client and server are generally remote from each other and ty pically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some embodiments, a server transmits data, e.g., an HTML page, to a user device, e.g., for purposes of displaying data to and receiving user input from a user interacting with the device, which acts as a client. Data generated at the user device, e.g., a result of the user interaction, can be received at the server from the device.

[0141] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular inventions. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment.Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially be claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

[0142] Similarly, while operations are depicted in the drawings and recited in the claims in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0143] Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some cases, multitasking and parallel processing may be advantageous.

[0144] What is claimed is:

Claims

CLAIMS1. A computer implemented method of generating a data item, comprising:initializing a representation of the data item as a sequence of data item tokens; receiving a characterizing input characterizing one or more desired properties for the data item;processing the characterizing input to encode the characterizing input as a plurality of conditioning tokens;generating a conditioning input from the plurality of conditioning tokens; identifying, from the plurality of conditioning tokens, one or more controlled conditioning tokens, wherein the one or more controlled conditioning tokens represent one or more particular characteristics of the data item that are to be controlled;for each of the controlled conditioning tokens, processing the controlled conditioning token using a respective token modulation neural network to generate a respective token modulation value;processing the representation of the data item using a generative neural network, wherein the processing is conditioned on the conditioning input modified by the respective token modulation value for each of the controlled conditioning tokens; andgenerating the data item from the final representation of the data item.

2. The method of claim 1, wherein the generative neural network comprises a plurality of transformer blocks; and wherein processing the representation of the data item using the generative neural network comprises:processing the representation of the data item using the transformer blocks to generate the processed representation of the data item and, for each transformer block:mapping the conditioning input to a latent modulation space to generate a set of one or more modulation space conditioning parameters in the latent modulation space;modifying the modulation space conditioning parameters by the respective token modulation values for the controlled conditioning tokens to obtain a set of modified modulation space conditioning parameters; andmodulating the transformer block processing using the set of modified modulation space conditioning parameters.

3. The method of claim 1 or 2, wherein the generative neural network comprises a diffusion model neural network; and wherein processing the representation of the data item using the generative neural network comprises:updating the representation of the data item to generate a final representation of the data item, the updating comprising, at each of a plurality of updating iterations:processing a diffusion input for the updating iteration that comprises a current representation of the data item and the conditioning input, using the diffusion model neural network, to generate a denoising output for the updating iteration, wherein the conditioning input is modified by the respective token modulation value for each of the controlled conditioning tokens; andupdating the current representation of the data item using the denoising output.

4. The method of claim 3, wherein the diffusion model neural network comprises a diffusion transformer neural network; and wherein processing the diffusion input using the diffusion transformer neural network to generate the denoising output comprises:processing the current sequence of data item tokens using a stack of transformer blocks to generate the denoising output for the updating iteration and, for each transformer block:mapping the conditioning input to a latent modulation space to generate a set of one or more modulation space conditioning parameters in the latent modulation space;modifying the modulation space conditioning parameters by the respective token modulation values for the controlled conditioning tokens to obtain a set of modified modulation space conditioning parameters; andmodulating the transformer block processing using the set of modified modulation space conditioning parameters.

5. The method of claim 4, wherein modulating the transformer block processing using the set of modified modulation space conditioning parameters comprises:processing an input to the transformer block using a self-attention neural network layer and one or more scale or shift neural network layers to generate an output from the transformer block, wherein the scale or shift neural network layers, respectively, apply a scaling factor, or add or subtract a value, dependent on one of the modified modulation space conditioning parameters.

6. The method of any of claims 4-5, further comprising, for each transformer block, and for each of the controlled conditioning tokens:processing the conditioning input using a respective correction neural network of the transformer block to generate a modulation space correction value; andapplying the modulation space correction value to the set of modified modulation space conditioning parameters modulating the transformer block processing.

7. The method of any of claims 1 -6, wherein identifying the one or more controlled conditioning tokens comprises receiving an input defining one or more conditioning tokens that represent the one or more particular characteristics of the data item that are to be controlled.

8. The method of any of claims 1 -7, wherein the data item comprises a still or moving image; wherein the data item tokens comprise image tokens; and wherein the characterizing input comprises text; the method further comprising:extracting a particular characteristic from a first image by training the token modulation neural network to generate the token modulation value for a particular token representing the particular characteristic; andgenerating a second image with the particular characteristic as represented in the first image by using the generated token modulation value representing the particular characteristic in the first image to modify a controlled conditioning token corresponding to the particular token that represents the particular characteristic of the second image.

9. The method of any of claims 1-8, further comprising training the token modulation neural network to generate the token modulation value representing a particular characteristic of the data item, by training a neural network system comprising the token modulation neural network and the generative neural network on a training data pair comprising a source data item with the particular characteristic and a source characterizing input that characterizes the particular characteristic, wherein the training comprises keeping trainable parameters of the generative neural network fixed whilst updating trainable parameters of the token modulation neural network.

10. The method of claim 9, wherein the generative neural network comprises a diffusion model neural network, and wherein the training comprises training using a diffusion model training objective.

11. The method of any of claims 1-10, further comprising training the token modulation neural network for a controlled conditioning token representing a particular characteristic of the data item to generate the token modulation value representing the particular characteristic; wherein the training comprises:obtaining a training data pair comprising a source data item with the particular characteristic and a source characterizing input that characterizes the particular characteristic;processing the source characterizing input to encode the source characterizing input as the plurality of conditioning tokens;generating the conditioning input from the plurality of conditioning tokens; processing the controlled conditioning token representing the particular characteristic of the source data item using the token modulation neural network to generate the token modulation value for the particular characteristic;processing a diffusion input that comprises a representation of the source data item and the conditioning input, using the diffusion model neural network, to generate the denoising output, wherein the conditioning input is modified by the token modulation value for the particular characteristic; andwhilst maintaining trainable parameters of the diffusion model neural network fixed: training the token modulation neural network for the controlled conditioning token that represents the particular characteristic by backpropagating gradients of a diffusion model training objective that depends on the denoising output, through the diffusion model neural network to update trainable parameters of the token modulation neural network.

12. The method of any of claims 9-11, wherein the diffusion model training objective depends on a difference betw een the denoising output and one or both of the source data item and noise added to the source data item to generate the representation of the source data item.

13. The method of any of claims 9-12, further comprising:obtaining a plurality' of training data pairs each comprising a source data item w ith one of the particular characteristics and a corresponding source characterizing input that characterizes the particular characteristic;training a plurality of the token modulation neural networks, each for a different controlled conditioning token representing a different particular characteristic of the data item, each to generate the token modulation value representing the respective particular characteristic.

14. The method of any of claims 9-13, when dependent on claim 6, wherein processing the diffusion input that comprises the representation of the source data item and the conditioning input, using the diffusion model neural network, to generate the denoising output comprises:for each transformer block:processing the conditioning input using a respective correction neural network of the transformer block to generate a modulation space correction value, andapplying the modulation space correction value to the set of modified modulation space conditioning parameters modulating the transformer block processing; and whilst maintaining trainable parameters of the diffusion model neural network fixed training the correction neural network of each transformer block by backpropagating gradients of the diffusion model training objective.

15. The method of any of claims 9-14, wherein the training further comprises:training the token modulation neural network on the diffusion model training objective using a concept isolation training data pair comprising a concept isolation data item and a concept isolation characterizing input, wherein the concept isolation data item comprises disjoint regions,wherein the concept isolation characterizing input comprises disjoint parts that respectively characterize the disjoint regions, andwherein only the conditioning input generated from one of the parts of the concept isolation characterizing input is modified by the token modulation value for a particular characteristic.

16. The method of any of claims 9-15, wherein the training further comprises:obtaining a concept isolation training data pair comprising a concept isolation data item and a concept isolation characterizing input, by:generating a second source data item characterized by a second source characterizing input, using the diffusion model neural network conditioned on a conditioning input generated from the second source characterizing input,generating the concept isolation data item having a first region comprising the source data item with the particular characteristic and a second region, disjoint with the first region, comprising the second source data item, andgenerating the concept isolation characterizing input having a first portion comprising the source characterizing input and a second portion, disjoint with the first portion, comprising the second source characterizing input;processing the concept isolation characterizing input to encode the concept isolation characterizing input as a plurality of concept isolation conditioning tokens including the controlled conditioning token representing the particular characteristic of the data item; generating a concept isolation conditioning input from the plurality of concept isolation conditioning tokens;processing the controlled conditioning token representing the particular characteristic of the source data item using the token modulation neural network to generate the token modulation value for the particular characteristic;processing a diffusion input that comprises a representation of the concept isolation data item and the concept isolation conditioning input, using the diffusion model neural network, to generate the denoising output, wherein the concept isolation conditioning input is modified by the token modulation value for the particular characteristic; andwhilst maintaining trainable parameters of the diffusion model neural network fixed: training the token modulation neural network for the controlled conditioning token that represents the particular characteristic by backpropagating gradients of a diffusion model training objective to update trainable parameters of the token modulation neural network, wherein the diffusion model training objective depends on a difference between the denoising output and one or both of the concept isolation data item and noise added to the concept isolation data item to generate the representation of the concept isolation data item, w herein the diffusion model training objective is evaluated for a region of the concept isolation data item that excludes the first region of the concept isolation data item comprising the source data item.

17. A computer-implemented method of training a token modulation neural network to generate a token modulation value representing a particular characteristic of a data item; comprising:obtaining a training data pair comprising a source data item with the particular characteristic and a source characterizing input that characterizes the particular characteristic;processing the source characterizing input to encode the source characterizing input as the plurality of conditioning tokens;generating the conditioning input from the plurality of conditioning tokens; processing the controlled conditioning token representing the particular characteristic of the data item using the token modulation neural network to generate the token modulation value for the particular characteristic;processing a diffusion input that comprises a representation of the source data item and the conditioning input, using the diffusion model neural network, to generate a denoising output, wherein the conditioning input is modified by the token modulation value for the particular characteristic;maintaining trainable parameters of the diffusion model neural network fixed; and training the token modulation neural network for the controlled conditioning token that represents the particular characteristic by backpropagating gradients of a diffusion model training objective that depends on the denoising output, through the diffusion model neural network to update trainable parameters of the token modulation neural network.

18. A method implemented by one or more computers, comprising:obtaining a trained generative neural network system configured to generate a data item by processing a plurality of conditioning tokens, wherein the conditioning tokens specify one or more desired properties for the data item;obtaining a source data item that has a particular characteristic desired for the data item represented by a particular conditioning token;training a token modulation neural network, using the source data item, to process a particular conditioning token to generate a token modulation value that represents a modification to apply to the particular conditioning token in a modulation space of the generative neural network system to obtain the particular characteristic;obtaining a plurality7of conditioning tokens that specify one or more desired properties for a data item generated by the generative neural network system, the plurality of conditioning tokens including the particular conditioning token;modifying the particular conditioning token by processing the particular conditioning token to generate the token modulation value and applying the token modulation value to the particular conditioning token to obtain a modified token;processing the plurality of conditioning tokens including the modified token using the generative neural network system to generate a data item with the particular characteristic.

19. The method of claim 18, wherein training the token modulation neural network comprises updating trainable parameters of the token modulation neural network whilst holding trainable parameters of the trained generative neural network system fixed.

20. The method of any of claims 1-19. wherein the data item comprises a still or moving image; wherein the data item tokens comprise image tokens; and wherein the characterizing input comprises text.

21. One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations of the respective method of any one of claims 1-20.

22. A system comprising one or more computers and one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations of the respective method of any one of claims 1-20.

Images