Training and using generative models to generate synthetic data

Abstract

Implementations are described herein for generating synthetic multi-source audio. In various implementations, an input prompt may be assembled that includes, as conditioning data, a request to generate audio content comprising multiple distinct sources of sound. The input prompt may be processed using one or more generative acoustic models to generate a plurality of distinct, time-synchronized stems of acoustic representations. Each stem may carry acoustic representations corresponding to a respective source of sound of the multiple distinct sources of sound. The plurality of distinct, time-synchronized audio source stems of acoustic representations may be decoded to generate a stream of audio output frames.

Description

Attorney Docket No. GOOG-0780-WO-01TRAINING AND USING GENERATIVE MODELS TO GENERATE SYNTHETIC DATAATTORNEY REFERENCE: GOOG-0780-WO-01 Background

[0001] Generative acoustic models such as text-to-speech (TTS) models may be applied to a conditioning signal as input to generate audio as output. Depending on the use case, the conditioning signal may assume different forms, such as text, audio, other modalities (e.g., images or videos), or any combination thereof. Some non-limiting examples of conditioning signals include text descriptions for text-to-audio or text to music generation, source speech for speech-to-speech translation or speech continuation, text transcripts and speech prompt(s) for voice controlled text-to-speech synthesis, etc.Summary

[0002] Generative acoustic models are advancing rapidly. Some combine neural audio compression, self-supervised representation learning, and language modeling to unlock use cases such as generation of long form content. This long form content can include, for instance, stories, narration of articles, and multi-speaker dialogs (e.g., podcasts), and so forth. Despite this progress, generative acoustic models such as TTS models tend to generate speech from a single speaker at a time. As a consequence, synthetic audio, such as turn-by-tum dialogues, created by these systems are not natural or engaging for listeners. More generally, generative acoustic models tend to generate synthetic audio that may include multiple sources of sound (e.g., speaker, musical instrument, sound effect, etc.), but without any logical separation between those sounds. For example a music generation model might generate full composition of different instruments but they will be mixed together in the output, rather than separated into separate per-instrument stems.

[0003] Implementations are described herein for training and application of generative acoustic models to generate improved synthetic audio that is capable of being controlled at a more granular level. More particularly, but not exclusively, implementations are described herein for training and / or fine tuning generative acoustic models to generate synthetic audio having a plurality of distinct, time-synchronized stems of acoustic representations. Each stem may carry acoustic representations, such as acoustic tokens or latent embeddings, corresponding to a respective source (or sources) of sound of multiple distinct sources of sound. Each respectiveAttorney Docket No. GOOG-0780-WO-01 stem of acoustic representations may be a distinct collection, cluster, and / or stream of acoustic representations that is attributable (artificially or in reality) to one or more delineated sources of sound, such as a speaker, instrument s), sound source, etc.Brief Description of the Drawings

[0004] Fig. 1 schematically depicts an example environment in which disclosed techniques may be employed, in accordance with various implementations.

[0005] Fig. 2 depicts an example method for practicing selected aspects of the present disclosure.

[0006] Fig. 3 depicts an example method for practicing selected aspects of the present disclosure.

[0007] Fig. 4 depicts an example method for practicing selected aspects of the present disclosure.

[0008] Fig. 5 depicts an example method for practicing selected aspects of the present disclosure.

[0009] Fig. 6 schematically depicts an example architecture of a computer system.Attorney Docket No. GOOG-0780-WO-01Detailed Description

[0010] Generative acoustic models are advancing rapidly. Some combine neural audio compression, self-supervised representation learning, and language modeling to unlock use cases such as generation of long form content. This long form content can include, for instance, stories, narration of articles, and multi-speaker dialogs (e.g., podcasts), and so forth. Despite this progress, generative acoustic models such as TTS models tend to generate speech from a single speaker at a time. As a consequence, synthetic audio, such as turn-by-tum dialogues, created by these systems are not natural or engaging for listeners. More generally, generative acoustic models tend to generate synthetic audio that may include multiple sources of sound (e.g., speaker, musical instrument, sound effect, etc.), but without any logical separation between those sounds. For example a music generation model might generate full composition of different instruments but they will be mixed together in the output, rather than separated into separate per-instrument stems.

[0011] Implementations are described herein for training and application of generative acoustic models to generate improved synthetic audio that is capable of being controlled at a more granular level. More particularly, but not exclusively, implementations are described herein for training and / or fine tuning generative acoustic models to generate synthetic audio having a plurality of distinct, time-synchronized stems of acoustic representations. Each stem may carry acoustic representations, such as acoustic tokens or latent embeddings, corresponding to a respective source (or sources) of sound of multiple distinct sources of sound. Each respective stem of acoustic representations may be a distinct collection, cluster, and / or stream of acoustic representations that is attributable (artificially or in reality) to one or more delineated sources of sound, such as a speaker, instrument s), sound source, etc.

[0012] As used herein, “stem” may refer to a collection of sources of sound that are mixed together and treated as a single unit. Put another way, a “stem” refers to a discrete or grouped collection of sound sources mixed together. In the context of the present disclosure, acoustic representations (e.g., tokens, embeddings) of a source of sound may be grouped together, e.g., in the same interleaved position at each timestep, so that they can be manipulated independently of other, time-synchronized stems of acoustic representations. In a recording of a multi-speaker conversation, for instance, each speaker’s acoustic representations may be organized into a different stem. The same may go for musical instruments, although in some cases, acousticAttorney Docket No. GOOG-0780-WO-01 representations of multiple constituent musical instruments may be grouped together in a single stem. For example, a drum set may include multiple different drums, cymbals, and / or other percussion instruments. In some cases, multiple percussion instruments may be grouped together in a single stem (although that is not required).

[0013] Generative acoustic models configured with selected aspects of the present disclosure may be conditioned on a variety of different types of inputs (or “conditioning data”), from as simple as a natural language command (“Generate a two-minute Samba Pagode with guitar, trumpet, bass, and drums”) to more detailed and / or structured data, such as audio source segmentation data. As used herein, “audio source segmentation metadata” refers to any data that delineates between multiple distinct sources of audio, e.g., by identifying boundaries between the different sources of audio. One form of audio source segmentation metadata is a transcript of a multi-speaker audio recording that identifies boundaries between different speakers (with or without identifying / recognizing the speakers themselves). Another form of audio source segmentation metadata is musical notation, such as sheet music, that identifies the sounds that are produced (or that should be produced) by different musical instruments in ensemble recording. Yet another form of audio source segmentation metadata is data identifying separate tracks in a multitrack sound recording.

[0014] Techniques described herein may be used to generate various types of multi-sound- source synthetic audio. Each different synthetic sound source may be allocated its own stem of acoustic representations, and the multiple stems of acoustic representations may be time- synchronized. Multi-sound-source synthetic audio may include, for instance, synthetic multispeaker conversations, synthetic musical performances, synthetic audio having multiple sound effects (alone and / or in combination with voices and / or music), etc.

[0015] Generating multi-sound-source synthetic audio, with each sound source being allocated its own stem of acoustic representations, may give rise to a variety of technical benefits. When existing generative acoustic models are used to generate single-stem synthetic audio that represents multiple different sound sources in the same stream of acoustic representations, it can be difficult to isolate and manipulate individual sources of sound without affecting others. By contrast, with various implementations described herein, each distinct source of sound (e.g., speaker, instrument, sound effect, etc.) is assigned to its own output stem, and therefore can be independently controlled. For instance, in a multi-speaker synthetic conversation, one speaker’s voice can be changed, e.g., using an example voice prompt (e.g., a voice recording sample) or aAttorney Docket No. GOOG-0780-WO-01 natural language command, without impacting other speaker(s)’ voices. For example, an accent can be added to or removed from one synthetic speaker’s voice. In a multi-instrument synthetic performance, one synthetic instrument can be altered while the other instruments’ sounds are held constant.

[0016] In various implementations, each synthetic source of sound (e.g., individual speaker, musical instrument, and / or sound effect) may be modeled as a separate output stem (or“channel” or “stream”) that is generated during a single inference of one or more generative acoustic models. In some implementations that use autoregressive models, acoustic representations generated for each distinct audio source may take the form of acoustic tokens. These acoustic tokens may be concatenated or interleaved along a depth dimension. For instance, at each timestep, a residual vector quantized (RVQ) token generated for each source of sound may be concatenated with a RVQ token generated for another source of sound, such that RVQ tokens for the multiple sources of sound are effectively interleaved with each other.

[0017] Suppose each RVQ representation has three layers (e.g., a three-dimensional vector, although more or less layers are possible). Single stem audio may be represented in the autoregressive setting as follows: timestepO timestep 1 timestep2 timestep3RVQO RVQO RVQO RVQO...RVQ1 RVQ1 RVQ1 RVQ1...RVQ2 RVQ2 RVQ2 RVQ2...

[0018] At least timestep, a concatenation of RVQ tokens {RVQO, RVQ1, RVQ2} may be processed as input for the autoregressive model. With multi-stem audio, by contrast, multiple RVQ representations (each including three individual tokens) may be concatenated with each other, effectively interleaving the different audio sources, e.g., as follows: timestepO timestep 1 timestep2 timestep3. . .RV QOspeakerl RV QOspeakerl RV QOspeakerl RV QOspeakerl • • • RV Q 1 speakerl RV Q 1 speakerl RV Q 1 speakerl RV Q 1 speakerl • • • RV Q2speakerl RV Q2speakerl RV Q2speakerl RV Q2speakerl • • • RV QOspeaker? RV QOspeaker? RV Q0speaker2 RV QOspeaker? • • • RV Q 1 speaker RV Q 1 speaker? RV Q 1 speaker RV Q 1 speaker? • • • RV Q2speaker2 RV Q2speaker2 RV Q2speaker2 RV Q2speaker2 • • •Attorney Docket No. GOOG-0780-WO-01

[0019] In other implementations in which latent diffusion models are used to generate multistem output, multiple audio sources may be represented differently than using RVQ representations. In some implementations in which latent diffusion models are used, the acoustic representations may take the form of latent embeddings. In some implementations, the latent embeddings that are generated at each timestep may be concatenated along the feature dimension. This may effectively increase the dimensionality of the latent feature by the maximum number of sources. As an example, suppose a latent embedding at each timestep for a single synthetic source of sound includes 128 dimensions. If there are two synthetic sources of sound, the concatenation of their respective latent embeddings at each timestep may include 256 dimensions.

[0020] Obtaining multi-sound-source audio training data to pretrain and / or fine tune generative acoustic models to generate synthetic audio (single stem or multi-stem) can be challenging. Fully diarized audio data — which may be particularly well-suited for such training — may be difficult to obtain and / or generate. Many recorded conversations are private, and regulations and / or agreements may prohibit or limit the use of biometric analysis on audio data (e.g., speaker recognition), etc. Thus, performing absolute diarization, where an absolute identity of each speaker (e.g., their full name) is ascertained, may be infeasible at scale. Even performing more generic speaker separation-where the specific identities (e.g., names) of the speakers are not ascertained but speaker-agnostic labels are persisted for each distinct speaker (e.g., “speaker 1,” “speaker 2,” . . .) within the conversation— may require consent from all speakers.

[0021] Accordingly, some implementations described herein relate to generating training data for generative acoustic models while avoiding the challenges set forth above. In some implementations, single-sound-source audio, such as a digital recording of a single speaker, instrument, or other single source of sound, may be processed to generate synthetic multi-sound- source training data. This multi-sound-source training data can then be used to pretrain a generative acoustic model to generate multi-sound-source synthetic audio. This may be beneficial because single- sound- source audio may be more readily attainable than multi-sound- source audio.

[0022] To generate multi-sound-source training data from single-sound-source audio, sound source changes may be synthetically inserted into sound source segmentation metadata (e.g., a transcript or musical notation) corresponding to the single-sound-source audio. As a result, what was originally single-sound-source audio can be used during training as a multi -sound- sourceAttorney Docket No. GOOG-0780-WO-01 audio. For example, in a single-speaker digital audio recording, the single speaker may be assigned to one audio stem (e.g., one stream of acoustic representations); other audio source stems may be set to silent. Alternatively, synthetic speaker “turn” or “change” annotations (e.g., symbols such as “|”) may be added to a transcript of the digital recording, e.g., during pauses, etc., so that different sentences or clauses spoken by the same speaker can be artificially (e.g., synthetically) segmented across multiple different sound source stems. More generally, audio source segmentation metadata for a digital recording having any number (n) of sound sources can be synthetically annotated and / or modified to yield a greater (>n) or lesser (<n) number of sound sources.

[0023] In various implementations, multi -sound- source audio, such as a recorded multi-speaker conversation, may be processed using “speaker turn detection” to obtain speaker turn information. In some cases, the speaker turn information may identify times at which the conversation shifts between different speakers, without necessarily providing or assigning a persistent or consistent identity (generic or absolute) to each speaker. For example, a three- person conversation could be mapped to a five-person conversation, to a ten-person conversation, etc. Segments of the conversation that are attributable in reality to the same speaker may be incorrectly assigned to one or more synthetic speakers. Moreover, to preserve privacy, specific speaker recognition may not be employed without express consent of each speaker.

[0024] The speaker turn information may be used to annotate audio source segmentation metadata, namely, a transcript of the recorded multi-speaker conversation. This transcript may then be processed using the acoustic generation model (e.g., a TTS model or similar) to generate a synthetic multi-speaker conversation. The synthetic multi-speaker conversation may then be compared to the ground truth recorded multi-speaker conversation. Based on this comparison, the generative acoustic model may be trained using techniques such as back propagation, gradient descent, cross entropy, etc.

[0025] In some implementations, the generative acoustic model may be finetuned based on one or more smaller, “diarized” datasets of audio segments and associated speaker identity information. For example, there may be recorded multi-speaker conversations for which relevant parties (e.g., the speakers themselves) have consented to diarization, or in which diarization is already provided in the training data because it was collected in a source separated manner to begin with. The generative acoustic model may be used to process these datasets to generateAttorney Docket No. GOOG-0780-WO-01 synthetic audio. The synthetic audio and associated speaker identity information may then be used to train the same generative acoustic model or a different generative acoustic model. Generative acoustic models such as TTS models that are configured with selected aspects of the present disclosure may take various forms, such as a decoder-only transformer, an encoder transformer coupled with a decoder transformer via cross attention, or even an encoder-only transformer in some use cases.

[0026] The generative acoustic model may be deployed at various locations having varied computational resources, e.g., at a resource-constrained edge device and / or on the cloud. When deployed at the edge, the generative acoustic model may have fewer than one billion parameters, two billion parameters, four billion parameters, eight billion parameters, ten billion parameters, twenty-seven billion parameters, etc., but may have fewer parameters than other generative acoustic models, such as some instances of cloud-based generative acoustic models. In some cases, a cloud-based generative acoustic model may have a similar number of parameters as the generative model deployed on the edge device (or even a lower number of parameters), e.g., to limit computational costs. However, in other implementations, the cloud-based generative acoustic model may have multiple times (e.g., 2x, 4x) as many parameters as the edge-based generative acoustic model. In some such implementations, a cloud-based generative acoustic model may have many more parameters, such as more than 70 billion, more than 100 billion, more than 200 billion, more than 400 billion, or more than 1 trillion parameters.

[0027] In some implementations, either the cloud-based generative acoustic model or the edgebased generative acoustic model may be selected for application. This selection may be based on factors such as network connection quality between the edge device and synthetic audio system 100, available computational resources of the edge device (or the cloud), battery power of the edge device, and so forth. For instance, if it is determined that a network connection between client device 132 and synthetic audio system 100 is congested, weak, or otherwise in a state that is likely to introduce latency and / or be unreliable, then an edge-based generative acoustic model onboard client device 132 may be used instead.

[0028] As used herein, an input “stream” refers to a logically and / or semantically self-contained sequence of digital information, such as acoustic tokens and / or embeddings. It is not required that a stream is continuous or entirely uninterrupted. In some cases, a stream can be derived and / or sampled from analog data such as a voltage waveform generated from sound waves, although this is not required.Attorney Docket No. GOOG-0780-WO-01

[0029] While examples described herein primarily relate to generating synthetic speech output from audio source segmentation metadata (e.g., textual transcripts annotated with speaker turn annotations), this is not meant to be limiting. Techniques described herein may be applicable to generate other types of audio output from other types of audio source segmentation metadata. In the case of music, various types of visual musical notations, such as sheet music, may be processed using a generative acoustic model to generate synthetic music.

[0030] Fig. l is a schematic diagram illustrating components that can cooperate to carry out selected aspects of the present disclosure, in accordance with various implementations. The various components depicted in Fig. 1, particularly those components forming a synthetic audio system 100, may be implemented using any combination of hardware and software. The components of Fig. 1 are depicted as being communicatively coupled with each other via one or more networks 199, which may include one or more personal area networks, local area networks, and / or wide area networks (e.g., the Internet). However, this is not meant to be limiting. Various aspects of the present disclosure that are described as being performed by and / or stored on system 100 can alternatively be performed by and / or stored elsewhere and / or distributed across multiple systems, such as between system 100 and a client device 132.

[0031] In some implementations, synthetic audio system 100 may include one or more computing devices cooperating to perform selected aspects of the present disclosure. An example of such a computing device is depicted schematically in Fig. 6. In some implementations, synthetic audio system 100 may include one or more servers forming part of what is often referred to as a “cloud” infrastructure, or simply “the cloud.” Alternatively, one or more components of system 100 may be operated by client device 132.

[0032] Synthetic audio system 100 may include a prompt assembly engine 102, a generative acoustic model (GM) engine 104 with access to one or more generative acoustic models (GAM) 106, and a decoder 108. Any of elements 102-108 may be implemented using any combination of hardware and software. Moreover, any of elements 102-108 may be combined with other(s) of elements 102-108.

[0033] Generative acoustic models 106 may take various forms, including, but not limited to, model(s) such as Gemini, Flamingo, PaLM, BERT, LaMDA, Meena, and / or any other single- modal or multimodal generative model, such as any other generative model that is encoder-only based, decoder-only based, sequence-to-sequence based and that optionally includes an attention mechanism or other memory, diffusion model(s), etc. Generative acoustic models may haveAttorney Docket No. GOOG-0780-WO-01 hundreds of millions, or even hundreds of billions of parameters. In some implementations in which a generative acoustic model 106 takes the form of an autoregressive model, it may take the form of and / or be derived from the architecture / framework described in “AudioLM: a Language Modeling Approach to Audio Generation” (arXiv:2209.03143), which is hereby incorporated by reference. In some implementations in which a generative acoustic model 106 takes the form of a diffusion model. In various implementations, a user 130 may interact with synthetic audio system 100 using a client device 132. While depicted as a tablet computer or smartphone in Fig. 1, client device 132 may take other forms, such as a desktop or laptop computer, in-vehicle computing device, augmented reality (AR) and / or virtual reality (VR) headset or glasses, standalone “smart” speakers that host automated assistants that can be interacted with, etc.

[0034] In various implementations, a user 130 may operate client device 132 to provide various types of conditioning data to synthetic audio system 100 to cause prompt assembly engine 102 to assemble an input prompt 116. As will be explained in more detail below, input prompt 116 may ultimately be processed by acoustic GM engine 104 using one or more generative acoustic models 106 to generate acoustic representations 118. Acoustic representations 118 may in turn be processed by decoder 108 (as described in more detail below) to generate digital output 120 that can be rendered at one or more speakers.

[0035] Referring back to prompt assembly engine 102, in some implementations, user 130 may provide various types of conditioning data 110 that can be assembled by prompt assembly enginel02 into an input prompt 116. During inference, the conditioning data 110 may include, for instance, a request 111 to generate synthetic, multi-stem audio. For example, user 130 could issue a request such as “Generate a two-speaker debate about Hawaiian pizza.” Without more, such a prompt may ultimately result in generation of synthetic audio that includes two synthetic speakers debating the merits of Hawaiian pizza. In such an instance, both the audio attributes of the synthetic speakers and the words they speak may be entirely synthetic. Similarly, user 130 may issue a request 111 to generate synthetic music in a particular style, and without more conditioning, the resulting synthetic audio may be entirely synthetic (e.g., not only the acoustic qualities, but the melody, rhythm, etc., may all be synthetic).

[0036] Additionally or alternatively, user 130 may wish to exert greater control over the generation of synthetic audio. In some such instances, user 130 may provide, in addition to or instead of request 111, one or more instances of audio source segmentation metadata 112 to beAttorney Docket No. GOOG-0780-WO-01 assembled by prompt assembly engine 102 into input prompt 116. If the user wishes to generate synthetic audio that includes a multi-speaker conversation, then audio source segmentation metadata 112 may take the form of a text transcript of a multi-speaker conversation. This text transcript may be transcribed from a real life recorded multi-speaker conversation, may be generated synthetically using a large language model, may be composed by one or more humans, etc. Contrastly, if the user wishes to generate synthetic music, audio source segmentation metadata 112 may include visual musical notation, such as sheet music.

[0037] User 130 may optionally provide other conditioning data for assembly into input prompt 116 by prompt assembly engine 102. This other conditioning data may be included to condition generative acoustic model(s) 106 to cause one or more stems of acoustic representations 118 to have various characteristics, without necessarily affecting other stems of the acoustic representations 118. For example, in Fig. 1, user 130 provides one or more voice prompts 114 in order to condition one or more of the stems of acoustic representations to ultimately generate speech that sounds like the one or more voice prompts. A voice prompt 114 may include a small recorded or synthetic audio sample or segment (e.g., a digital audio waveform) of a particular voice. For example, user 130 may, with permission, obtain an audio sample for a particular person, such as a voice actor, that has a voice user 130 would like to incorporate into the digital output 120. The user may select a voice prompt based on, for example, a gender of the voice, a perceived age of the voice, an intonation or cadence associated with the voice, an accent associated with the voice, etc. In the context of generating synthetic music, voice prompts 114 may be replaced with musical instrument prompts, e.g., in the form of audio samples of different musical instruments being played.

[0038] In some implementations, user 130 may condition the generative acoustic model 106 on other types of data. For instance, user 130 may provide one or more natural language commands to cause particular stem(s) of acoustic representations to carry various desired characteristics. As non-limiting examples, user 130 may provide natural language commands such as “Please make speaker 2’s voice have a lower pitch” or “make speaker 3 speak with rising inflections.” In the music context, user 130 might issue a command such as “replace the bassoon with a bass saxophone” or “use xylophone instead of the marimba.”

[0039] Decoder 108 may be configured to process (e.g., detokenize) acoustic representations 118 to generate a streaming digital output 120. Digital output 120 may be used, e.g., by client device 132, to render audio output at one or more speakers (not depicted).Attorney Docket No. GOOG-0780-WO-01Depending on the use case, digital output 120 may represent (e.g., be rendered as), for instance, a synthetic multi-speaker conversation (e.g., a synthetic podcast), a translation of speech into a target language, other acoustic output e.g., automatically generated background music, rhythm section., etc. , a response from a dialogue agent, enhanced speech in a noisy environment and / or over a poor connection, etc.

[0040] Referring now to Fig. 2, an example method 200 of practicing selected aspects of the present disclosure is described. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system may include various components of various computer systems, including those depicted in Fig. 1. Moreover, while operations of method 200 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted or added.

[0041] At block 202, the system, e.g., by way of prompt assembly engine 102, may assemble an input prompt 116 that includes, as conditioning data 110, a request 111 to generate audio content comprising multiple distinct sources of sound. In some implementations, the user 130 may not submit anything else, e.g., if the user 130 does not intend to impose semantic or syntactic constraints on the synthetic audio. If the user 130 intends to exert relatively granular control over the synthetic audio, the user 130 might additionally or alternatively provide audio source segmentation metadata 112. For example, if the user 130 intends to generate synthetic audio representing a multi-speaker conversation that follows a particular script, the conditioning data 110 may further include the audio source segmentation metadata 112 may include the script (e.g., a transcript) of the desired conversation that identifies, as the distinct sources of sound, different speakers. If the user intends to generate synthetic music, the audio source segmentation metadata 112 may include musical notation, such as sheet music, that identifies, as the distinct sources of sound, different musical instruments, singers, etc.

[0042] In some implementations, other data may be assembled by prompt assembly engine 102 into input prompt 116. For example, in some implementations, input prompt 116 may be assembled by prompt assembly engine 102 to include a first voice prompt (which may or may not be provided by the user 130) that will be used to condition one of multiple distinct, time- synchronized stems of acoustic representations, and a second voice prompt to condition another of the distinct, time-synchronized stems of acoustic representations. In other implementations, input prompt 116 may be assembled by prompt assembly engine 102 to include other types of conditioning data, such as a textual prompts (e.g., “make speaker 1 have a German accent”).Attorney Docket No. GOOG-0780-WO-01

[0043] In yet other implementations, other data may be assembled into the input prompt, such as images, documents, videos, audio, etc. For instance, the user 130 could provide one or more documents (e.g., articles, technical documents, etc.) and request that synthetic audio be generated that summarizes the submitted document(s). This synthetic audio could include, for instance, a multi-speaker conversation about content of the document(s), a single speaker monologue that summarizes the document(s), etc. As another non-limiting example, the user 130 could submit a plurality of images depicting moments in time, and may request generation of multi-sound-source synthetic audio that, for instance, tells a story that incorporates the images’ visual content using multiple different characters’ or narrators’ voices, multiple different sound effects, a multi-instrument musical composition, etc.

[0044] At block 204, the system, e.g., by way of generative acoustic model engine 104, may process input prompt 116 using one or more generative acoustic models 106 to generate a plurality of distinct, time-synchronized stems of acoustic representations 118. In various implementations, each time-synchronized stem may carry or otherwise represent acoustic representations corresponding to a respective source of sound of the multiple distinct sources of sound. For example, one stem may include acoustic representations corresponding to a first speaker, another stem may include acoustic representations corresponding to a second speaker, another stem may include acoustic representations representing music (e.g., multiple instruments or a single instrument), another stem may include acoustic representations representing sound effects, etc.

[0045] As alluded to by this example, there is no requirement that the multiple stems (and the resulting synthetic audio) be limited to a single type of sound. For example, one stem could include acoustic representations of a speaker, another stem could include acoustic representations of music, another stem could include acoustic representations of sounds effects (e.g., blowing wind if the synthetic output is to include a ghost story), and so forth. And as noted previously, acoustic representations of multiple sound sources (e.g., constituent percussion instruments forming a drum set) may be grouped together in the same stem.

[0046] Multiple stems of acoustic representations may be implemented in various ways. If the generative acoustic model is an autoregressive model, then the acoustic representations may be acoustic tokens (e.g., RVQ). In some such implementations, the acoustic tokens from each stem at each timestep may be interleaved and / or concatenated with each other, e.g., along the depth dimension. This may be true at each timestep even where some of those stems are silent,Attorney Docket No. GOOG-0780-WO-01 resulting in a uniform dimension across timesteps. If the generative acoustic model is a diffusion model, on the other hand, then the acoustic representations may take the form of latent embeddings. As noted previously, in some such implementations, these latent embeddings may be concatenated with each other along the feature dimension.

[0047] At block 206, the system, e.g., by way of decoder 108, may decode the plurality of distinct, time-synchronized audio source stems of acoustic representations to generate a stream of audio output frames (e.g., digital content 120). For example, the system may detokenize the interleaved / concatenated RVQ tokens generated from an autoregressive generative acoustic model. In some implementations, this stream of audio output frames may then be used to render synthetic audio at one or more speakers.

[0048] In some implementations, method 200 may proceed to block 208. At block 208, the stream of audio output frames may be used to pretrain and / or finetune one or more of the generative acoustic models 106. For example, if the instance of audio content referenced in block 202 constitutes actual renderable audio (e.g. , a ground truth recording or synthetic audio), then the stream of output frames (or audio waveform derived therefrom) may be compared to that of the renderable audio. Based on this comparison, techniques such as back propagation may be performed to train, pretrain, and / or finetune the generative acoustic model 106.

[0049] Additionally or alternatively, in some implementations, method 200 may proceed to optional block 210. At block 210, the system may condition one of the distinct, time- synchronized stems of acoustic representations based on additional conditioning data, e.g, without conditioning or alternating other(s) of the distinct, time-synchronized stems of acoustic representations. For instance, after hearing the synthetic audio rendered based on the stream of audio output frames generated at block 206, the user may decide to modify acoustic attribute(s) of one of the stems, with or without modifying the other stems. In the multi-speaker context, for instance, the user may provide an alternative voice prompt sample of a different voice than was used during the previous iteration. During the next iteration, the generative acoustic model may be conditioned with data that causes it to generate the same or similar synthetic audio, except with the new voice replacing the old. For example, at block 212, prompt assembly engine 102 may assemble a new input prompt 116 that includes the previously generated synthetic audio, plus the alternative voice prompt sample and a request to modify one of the audio stems to sound like the voice in the alternative voice prompt sample. In some implementations, and as indicated by the arrow back to block 204, this may cause the generative acoustic model 106 toAttorney Docket No. GOOG-0780-WO-01 generate new synthetic audio having a plurality of distinct, time-synchronized stems of acoustic representations. The stem affected by the alternative voice prompt sample may be altered; the rest of the stems may remain unaltered (e.g., the prior acoustic representations may simply be passed through).

[0050] Referring now to Fig. 3, an example method 300 of pretraining / training and finetuning a generative acoustic model for generating synthetic, multi-speaker audio using multi-speaker recordings is depicted. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system may include various components of various computer systems, including those depicted in Fig. 1. Moreover, while operations of method 300 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted or added.

[0051] At block 302, the system may obtain a first dataset of digital audio segments. Each digital audio segment of the first dataset may include a digital recording of multiple speakers. At block 304, the system may perform speaker change detection on the digital audio segments of the first dataset to obtain speaker turn information. The speaker turn information may indicate, for each digital audio segment, one or more points in time within the digital audio segment at which there is a transition (or “turn”) between speakers. For example, in a transcript of the digital recording, a special symbol (e.g., “|”) may be used to represent a transition or turn between speakers.

[0052] However, due to rules and / or agreements which prohibit biometric processing of audio data without consent, in various implementations, the speaker turn information may not provide any additional information about the speakers. For example, the speaker turn information may not assign persistent identities to the speakers, whether absolute identities (e.g., “Mavis Powell,” “Delia Jones”) or agnostic / generic identities (e.g., “speaker 1,” “speaker 2,” . . .). Instead, in some implementations, each different turn of speaker may simply be identified as a different random speaker, even if multiple speaker turns are in fact attributable to the same speaker.

[0053] Referring back to Fig. 3, at block 306, the system may train (or pretrain) a generative acoustic model 106 using the first dataset and the speaker turn information. For example, prompt assembly engine 102 may assemble input prompt 116 to include the transcripts annotated with speaker turns, and data indicative of the audio segments. This input prompt 116 may be processed by generative acoustic model engine 104 using generative acoustic model 106 to generate acoustic representations 118. The acoustic representations 118 may be decoded (e.g.,Attorney Docket No. GOOG-0780-WO-01 detokenized) by decoder 108 as described previously to generate synthetic digital audio output 120. The synthetic digital audio output 120 may then be compared to the original input audio segment. Based on the comparison, techniques such as back propagation may be performed to train the generative acoustic model 106.

[0054] In some implementations, and as described in the AudioLM paper referenced previously, data indicative of an audio segment may be generating by encoding the audio segment (e.g., audio frames, a digital waveform) in parallel into semantic tokens that represent attributes such as linguistic content and intonation, and acoustic tokens that represent other attributes, such as recording conditions, acoustic quality, speaker identity (if consent provided), etc. In some implementations, the semantic tokens may be generated using techniques such as those described in “W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training” (arXiv:2108.06209), which is incorporated herein by reference. In some implementations, the acoustic tokens may be generated using techniques such as those described in SoundStream: An End-to-End Neural Audio Codec” (arXiv:2107.03312), which is incorporated herein by reference.

[0055] At block 308, the system may obtain a second dataset of audio segments and associated speaker identity information. Each digital audio segment of the second dataset may once again include a recording of multiple speakers, but in this instance, the speaker identity information may either persist generic / agnostic identities of the speakers (e.g., “speaker 1,” “speaker 2,” . . .) or it may actually include (with the consent of the speakers) absolute speaker identities (e.g., it may be diarized). For example, the speaker identity information may include a transcript with each utterance labeled with the respective speaker.

[0056] At block 310, the system may train or finetune the generative acoustic model 106 using the second dataset and associated speaker identity information. For example, prompt assembly engine 102 may once again assemble an input prompt 116, this time to include the speaker identity information (e.g., transcript annotated with absolute or generic speaker identifiers) and data indicative of audio segment(s) of the second dataset. This new input prompt 116 may be processed by generative acoustic model engine 104 using generative acoustic model 106 to generate acoustic representations 118. The acoustic representations 118 may be decoded by decoder 108 as described previously to generate synthetic digital audio output 120. The synthetic digital audio output 120 may then be compared to the original input audio segment.Attorney Docket No. GOOG-0780-WO-01Based on the comparison, techniques such as back propagation may be performed to finetune the generative acoustic model 106.

[0057] In some implementations, additional synthetic audio, such as a fully diarized synthetic multi-speaker conversation, may be generated and used as additional training data. At block 312, for instance, the system may obtain additional speaker identity information, such as a speaker- annotated transcript of a conversation (recorded, synthesized, human composed), and a third dataset of synthetic audio segments, e.g., by using the generative acoustic model 106 to generate the synthetic audio segments (in which case the additional speaker identity information may be used to generate the synthetic audio segment and therefore may be known already, or may be derived from the synthetic audio segment using a diarization process). Each synthetic audio segment of the third dataset may be usable to render audio output comprising multiple speakers. Because the third dataset is synthetic, the additional speaker identity information is already known, and no speaker consent may be necessary (since the speakers may be synthetic creations). Accordingly, at block 314, the system may train or finetune the same acoustic generative acoustic model or a different generative acoustic model using the synthetic audio and the additional speaker identity information, similar to as described above.

[0058] Referring now to Fig. 4, an example method 400 of pretraining / training and finetuning a generative acoustic model for generating synthetic audio is depicted. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system may include various components of various computer systems, including those depicted in Fig. 1. Moreover, while operations of method 400 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted or added. Many of the operations of method 400 are similar to those of method 300, except applicable to synthetic audio beyond multi-speaker synthetic audio.

[0059] At block 402, the system may obtain a first dataset of audio segments. Each audio segment may include a recording of multiple sources of sound. These sources of sound could include multiple speakers, but could also include, for instance, multiple instruments, multiple sound effects, or any combination thereof.

[0060] At block 404, the system may perform source change detection on the audio segments of the first dataset to obtain sound source transition information. The sound source transition information may indicate, for each audio segment, one or more transitions between sources of sound. These transitions may be temporal and / or otherwise. For example, in a multi-speakerAttorney Docket No. GOOG-0780-WO-01 context, speakers may, in general (though not always), wait to speak until someone else is done speaking. But transitions are not limited to temporal transitions. For example, different people may speak over each other at the same time. Additionally, different musical instruments may be played simultaneously (and may be isolated from each other in some instances based on tracks of the digital recording). In various implementations, the sound source transition information does not assign persistent identities to the sources of sound of the audio segment of the first dataset.

[0061] At block 406, the system may train (or pretrain) a generative acoustic model using the first dataset and the source transition information, similar to block 306 of Fig. 3. At block 408, the system may obtain a second dataset of audio segments and associated sound source identity information. Similar to block 308, at block 408, each audio segment of the second dataset may include a recording of multiple sources of sound. In this instance, the sound source identity information may assign persistent identities to the sources of sound of the audio segment of the second dataset. Similar to block 310, at block 410, the system may finetune the generative acoustic model using the second dataset and the associated sound source identity information.

[0062] At block 412, the system may obtain additional sound source identity information and a third dataset of synthetic audio segments, e.g., by using the finetuned generative acoustic model to generate the synthetic audio segments. Each synthetic audio segment of the third dataset may be usable to render audio output comprising multiple sources of sound. The additional sound source identity information may be known up front, e.g., if used to generate a synthetic audio segment, or it may be derived from the synthetic audio segment using a diarization process. At block 414, similar to block 314, the system may train or finetune the same generative acoustic model or a different generative acoustic model using the third dataset of synthetic audio segments and additional sound source identity information associated with the third dataset.

[0063] Techniques described herein are not limited to generation of synthetic audio. In various implementations, techniques described herein may be used to generate any type of synthetic data, such as videos, images, documents, or even trajectories of geographic locations. For example, a different trajectory of geographic locations may be accumulated for each rider of a rideshare driver’s shift. These different trajectories can be used, e.g., without persisting identities of the riders, to train generative models as described herein to generate realistic synthetic data that simulates rideshare usage.Attorney Docket No. GOOG-0780-WO-01

[0064] Referring now to Fig. 5, an example method 500 of pretraining / training and finetuning a generative model for generating synthetic data is depicted. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system may include various components of various computer systems, including those depicted in Fig.1. Moreover, while operations of method 500 are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted or added. Many of the operations of method 500 are similar to those of methods 300 and 400, except applicable to any synthetic data.

[0065] At block 502, the system may obtain a first dataset of data segments, each data segment of the first dataset comprising data from multiple sources, similar to blocks 302 and 402. At block 504, the system may obtain transition information associated with the first dataset. The transition information may indicate, for each data segment of the first dataset, transitions between sources without identifying one or more sources to which the transition relates. As in previous implementations, these transitions are not limited to transitions in time, and could include transitions between different, simultaneous data sources. At block 506, the system may train (or pretrain) a generative model using the first dataset and the transition information.

[0066] At block 508, the system may obtain a second dataset of data segments and associated data source identity information, similar to blocks 308 and 408. Each data segment of the second dataset may include data from multiple sources, such as multiple sources of visual data, multiple geographic location trajectories associated with (e.g., accumulated by) multiple different mobile devices, etc. At block 510, similar to blocks 310 and 410, the system may finetune the generative model using the second dataset and the associated data source identity information.

[0067] At block 512, similar to blocks 312 and 412, the system may obtain additional data source identity information and a third dataset of synthetic data segments. In some implementations, this may include generating the third dataset of synthetic data segments using the finetuned generative model. At block 514, the system may train or finetune the same generative model or a different generative model using the third dataset of synthetic data segments and the additional data source identity information.

[0068] Fig. 6 is a block diagram of an example computer system 610. Computer system 610 typically includes at least one processor 614 which communicates with a number of peripheral devices via bus subsystem 612. These peripheral devices may include a storage subsystem 624, including, for example, a memory subsystem 625 and a file storage subsystem 626, userAttorney Docket No. GOOG-0780-WO-01 interface output devices 620, user interface input devices 622, and a network interface subsystem 616. The input and output devices allow user interaction with computer system 610. Network interface subsystem 616 provides an interface to outside networks and is coupled to corresponding interface devices in other computer systems.

[0069] User interface input devices 622 may include a keyboard, pointing devices such as a mouse, trackball, touchpad, or graphics tablet, a scanner, a touch screen incorporated into the display, audio input devices such as voice recognition systems, microphones, and / or other types of input devices. In general, use of the term "input device" is intended to include all possible types of devices and ways to input information into computer system 610 or onto a communication network.

[0070] User interface output devices 620 may include a display subsystem, a printer, a fax machine, or non-visual displays such as audio output devices. The display subsystem may include a cathode ray tube (CRT), a flat-panel device such as a liquid crystal display (LCD), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide non-visual display such as via audio output devices. In general, use of the term "output device" is intended to include all possible types of devices and ways to output information from computer system 610 to the user or to another machine or computer system.

[0071] Storage subsystem 624 stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem 624 may include the logic to perform selected aspects of methods 200-500, and / or to implement one or more aspects of the various components depicted in Fig. 1. Memory 625 used in the storage subsystem 624 can include a number of memories including a main randomaccess memory (RAM) 630 for storage of instructions and data during program execution and a read only memory (ROM) 632 in which fixed instructions are stored. A file storage subsystem 626 can provide persistent storage for program and data files, and may include a hard disk drive, a CD-ROM drive, an optical drive, or removable media cartridges. Modules implementing the functionality of certain implementations may be stored by file storage subsystem 626 in the storage subsystem 624, or in other machines accessible by the processor(s) 614.

[0072] Bus subsystem 612 provides a mechanism for letting the various components and subsystems of computer system 610 communicate with each other as intended. Although bus subsystem 612 is shown schematically as a single bus, alternative implementations of the bus subsystem may use multiple buses.Attorney Docket No. GOOG-0780-WO-01

[0073] Computer system 610 can be of varying types including a workstation, server, computing cluster, blade server, server farm, smart phone, smart watch, smart glasses, set top box, tablet computer, laptop, or any other data processing system or computing device. Due to the everchanging nature of computers and networks, the description of computer system 610 depicted in Fig. 6 is intended only as a specific example for purposes of illustrating some implementations. Many other configurations of computer system 610 are possible having more or fewer components than the computer system depicted in Fig. 6.

[0074] Various implementations may include a transitory or non-transitory computer readable storage medium storing instructions executable by a processor to perform a method such as one or more of the methods described above. Yet another implementation may include a control system including memory and one or more processors operable to execute instructions, stored in the memory, to implement one or more modules or engines that, alone or collectively, perform a method such as one or more of the methods described above.

[0075] It should be appreciated that all combinations of the foregoing concepts and additional concepts described in greater detail herein are contemplated as being part of the subject matter disclosed herein. For example, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the subject matter disclosed herein.

[0076] In situations in which the systems described herein collect or otherwise monitor personal information about users (or make use of personal and / or monitored information), the users may be provided with an opportunity to control whether programs or features collect user information (e.g., information about a user’s social network, social actions or activities, profession, a user’s preferences, or a user’s current geographic location), or to control whether and / or how to receive content from the content server that may be more relevant to the user. Also, certain data may be treated in one or more ways before it is stored or used, so that personal identifiable information is removed. For example, a user’s identity may be treated so that no personal identifiable information can be determined for the user, or a user’s geographic location may be generalized where geographic location information is obtained (such as to a city, ZIP code, or state level), so that a particular geographic location of a user cannot be determined. Thus, the user may have control over how information is collected about the user and / or used.

[0077] While several implementations have been described and illustrated herein, a variety of other means and / or structures for performing the function and / or obtaining the results and / or one or more of the advantages described herein may be utilized, and each of such variations and / orAttorney Docket No. GOOG-0780-WO-01 modifications is deemed to be within the scope of the implementations described herein. More generally, all parameters, dimensions, materials, and configurations described herein are meant to be exemplary and that the actual parameters, dimensions, materials, and / or configurations will depend upon the specific application or applications for which the teachings is / are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific implementations described herein. It is, therefore, to be understood that the foregoing implementations are presented by way of example only and that, within the scope of the appended claims and equivalents thereto, implementations may be practiced otherwise than as specifically described and claimed. Implementations of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent, is included within the scope of the present disclosure.

Claims

Attorney Docket No. GOOG-0780-WO-01CLAIMSWhat is claimed is:

1. A method implemented using one or more processors and comprising: assembling an input prompt that includes, as conditioning data, a request to generate audio content comprising multiple distinct sources of sound; processing the input prompt using one or more generative acoustic models to generate a plurality of distinct, time-synchronized stems of acoustic representations, each stem carrying acoustic representations corresponding to a respective source of sound of the multiple distinct sources of sound; and decoding the plurality of distinct, time-synchronized audio source stems of acoustic representations to generate a stream of audio output frames.

2. The method of claim 1, wherein the plurality of distinct, time-synchronized stems comprises acoustic representations corresponding to a first stem interleaved with acoustic representations corresponding to a second stem.

3. The method of claim 2, wherein the acoustic representations are interleaved in a depth dimension.

4. The method of claim 1, wherein the plurality of distinct, time-synchronized stems comprises acoustic representations corresponding from a first stem of the plurality of distinct, time-synchronized stems concatenated with acoustic representations from to a second stem of the plurality of distinct, time-synchronized stems.

5. The method of claim 4, wherein the acoustic representations are concatenated in a depth dimension.

6. The method of any of the preceding claims, wherein the acoustic representations comprise residual vector quantized (RVQ) tokens.

7. The method of any of the preceding claims, wherein the one or more generative acoustic models comprise an autoregressive model.

8. The method of any of the preceding claims, wherein the one or more generative acoustic models comprise a latent diffusion model.

9. The method of claim 8, wherein the acoustic representations comprise latent space embeddings, and the latent space embeddings are concatenated along a feature dimension.Attorney Docket No. GOOG-0780-WO-0110. The method of any of the preceding claims, wherein the one or more generative acoustic models comprise a single machine learning model.

11. The method of any of the preceding claims, wherein the plurality of distinct, time-synchronized stems of acoustic representations are generated during a single inference of the one or more generative acoustic models.

12. The method of any of the preceding claims, further comprising conditioning one of the distinct, time-synchronized stems of acoustic representations using a first voice prompt.

13. The method of claim 12, further comprising conditioning another of the distinct, time-synchronized stems of acoustic representations using a second voice prompt.

14. The method of claim 12, further comprising conditioning another of the distinct, time-synchronized stems of acoustic representations using a textual prompt.

15. The method of any of the preceding claims, wherein the audio content comprises a multi-speaker dialog.

16. The method of any of the preceding claims, wherein the request comprises audio source segmentation metadata identifying the multiple distinct sources of sound.

17. The method of claim 16, wherein the audio source segmentation metadata comprises a textual transcript of the dialog that includes speaker turn annotations.

18. The method of any of claims 16-17, wherein the audio content comprises a multiinstrument musical recording.

19. The method of claim 18, wherein the audio source segmentation metadata comprises visual musical notation.

20. The method of claim 19, wherein the visual musical notation comprises sheet music.

21. The method of claim 1, further comprising pretraining or fine tuning one or more of the generative acoustic models by comparing the stream of audio output frames with the digital audio content.

22. A method implemented using one or more processors and comprising: assembling an input prompt that includes audio source segmentation metadata identifying multiple distinct sources of audio in a digital audio content; processing the input prompt using one or more generative acoustic models to generate a plurality of distinct, time-synchronized stems of acoustic representations, each stem carryingAttorney Docket No. GOOG-0780-WO-01 acoustic representations corresponding to a respective source of audio of the multiple distinct sources of audio; decoding the plurality of distinct, time-synchronized audio source stems of acoustic representations to generate a stream of audio output frames; and pretraining or fine tuning one or more of the generative acoustic models by comparing the stream of audio output frames with the digital audio content.

23. A method implemented using one or more processors, comprising: obtaining a first dataset of digital audio segments, each digital audio segment of the first dataset comprising a recording of multiple speakers; performing speaker change detection on the digital audio segments of the first dataset to obtain speaker turn information, wherein the speaker turn information indicates, for each digital audio segment, one or more points in time within the digital audio segment at which there is a transition between speakers, wherein the speaker turn information does not assign persistent identities to the speakers; training a generative acoustic model using the first dataset and speaker turn information; obtaining a second dataset of audio segments and associated speaker identity information, each digital audio segment of the second dataset comprises a recording of multiple speakers; finetuning the generative acoustic model using the second dataset and associated speaker identity information; obtain additional speaker identity information and a third dataset of synthetic audio segments using the generative acoustic model, each synthetic audio segment of the third dataset being usable to render audio output comprising multiple speakers; and training or finetuning the same generative acoustic model or a different generative acoustic model using the synthetic audio and the additional speaker identity information associated with the third dataset.

24. The method of claim 23, wherein the first dataset is larger than the second or third dataset.

25. The method of claim 23 or 24, wherein the generative acoustic model comprises a text-to-speech (TTS) model.Attorney Docket No. GOOG-0780-WO-0126. The method of any of claims 23-25, wherein the additional speaker identity information comprises a transcript of a multi-speaker conversation with each speaker persistently identified across the multi-speaker conversation.

27. The method of claim 26, wherein the transcript comprises a synthetic transcript generated using a large language model.

28. The method of claim 26, wherein the additional speaker identity information associated with the third dataset is derived using one or more machine learning models trained for audio diarization.

29. The method of any of claims 23-28, wherein the additional speaker identity information associated with the third dataset is received with a request to generate the third dataset, and is used to generate at least some audio segments of the third dataset.

30. A method implemented using one or more processors, comprising: obtaining a first dataset of audio segments, each audio segment comprising a recording of multiple sources of sound; performing source change detection on the audio segments of the first dataset to obtain sound source transition information, wherein the sound source transition information indicates, for each audio segment, one or more points in time within the audio segment at which there is a transition between sources of sound, and wherein the sound source transition information does not assign persistent identities to the sources of sound of the audio segment of the first dataset; training a generative acoustic model using the first dataset and the source transition information; obtaining a second dataset of audio segments and associated sound source identity information, each audio segment of the second dataset comprising a recording of multiple sources of sound, wherein the sound source identity information assigns persistent identities to the sources of sound of the audio segment of the second dataset; finetuning the generative acoustic model using the second dataset and the associated sound source identity information; generating a third dataset of synthetic audio segments using the finetuned generative acoustic model, each synthetic audio segment of the third dataset being usable to render audio output comprising multiple sources of sound; andAttorney Docket No. GOOG-0780-WO-01 training or finetuning the same generative acoustic model or a different generative acoustic model using the third dataset of synthetic audio segments and additional sound source identity information associated with the third dataset.

31. The method of claim 30, wherein the first dataset is larger than the second or third dataset.

32. The method of claim 30 or 31, wherein the additional sound source identity information associated with the third dataset is derived using one or more machine learning models trained for audio diarization.

33. The method of claim 30, wherein the additional speaker identity information associated with the third dataset is received with a request to generate the third dataset.

34. A method implemented using one or more processors, comprising: obtaining a first dataset of data segments, each data segment of the first dataset comprising data from multiple sources; obtaining transition information, wherein the transition information indicates, for each data segment, one or more points within the data segment where there is a transition between sources without identifying one or more sources to which the transition relates; training a generative model using the first dataset and the transition information; obtaining a second dataset of data segments and associated data source identity information, each data segment of the second dataset comprising data from multiple sources; finetuning the generative model using the second dataset and the associated data source identity information; obtaining additional data source identity information and a third dataset of synthetic data segments, wherein the additional data source identity information is associated with the third dataset of synthetic data segments; and training or finetuning the same generative model or a different generative model using the third dataset of synthetic data segments and the additional data source identity information.

35. The method of claim 34, wherein obtaining the third dataset comprises generating the third dataset of synthetic data segments using the finetuned generative model.

36. The method of claim 34 or 35, wherein each data segment comprises audio content, wherein the audio content comprises multiple distinct sources of sound.

37. The method of any of claims 34-36, wherein each data segment comprises video content, wherein the video content comprises multiple distinct sources of visual features.Attorney Docket No. GOOG-0780-WO-0138. The method of any of claims 34-37, wherein each data segment comprises one or more digital images.

39. The method of any of claims 34-38, wherein each data segment comprises one or more geographic locations associated with an entity, and the multiple sources comprise multiple geographic location trajectories associated with the entity.

40. A method implemented using one or more processors, comprising: obtaining a first dataset of segmented data; obtaining transition information, wherein the transition information indicates, for each segment of the segmented data of the first dataset, one or more points within the segment where there is a transition between entities without identifying one or more entities to which the transition relates; training a generative model using the first dataset and the transition information; obtaining a second dataset of segmented data and associated entity identity information, each segment of the segmented data of the second dataset comprising data corresponding to multiple entities; finetuning the generative model using the second dataset and the associated entity identity information; obtaining additional entity identity information and a third dataset of synthetic segmented data, wherein the additional entity identity information is associated with the third dataset of synthetic segmented data; and training or finetuning the same generative model or a different generative model using the third dataset of synthetic segmented data and the additional entity identity information.

41. The method of claim 40 wherein obtaining the third dataset comprises generating the third dataset of synthetic segmented data using the finetuned generative model.

42. The method of claim 40 or 41, wherein each segment of the segmented data of the first, second, and / or third datasets comprises audio content, wherein the audio content comprises multiple distinct sources of sound.

43. The method of any of claims 40-42, wherein each segment of the segmented data of the first, second, and / or third datasets comprises video content, wherein the video content comprises multiple distinct sources of visual features.

44. The method of any of claims 40-43, wherein each segment of the segmented data of the first, second, and / or third datasets comprises one or more digital images.Attorney Docket No. GOOG-0780-WO-0145. The method of any of claims 40-44, wherein each segment of the segmented data of the first, second, and / or third datasets comprises a trajectory of geographic locations associated with a different entity.

46. The method of claim 45, wherein the trajectory of geographic locations associated with a given entity comprises a trajectory of a rideshare passenger.

47. A system comprising one or more processors and memory storing instructions that, in response to execution by the one or more processors, cause the one or more processors to perform the method of any of claims 1-46.

48. At least one transitory or non-transitory computer-readable medium comprising instructions that, in response to execution by one or more processors, cause the one or more processors to perform the method of any of claims 1-46.

Images