Autoregressive Neural Audio Generation with Acoustic Token Quantization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing audio generation systems struggle to produce coherent and high-quality audio without textual annotations, particularly in generating speech that preserves speaker identity and recording conditions, and often require extensive computational resources for training and inference.
Innovation Solution
A system utilizing generative neural networks to generate audio signals by obtaining semantic representations and processing them through a decoder neural network, which includes a hierarchy of vector quantizers to produce acoustic tokens, allowing for efficient and coherent audio synthesis.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If existing audio generation systems are used without textual annotations, then audio generation can proceed without text input, but the coherence and quality of generated speech deteriorates, particularly in preserving speaker identity and recording conditions
Solution Approach 1:
The patent introduces acoustic tokens as intermediary representations that bridge the gap between raw audio input and generated speech. These tokens capture acoustic properties (speaker identity, recording conditions) and serve as conditional inputs to the generation model, enabling high-quality speech synthesis without requiring explicit textual annotations about the audio characteristics
Solution Approach 2:
The patent segments the audio generation process into distinct components: extracting acoustic tokens from reference audio, generating semantic tokens from text, and combining them in the generation model. This segmentation allows the system to handle different aspects of speech generation separately, maintaining coherence while operating without textual annotations about acoustic properties
2Manufacturing precision
If existing audio generation systems process audio at high sample rates, then audio quality is improved, but computational resources required for training and inference increase
Solution Approach 1:
The patent transforms the audio processing problem from the time domain to a token space dimension. By converting audio samples into acoustic tokens and text into semantic tokens, the system operates in a compressed representational space that captures essential information with fewer computational operations, maintaining audio quality while reducing processing demands
Solution Approach 2:
The patent changes the parameter representation from raw audio samples to discrete acoustic tokens. This parameter transformation allows the model to work with compressed representations that retain essential acoustic information (speaker identity, recording conditions) while requiring significantly fewer computational resources for processing and generation
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating a prediction of an audio signal. One of the methods includes receiving a request to generate an audio signal; obtaining a semantic representation of the audio signal; generating, using one or more generative neural networks and conditioned on at least the semantic representation, an acoustic representation of the audio signal; and processing at least the acoustic representation using a decoder neural network to generate the prediction of the audio signal.