Neural Speech Generation from Noisy Audio with Dual Encoders
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural networks struggle to accurately generate high-quality speech from low-quality audio data contaminated with noise and distortion due to poor recording equipment or environmental factors, making it difficult to understand speech.
Innovation Solution
A neural network system comprising a spectrogram encoder, context generator, waveform encoder, inferencing module, and waveform decoder processes low-quality audio data by encoding spectral and phase features, utilizing a reference audio signal to generate high-quality speech signals by learning desired characteristics and adjusting features accordingly.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Manufacturing precision
If neural networks are used to generate speech from low-quality audio data, then speech quality can be enhanced, but the accuracy of generating high-quality speech from noisy audio data remains insufficient
Solution Approach 1:
The system segments the speech generation process into multiple specialized encoders (spectrogram encoder, waveform encoder) and a separate context generator. Each encoder processes different features (spectral vs. phase/temporal) independently, allowing the system to address specific limitations of individual encoding approaches while maintaining overall speech quality and accuracy.
Solution Approach 2:
The system transforms the speech enhancement problem from a single-dimensional quality improvement task into a multi-dimensional process by separately processing spectral features and phase/temporal features through different encoders, then integrating them in the latent space. This dimensional separation allows independent optimization of each feature type to overcome the accuracy limitations of unified approaches.
2Manufacturing precision
If neural networks process low-quality audio data, then speech can be generated, but the process takes considerable time
Solution Approach 1:
The system divides the audio processing task into parallel segments: the spectrogram encoder processes spectral information while the waveform encoder handles phase and temporal information simultaneously. This parallel segmentation of processing tasks reduces the sequential processing time while maintaining the quality improvements achieved through multi-feature analysis.
Solution Approach 2:
The context generator performs preliminary action by pre-processing reference audio data to extract and store contextual information about desired speech characteristics before the actual speech generation occurs. This preliminary extraction of spectral and phase context allows the inferencing module to make faster, more accurate decisions during real-time speech reconstruction without re-processing reference data.
Data Source
AI summary
Apparatuses, systems, and techniques to generate audio of speech. In at least one embodiment, a processor uses one or more neural networks to generate first audio of speech based, at least in part, on second audio of speech and reference audio.