Speech Codec Generative Method for Adverse Condition Enhancement

Solution Overview

Problem

Existing deep learning-based speech enhancement methods struggle to effectively remove background noise and reverberation in real-world scenarios, often resulting in enhanced speech that retains noise or distortion.

Innovation Solution

A neural-network pipeline comprising a convolutional network followed by a deep complex convolutional recurrent network (DCCRN) that utilizes a mel spectrogram and self-supervised learning (SSL) features to enhance audio signals, with the option to use a HifiGAN vocoder for final speech resynthesis.

Engineering Contradictions & Design Principles

VSEngineering Contradiction Analysis

1Measurement precision

If deep neural networks are used to suppress background noise and reverberation, then speech enhancement performance is improved, but performance notably deteriorates in real-world complicated scenarios

Engineering Contradiction:

Improvespeech enhancement performanceVSAvoidperformance in real-world scenarios

Core Design Contradiction:

Measurement precisionVSReliability

Solution Approach 1:

The patent introduces code tokens as an intermediary representation between the noisy speech input and the enhanced speech output. Instead of directly mapping noisy speech to enhanced speech, the system first encodes the input into code tokens using a speech codec, then processes these tokens through a transformer model, and finally decodes them back to speech. This intermediary representation captures essential speech characteristics while being more robust to noise and reverberation, resolving the contradiction between achieving high enhancement performance and maintaining reliability in real-world scenarios.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent transforms the speech signal from the time domain to a discrete code token space, fundamentally changing the representation parameters. By operating in this transformed space rather than directly on raw speech waveforms or spectrograms, the system achieves better robustness to environmental disturbances while maintaining enhancement effectiveness. The code tokens provide a compressed, semantically meaningful representation that is less sensitive to noise and reverberation.

Inventive Principle:

Principle #35Parameter changes

2Ease of manufacture

If direct speech signal representations are used for enhancement, then mapping from noisy inputs to clean targets is established, but performance deteriorates in complicated scenarios with noise or reverberation retention

Engineering Contradiction:

Improveenhancement mapping establishmentVSAvoidenhanced speech quality

Core Design Contradiction:

Ease of manufactureVSManufacturing precision

Solution Approach 1:

The patent introduces code tokens as an intermediary representation between the noisy speech input and the enhanced speech output. Instead of directly mapping noisy speech to enhanced speech, the system first encodes the input into code tokens using a speech codec, then processes these tokens through a transformer model, and finally decodes them back to speech. This intermediary representation captures essential speech characteristics while being more robust to noise and reverberation, resolving the contradiction between achieving high enhancement performance and maintaining reliability in real-world scenarios.

Inventive Principle:

Principle #24Intermediary (Mediator)

Solution Approach 2:

The patent replaces traditional direct signal processing approaches with a discrete code token-based transformation system. Instead of operating directly on continuous speech signals or spectrograms, the system uses speech codecs to convert signals into discrete code sequences, which are then processed by transformer models. This substitution of the processing mechanism operates in a discrete mathematical space rather than continuous signal space, providing better robustness and precision in enhancing speech quality under adverse conditions.

Inventive Principle:

Principle #28Mechanics substitution (Replace mechanical system)

Data Source

PatentUS20250140265A1Speech codec based generative method for speech enhancement in adverse conditions

Publication Date: 2025.05.01 TENCENT AMERICA LLC

AI summary

A method and apparatus comprising computer code configured to cause a processor or processors to receive an audio signal obtained from a microphone, input the audio signal into a neural-network pipeline, the neural-network pipeline including a convolutional network that receives the audio signal and provides a first output of the convolutional network to an enhancer, the enhancer including a deep complex convolutional recurrent network that receives the first output along with a mel spectrogram of the audio signal and outputs a second output to at least one of a vocoder and a decoder, and control an output of an enhanced audio signal from the at least one of the vocoder and the decoder.