47 results about "Speech reconstruction" patented technology

The invention provides a voice reconstruction method and system based on entropy coding residual quantization and frequency spectrum restoration, and relates to the technical field of artificial intelligence voice signal processing, and the method comprises the steps: obtaining an original voice waveform, inputting the voice waveform into a neural voice coding and decoding model, firstly entering a coder to map the input voice waveform into acoustic potential representation, and then entering a frequency spectrum restoration model; performing residual quantization on the acoustic potential characterization layer by layer through a residual vector quantization module, introducing a gating-based dynamic layer number selection mechanism and entropy regularization constraint, enabling bits to be adaptively distributed among different voice segments, reconstructing reconstructed acoustic features of the potential characterization, inputting the reconstructed acoustic features into a decoder, restoring the reconstructed acoustic features into a time domain waveform, and outputting the time domain waveform. And mapping to a logarithmic magnitude spectrum domain through a spectrum repairing module, predicting a residual error in the logarithmic magnitude spectrum domain and performing confidence gating fusion to obtain a complex spectrum, and outputting after time domain synthesis to obtain reconstructed speech. According to the invention, high fidelity, intelligibility and transmission reliability of the voice can be considered at an extremely low bit rate.

The invention provides a multi-mode dysarthria speech reconstruction system and a multi-mode dysarthria speech reconstruction method. The system comprises an encoder and a decoder. The encoder is configured to: extract a multi-modal feature from a multi-modal input of dysarthria speech and generate phoneme embedding based on the multi-modal feature; predicting a sequence of phoneme durations based on the phoneme embedding and generating an extended phoneme embedding and a predicted tone sequence; and extracting a speaker identity characterization of the dysarthria speech speaker. The decoder is configured to reconstruct a normal speech waveform corresponding to dysarthria speech with the extended phoneme embedding, the predicted tone sequence, and the speaker identity representation as inputs.

The invention relates to the technical field of artificial intelligence, can be applied to the fields of financial science and technology and medical science and technology, and discloses a voice coding and decoding method, device, equipment and medium. A continuous vector is obtained by encoding the data through an encoder trained by a first-stage mirror image architecture; segmenting a continuous vector into sub-vectors, matching the sub-vectors with corresponding sub-coding dictionaries through a quantizer to obtain sub-indexes, and combining the sub-indexes to generate an overall index; analyzing the overall index to obtain sub-indexes, calling sub-discrete vectors and splicing the sub-discrete vectors into a discrete vector; and reconstructing the discrete vector through a decoder trained by a second-stage non-mirror-image architecture to obtain a target speech spectrum feature, and converting the target speech spectrum feature into a target speech signal. According to the method, double-stage training is adopted, the first-stage mirror image architecture guarantees the coding stability, the second-stage non-mirror image architecture improves the decoding flexibility, and the product quantization technology is combined, so that the calculation efficiency and the storage overhead are balanced, and meanwhile, the voice reconstruction quality is improved.

The embodiment of the invention belongs to the technical field of research and development design and audio processing, is applied to an audio reconstruction result evaluation scene, and relates to a voice reconstruction result evaluation method, device and equipment and a storage medium thereof. Therefore, the effect of converting the source voice into the target voice can be evaluated from the main information part or the pronunciation emphasizing part in the voice. According to the method, the voice reconstruction result is evaluated by only combining related accent features and neglecting slightly-affected genuine features, so that the feature data volume during result evaluation is reduced, it can be ensured that evaluation focuses on key features, reconstruction result evaluation is performed by only using relatively important voice features, and the reconstruction evaluation efficiency is improved. When the method is applied to an intelligent customer service voice reconstruction scene, the intelligent customer service reconstruction voice with the quality reaching the standard can be sent to a related target customer more quickly, and the service efficiency is improved.

The embodiment of the invention provides an anonymization privacy protection method and system for voice information retention. The method comprises the following steps: extracting speaker embedding of an original audio, eliminating the tone of the speaker, and keeping semantic and rhythm speaker irrelevant features; embedding and inputting a speaker into a speaker anonymous module matched with a three-stage stream based on a U-Net architecture to obtain anonymous embedding; and combining irrelevant features of the speaker with anonymous embedding by using a pre-trained voice reconstruction model to generate anonymized voice with tone privacy. According to the embodiment of the invention, voice anonymization facing content privacy and tone privacy reserved by voice information is realized, the effectiveness of the anonymized voice generated by using the method in a downstream task is superior to that of a baseline model, meanwhile, the privacy of a speaker is also guaranteed, and safe use of data is realized.

The invention relates to the field of artificial intelligence, medical treatment and financial science and technology, and discloses a voice synthesis method and device based on accent control, equipment and a medium, and the method comprises the steps: obtaining a to-be-synthesized target text and reference voice; inputting the target text into a text encoder to obtain a corresponding target phoneme sequence; inputting the reference voice into a voice encoder for voice encoding and feature separation processing to obtain separated reference speaker features and reference accent features; in response to a voice synthesis control instruction, acquiring a currently selected target speaker feature or target accent feature, and correspondingly replacing the reference speaker feature or reference accent feature to obtain a to-be-synthesized voice feature; and inputting the target phoneme sequence and the to-be-synthesized speech features into a decoder for speech reconstruction, and generating target speech corresponding to the target text. Decoupling and arbitrary combination of the speaker features and the accent features are achieved by separating the speaker features and the accent features, and the flexibility of accent control in speech synthesis is improved.

A speech recognition system may determine speech in the presence of a plurality of different forms of corrupted audio. The system may obtain audiovisual data including visual data associated with a person and audio data associated with the person. The system may also determine pronunciation data associated with the speech of the person based on the visual data. The system may also convert the speech to encoded data. The system may also synthesize speech based on the encoded data to obtain synthesized speech.

The invention discloses a voice reconstruction method and device, a storage medium and a program product, and relates to the technical field of audio processing, and the method comprises the steps: obtaining to-be-reconstructed original voice data, and extracting the granularity voice frame features of the original voice data at each frame rate; calculating the voice information density of each granularity voice frame feature in each time window; screening granularity frame feature fragments according with a preset granularity proportion from each granularity voice frame feature according to the voice information density; the granularity proportion predefines the time proportion of the fragment duration of each granularity frame feature fragment in a unit time period; and performing voice reconstruction based on each granularity frame feature fragment to obtain corresponding reconstructed voice data. Therefore, through voice information density analysis, dynamic frame segment distribution of a variable frame rate is realized, high-sound-quality reconstruction is ensured, the coding complexity and delay are remarkably reduced, and the real-time performance and the self-adaptive capability of the whole system are improved.

The present disclosure relates to a speech signal processing method and device, electronic equipment and storage medium, and relates to the technical field of audio. The method comprises: acquiring a multi-channel far-field speech signal collected for a target sound source; inputting the multi-channel far-field speech signal into a preset multi-channel filtering model to perform speech enhancement processing to obtain target speech spectrum information; the preset multi-channel filtering model is a filtering model obtained by adaptively updating an original multi-channel filtering model according to the multi-channel far-field speech signal, or is a filtering model obtained by adaptively constructing according to a prediction result of a first neural network model for the multi-channel far-field speech signal; and inputting the target speech spectrum information into a preset near-field speech generation model to perform speech reconstruction processing to obtain a near-field speech signal corresponding to the multi-channel far-field speech signal. The technical scheme provided by the embodiments of the present disclosure can realize the transformation of speech hearing sensation and improve the quality and intelligibility of speech.

The invention discloses a lightweight semantic preserving coding and decoding and voice signal reconstruction method, which comprises the following steps of: acquiring an analog audio signal from an environment, inputting an original audio signal and an audio signal subjected to high-frequency filtering to two ends of a comparator, outputting an ultralow-bit-rate binary bit stream and transmitting the ultralow-bit-rate binary bit stream to a receiving end, and after the receiving end receives the signal, outputting the ultralow-bit-rate binary bit stream to the voice signal reconstruction end. The method comprises the following steps: inputting a voice signal to an end-to-end U-Net convolutional neural network of a bottleneck layer integrated LSTM module, outputting a reconstructed voice signal, training the network by using voice coding sample reconstruction loss and multi-scale short-time Fourier transform, and performing voice signal reconstruction on an ultra-low bit rate binary bit stream of a to-be-reconstructed audio after training. The technical contradiction that in resource-limited remote audio collection and distributed sensing tasks, node hardware resources are extremely limited, and ultra-low bit rate signal compression distortion causes that node microminiaturization deployment and high voice reconstruction quality cannot be considered at the same time is solved.

The invention discloses a voice reconstruction method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a coding feature of a first voice unit corresponding to a first moment, and obtaining a text prediction result of a second voice unit related to the first voice unit; obtaining an explicit semantic feature of the first voice unit based on the text prediction result of the second voice unit, and obtaining an implicit semantic feature of the first voice unit based on the coding feature of the first voice unit; fusing the explicit semantic feature and the implicit semantic feature to obtain a fused semantic feature of the first voice unit; and based on the fused semantic features of the first voice unit, synthesizing a reconstructed audio corresponding to the first moment. Through the method, the accuracy of audio reconstruction can be improved.

The present invention discloses a microphone array speech enhancement system and method based on a multi-task network. The system comprises a speech preprocessing module, a multi-task network module, a multi-task loss statistics module, a network weight calculation module, and a speech reconstruction module. The speech preprocessing module obtains array speech, reference echo speech, and target speech of each task as input speech and performs preprocessing; the multi-task network module performs dereverberation, echo cancellation, and noise reduction tasks for each channel of the array speech, and outputs multi-channel speech fusion as enhanced speech; the multi-task loss statistics module is used to calculate the loss value of each task in the multi-task network module and calculate the total loss of the network; the network weight calculation module calculates the gradient based on the total loss of the network, back-propagates the gradient, and calculates the weight of the updated network; and the speech reconstruction module completes the mapping from frequency domain features to time domain speech to obtain enhanced clean speech.

The invention relates to the technical field of intelligent plotting, and provides an intelligent plotting method based on voice input. The intelligent plotting method based on voice input comprises the following steps: S1, receiving an original audio signal containing a voice instruction input by a user in real time at a terminal equipment side; according to the intelligent plotting method based on voice input, the problem of communication interruption is solved, the problem of voice recognition under high-strength physical vibration is solved through an innovative voice reconstruction technology, full-domain and full-condition adaptation in the true sense is achieved, vibration distortion reconstruction is preferentially carried out on the voice signals, and therefore the voice recognition efficiency is improved. The method has the advantages that the jittering voice signals are corrected before the jittering voice signals are sent to the recognition engine, so that the voice definition and intelligibility are guaranteed from the information source, the accuracy of all subsequent processing links is greatly improved, and the method is particularly suitable for key instructions which are issued in high-pressure and high-risk environments and have success or failure of events and even life safety. And the reliability guarantee is provided.

This invention relates to the field of speech synthesis technology, and discloses a speech generation method, apparatus, device, and medium based on semantic adjustment. The method includes: encoding text content and style cues to obtain text semantic representation and style semantic representation; performing semantic consistency analysis based on the two to obtain semantic mismatch results; determining dynamic guidance strength based on the semantic mismatch results, and performing autoregressive generation processing in conjunction with alternative style cues to obtain an acoustically labeled sequence; and reconstructing the acoustically labeled sequence to obtain the target speech. This invention can be applied to business scenarios such as fintech and healthcare. By obtaining semantic mismatch results through semantic consistency analysis and determining dynamic guidance strength based on the semantic mismatch results, the speech generation process can be adjusted according to changes in semantic consistency, thereby reducing the deviation caused by inconsistencies between semantic and emotional expression and improving the expressive coordination of the target speech.

The application discloses a cross-modal silent speech reconstruction method and system based on ear canal air pressure micro-motion sensing, and belongs to the technical field of human-computer interaction and wearable computing. The method uses a micro-pressure sensing unit placed in an in-ear earphone to collect a non-acoustic air pressure sequence caused by the movement of a sound-producing organ; through adaptive baseline drift suppression and rhythm perception data enhancement processing, a robust feature space is constructed; further, an end-to-end deep neural network containing domain adversarial adaptation, cross-modal semantic alignment, coarse-grained mel-spectrogram generation and residual detail correction is used to map the TPVS to a high-fidelity acoustic mel spectrum. The application effectively breaks through the technical bottleneck of the lack of high-frequency acoustic features in low-frequency mechanical signals, realizes high-precision silent speech command analysis in a mobile and noisy scene, and introduces a coupled quality evaluation gate and trigger-based start / stop control at the inference end to suppress invalid inference and reduce power consumption when wearing is poor or there is no trigger condition.

An audio processor can include a circuit configured to provide a digital signal representative of a sound, and a digital signal processor configured to process the digital signal. The digital signal processor can be configured to process an enhancement model and including a loss function having a first penalty term selected to control sound reconstruction, and a second penalty term selected to control noise suppression. The first and second penalty terms can be weighted by an adjustable parameter to allow the enhancement model to be tuned for quality of sound or performance of noise suppression.

The voice conversion method, the training method, the device, the equipment and the medium provided by the embodiments of the present application obtain the source linear spectrum of the source speaker according to the source voice data of the source speaker; input the source linear spectrum into a pre-trained voice coding model to output corresponding spectral feature prediction data; input the spectral feature prediction data and target speaker feature data of a target speaker into a voice reconstruction model to output corresponding target voice data; in the above manner, the content information of the source voice data and the speaker feature are decoupled, and in the training stage and the voice conversion stage of the voice coding model, the input and the output of the voice coding model respectively only contain content information, the voice coding model is used to reconstruct voice data together with the content information and the speaker feature through the voice reconstruction model, which is beneficial to improve the training speed and the training effect of the voice coding model, and further improves the voice conversion effect.

This invention relates to the field of artificial intelligence technology and discloses a speech extraction method, apparatus, device, and medium. The method includes: first, acquiring reference speech of the target speaker and mixed speech of all speakers; preprocessing and encoding the reference speech and mixed speech to generate two discrete token sequences; fusing the two discrete token sequences to form a fused discrete token sequence; using a language model to predict the fused discrete token sequence to generate candidate discrete token sequences for the target speaker; calculating the probability distribution of the candidate token sequences using a linear classifier and selecting the sequence with the highest probability as the target discrete token sequence; and then reconstructing the target discrete token sequence into a speech waveform to obtain the speech of the target speaker. This invention transforms the complex audio generation problem into a classification problem, simplifying model training; and utilizes the sequence modeling capability of a language model to capture long-term dependencies between speech tokens, achieving high-quality speech reconstruction.

The invention relates to the technical field of artificial intelligence, and discloses a voice extraction method, device and equipment and a medium, and the method comprises the steps: firstly obtaining the reference voice of a target speaker and the mixed voice of all speakers; the reference voice and the mixed voice are preprocessed and coded, and two discrete token sequences are generated; fusing the two discrete token sequences to form a fused discrete token sequence; predicting the fused discrete token sequence by using a language model, and generating a candidate discrete token sequence of the target speaker; calculating probability distribution of the candidate token sequences through a linear classifier, and selecting a sequence with high probability as a target discrete token sequence; and reconstructing the target discrete token sequence into a voice waveform to obtain the voice of the target speaker. According to the method, a complex audio generation problem is converted into a classification problem, and model training is simplified; and capturing the long-term dependency relationship between the speech tokens by using the sequence modeling capability of the language model to realize high-quality speech reconstruction.

A millimeter-wave radar voice reconstruction and recognition method based on a physical guide network comprises the steps that a millimeter-wave radar is used for transmitting a radio-frequency signal to a to-be-detected target and receiving an echo signal, and meanwhile a reference audio signal is collected; extracting a steady-phase signal Mel spectrum according to the echo signal; generating a simulated radar Mel spectrum by performing an audio signal simulation on the reference audio signal and the common speech data set; synchronizing and standardizing the stable-phase signal Mel spectrum and the simulated radar Mel spectrum, and constructing a voice signal data set; constructing a multi-mode voice reconstruction network model; training a multi-modal voice reconstruction network model according to the voice signal data set; inputting a newly collected real millimeter wave radar signal into the trained multi-mode voice reconstruction network model for voice reconstruction, and outputting a non-contact voice Mel-frequency spectrogram; and inputting the voice Mel spectrogram into the constructed lightweight convolutional neural network classifier, and outputting an identity category label of the speaker.

The present invention relates to the field of financial technology, and discloses a speech synthesis method, apparatus, device and medium for an insurance business end, including obtaining a target splicing feature by splicing and fusing a target text embedding feature with an acoustic feature; inputting the target splicing feature into an encoder of a pre-trained semantic feature processing model for downsampling to obtain a target semantic feature; inputting the target semantic feature into a pre-trained noise-filtered speech reconstruction model to output a first target speech data; inputting the first target speech data, the target semantic feature and the target splicing feature into a decoder of a pre-trained semantic feature processing model to obtain a target speech. This solution solves the problem that the existing text-to-speech conversion technology for the insurance business end is prone to deterioration in sound quality when processing complex phonetics or frequently switched speech, thereby resulting in a poor experience for customers when using intelligent voice customer service for conversational services.

The application belongs to the field of artificial intelligence, and relates to a training method of a speech generation model, comprising the following steps: obtaining reference timbre spectrum, phoneme information and speech spectrum of a target object; training a preset initial speech generation model based on the reference timbre spectrum, the phoneme information and the speech spectrum to obtain model parameters; and adjusting parameters of a multi-timbre feature extraction network, a phoneme feature extraction network, a prosody feature discretization network, a time sequence alignment module, an attention fusion module and a speech reconstruction decoding network of the initial speech generation model based on the model parameters to construct the speech generation model. The application also provides an apparatus, a device and a medium. In addition, the application also relates to blockchain technology, and speech training data and model parameters can be stored in a blockchain. The application can realize decoupling of timbre and prosody information, and flexibly adjust the timbre and prosody information to generate synthesized speech with diversity and flexibility.

The invention provides a speech synthesis method, a speech synthesis device, electronic equipment and a storage medium, relates to the technical field of artificial intelligence, and is suitable for the financial field and the medical field. The method comprises the steps of obtaining a target text, and determining a target language to which the target text belongs; acquiring a sample voice set and a sample text belonging to the target language; performing semantic modeling on the sample text and the target language through an initial semantic modeling device to obtain a first sample semantic unit sequence; performing voice quantization on the sample voice to obtain a second sample semantic unit sequence; performing parameter adjustment on the initial semantic modeler according to the first sample semantic unit sequence and the second sample semantic unit sequence to obtain a target semantic modeler; performing semantic modeling on the target text and the target language through a target semantic modeling device to obtain a text semantic unit sequence; and performing voice reconstruction according to the text semantic unit sequence to obtain a target synthetic voice. According to the invention, the accuracy of cross-language speech synthesis can be improved.

The invention discloses a real-time single-microphone voice noise reduction algorithm based on voice enhancement residual error and continuous spectrum estimation, and relates to a real-time single-microphone voice noise reduction algorithm. The invention aims to solve the problem that conversation voice is buried by noise due to noisy and diverse background noise of an interphone in a special communication scene. Noise power spectrum estimation is optimized by fusing a continuous minimum value tracking algorithm, a speech enhancement residual error is introduced as a real noise approximate value to participate in a recursive average process, and the response speed and accuracy of noise estimation are improved; a gain function is calculated in combination with an optimal correction logarithm MMSE estimator, and a closed expression approximates exponential integration to reduce the calculation complexity. The algorithm specifically comprises the steps of preprocessing and framing, noise power spectrum estimation, gain function calculation, voice reconstruction and the like, the segmentation signal-to-noise ratio is remarkably improved on the premise that low voice distortion is guaranteed, and the real-time processing requirement is met. The invention belongs to the technical field of voice signal processing.

This application relates to the field of speech conversion technology, and provides a speech conversion model training method, speech conversion method, apparatus, and medium. The method includes: extracting speech sample features from preset speech samples using an encoder; then decoupling the speech samples based on a preset masking strategy to obtain sample feature representations; inputting the sample feature representations into a generator to reconstruct the Mel spectrogram of the speech samples based on the sample feature representations to obtain the Mel spectrogram of the target sample; calculating the speech reconstruction loss of the speech conversion model based on the Mel spectrogram of the target sample and the original sample Mel spectrograms corresponding to the preset speech samples; and optimizing the parameters in the speech conversion model based on adversarial loss and speech reconstruction loss to obtain a trained speech conversion model. By decoupling the speech sample features through a preset masking strategy and a preset adversarial network, the robustness of the speech conversion model is improved, thereby increasing training efficiency.

The present application provides a method, device, equipment and medium for reconstructing clean speech, the method comprising: extracting deep feature information from the noisy speech to be enhanced, determining the fundamental frequency information of the speech content in the noisy speech according to the deep feature information, constructing a filter matching the noisy speech based on the deep feature information, and reconstructing the clean speech according to the filter matching the fundamental frequency signal and the noisy speech. The present application takes into account the vocalization principle of the human vocal system, extracts deep feature information that can characterize the speech characteristics of the speaker when emitting the clean speech corresponding to the noisy speech, determines the fundamental frequency information of the speech content in the noisy speech according to the deep feature information, so as to simulate the sound wave signal generated by the vibration of the vocal cords, constructs a filter matching the noisy speech according to the deep feature information, so as to simulate the vocal channel that modulates the sound wave signal, and reconstructs the clean speech that is more in line with the speech characteristics of the speaker when speaking based on the filter matching the fundamental frequency signal and the noisy speech.

The invention relates to the technical field of voiceprint recognition, financial science and technology and smart medical treatment, and discloses a voiceprint model enhancement training method and device, a storage medium and computer equipment. The method comprises the following steps: firstly, acquiring a clean voice sample and actual voiceprint information thereof, adding noise to obtain a noisy sample, then extracting voice features of two types of samples by using a pre-trained voiceprint model, converting the noisy voice features into frame-level features, recognizing the voiceprint information, then inputting the frame-level features into a voice reconstruction model to determine reconstructed voice features, and finally, carrying out voice reconstruction. The reconstruction loss of the clean voice features and the reconstruction voice features and the discrimination loss of the voiceprint recognition information and the actual voiceprint information are calculated, and finally the pre-training voiceprint model is optimized according to the two losses. The voiceprint model can accurately focus clean voice information in a complex environment, interference factors such as environmental noise and channel characteristics are effectively suppressed, the accuracy and robustness of voiceprint recognition are improved, and the performance of a voiceprint recognition system in an actual application scene is enhanced.

The present invention discloses a multi-channel speech separation method and device based on multi-scale feature channel fusion, the method comprising: obtaining a plurality of multi-channel mixed speech signals with different noises, reverberations and speakers to form a training data set; constructing a multi-channel speech separation network based on multi-scale feature channel fusion, specifically comprising a high-dimensional feature extraction module, a spatial feature extraction module, a multi-scale feature extraction module, a convolutional separation network based on feature channel fusion, and a speech reconstruction module; inputting the training data set into the multi-channel speech separation network to perform network training; inputting the mixed multi-channel speech signal containing noise, reverberation and multiple speakers to be separated into the trained multi-channel speech separation network to obtain the single-channel speech signal of each speaker. The present invention has stronger separation ability and generalization ability.

The invention discloses a voice signal generation method and system based on a millimeter wave radar, and belongs to the technical field of Internet of Things sensing and voice signal processing. The method comprises the following steps: designing a millimeter-wave radar fine-grained vibration signal acquisition model; obtaining noisy material vibration information based on the model, and constructing an end-to-end voice reconstruction model by taking a spectrogram of the noisy material as input; performing data enhancement by synthesizing the simulated noisy millimeter wave spectrogram; noisy vibration data captured by a radar is input, and a priori-free and unconstrained recognizable voice is reconstructed by using a deep neural network. The system correspondingly comprises a design module, a construction module, an enhancement module and an output module. The method breaks through the environmental limitation of traditional voice acquisition, improves the model robustness and the voice generation quality, and is suitable for the voice generation demand in a non-contact and complex interference scene.

The embodiment of the invention provides a voice reconstruction method and device based on multi-modal signal fusion and an artificial laryng.The method comprises the steps that when a patient makes a sound, a voice signal, a laryngeal muscle movement signal and a facial muscle movement signal of the patient are collected at the same time; according to the voice signal, the throat muscle movement signal and the face muscle movement signal, generating an acoustic feature vector; and inputting the acoustic feature vector into a preset speech recognition model to obtain speech information or text information. The technical scheme provided by the invention is used for solving the problem that the prior art is not suitable for neuromuscular dysfunction patients.