329 results about "Speech coding" patented technology

The invention discloses a dynamic interaction method based on cooperation of a multi-modal dynamic fusion large model and an intelligent agent. The method comprises the following steps: performing feature extraction on user voice information to obtain a voice coding vector, a text semantic vector and an emotion feature vector; performing dynamic weight feature fusion on the voice coding vector, the text semantic vector and the emotion feature vector through a multi-modal dynamic fusion large model to obtain a fusion feature vector; inputting the fusion feature vector into an intention-scene coupling network, and identifying to obtain a user intention label; and identifying according to the user behavior log to obtain a user portrait tag, inputting the user intention tag and the user portrait tag into an autonomous decision-making agent, generating a target decision-making action through a lightweight policy network, and then interacting with the user according to the target decision-making action. The intelligent interaction efficiency and accuracy of the customer service system are improved, the interaction experience of the user is also improved, and the method can be widely applied to the technical field of artificial intelligence.

The invention relates to the technical field of speech synthesis, can be applied to business scenes such as financial science and technology, medical health and the like, and discloses a speech generation method, device, equipment and medium. And inputting a text speech language model to generate an intermediate code in combination with a speech code extracted based on a codebook generation mode, extracting a speaker feature vector in the prompt speech, decoding the intermediate code and the speaker feature vector by a generative adversarial decoder, and outputting a target speech. According to the method, fine-grained text representation is established by fusing character semantics and pinyin pronunciation information, and voice cloning is completed through unified voice language modeling and an adversarial generation mechanism in combination with codebook-driven acoustic coding and speaker personality characteristics, so that the naturalness, similarity and pronunciation accuracy of generated voices are improved.

The invention relates to the technical field of artificial intelligence and the field of financial science and technology, and discloses a speech synthesis method based on a gated attention mechanism, and the method comprises the steps: obtaining text data and speech data, converting the text data into a text symbol sequence through a text encoder, speech features of the speech data are extracted through a speech encoder trained based on a self-supervised learning model, and the speech features are quantized into a discrete speech symbol sequence; then, respectively organizing the text symbol sequence and the voice symbol sequence into a text sequence and a voice sequence; the corresponding relation between the text symbols and the voice symbols is established through preliminary alignment processing, the matching degree between the text symbols and the voice symbols is dynamically adjusted through a gating attention mechanism, and finally a final voice signal is generated through a decoder. According to the invention, through the voice encoder trained by the self-supervised learning model, the extraction capability of voice features is effectively improved, and effective feature representation can still be learned from unlabeled data especially in a scene lacking a large amount of labeled data.

An apparatus for virtual avatar generation is provided, including a face encoder, a face fusion engine, a voice encoder, and a voice fusion engine. The face encoder extracts first facial features from the first facial image and then encodes they into M first facial feature vectors and extracts second facial features from the second facial image and then encodes they into M second facial feature vectors. The face fusion engine generates a facial fused feature vector from the facial feature vectors, thereby generating a fused image. The voice encoder extracts first voice features from the first audio clip and then encodes they into N first voice feature vectors and extracts second voice features from the second audio clip and then encodes they into N second voice feature vectors. The voice fusion engine synthesizes a voice fused feature vector from the voice feature vectors, thereby generating a fused voice.

The invention discloses a cross-modal attention, global memory and dynamic convolution-based speech translation model training method and device, and a speech translation method and device, and relates to the technical field of speech processing and machine translation. A speech translation model is designed to comprise a speech encoder, a text embedding layer, a cross-modal attention adapter, a large language model decoder, a global memory network, a dynamic convolution decoder and an output layer. The cross-modal attention adapter projects audio features and performs multi-head cross attention fusion with text embedding; the global memory network updates and enhances historical memory on the basis of a gating mechanism and a Transform Encoder; and the dynamic convolution decoder performs multi-scale convolution extraction on the decoded hidden representation and fuses with the memory, so that the translation quality is improved. According to the method, deep fusion of voice and text, continuous memory with contextual coherence and high-quality translation generation can be realized, the end-to-end voice translation performance is remarkably improved, and the actual requirements of real-time and high-quality end-to-end voice translation in a complex scene are met.

The invention relates to the technical field of voice coding, in particular to a variable-bit-rate redundancy-eliminating voice semantic coding method and device, and the method comprises the steps: obtaining the original audio of a target speaker based on a preset deep neural network model; extracting linguistic discrete semantic representation in the original audio, and obtaining voiceprint features and compressed spectrum features of the original audio; and respectively carrying out reconstruction processing on the linguistic discrete semantic representation, the voiceprint feature and the compressed spectrum feature to obtain a target audio acoustic intermediate representation, and obtaining a target audio to be output by utilizing the target audio acoustic intermediate representation so as to carry out redundancy elimination voice semantic coding on the original audio. Therefore, the problems that in the related technology, a voice synthesis model is a pre-training model with fixed parameters, speaker features in voice are difficult to change and recover according to different speakers, part of voice information is distorted, the quality and efficiency of voice recognition are reduced, and the accuracy of voice recovery is reduced are solved.

The embodiment of the invention discloses a discrete voice coding method, electronic equipment and a storage medium, and the method comprises the steps: constructing a voice input and frequency spectrum output VAE, enabling the input voice to have speaker tone disturbance, and enabling the VAE structure to be used for multi-task learning; vector quantization is carried out on the VAE, and a VQ-VAE is initialized, so that a quantization layer is added to continue to reduce an information bottleneck; and additionally training a vocoder, and carrying out waveform reconstruction by starting from the code obtained in the VQ-VAE.

The invention discloses a fine emotion control TTS method and device based on a large model, equipment and a medium, relates to the technical field of artificial intelligence, and can generate a voice service with fine emotion expression and improve the user experience in the high-sensitivity emotion interaction fields of banks, financial customer service, insurance consultation, medical hospital guide and the like. Acquiring an emotional feature vector of the input text based on a preset large language model; encoding the emotion feature vector to obtain a voice encoding vector; determining an emotion curve vector of the input text based on a pre-trained neural network model; and generating target voice corresponding to the input text based on the voice coding vector and the emotion curve vector. According to the method, emotion feature vectors are extracted through a large language model, a dynamic emotion curve is generated in combination with a neural network, speech synthesis is cooperatively driven, emotion fineness and continuity are remarkably improved, and the problems of traditional TTS emotion expression roughening and fragmentation are solved.

The present disclosure provides a pitch-based speech conversion model training method and a speech conversion system, wherein an audio feature code is output by a priori encoder, and a pitch feature is extracted by a pitch extraction module. A linear spectrum corresponding to the reference speech is input into the posteriori encoder to obtain an audio latent variable. In addition, the audio feature code, a speech concatenation feature obtained by concatenation of the audio feature code and the pitch feature, and the audio latent variable are input into a temporal alignment module to obtain a converted speech code, and the converted speech code is decoded by a decoder to obtain a converted speech. The training loss of the converted speech is then calculated to determine the degree of convergence of the speech conversion model.

The invention discloses a layered voice coding and decoding method based on MDCT coefficient residual vector quantization, which comprises the following steps: preprocessing voice data for training, and training to generate a multi-level quantization codebook of MDCT; the method comprises the following steps of: enhancing a high-frequency component of voice to be coded through a pre-emphasis filter, windowing a pre-emphasis voice signal, generating an MDCT coefficient based on an FFT (Fast Fourier Transform) algorithm, determining a bit allocation mode according to a code rate to generate a coding control bit, compressing a codebook index and the coding control bit through Huffman coding, and transmitting a code stream to a channel; the method comprises the steps of reading a code stream in a channel, obtaining a coding control bit and an MDCT codebook index through Huffman decoding, determining a voice sampling rate and a bit distribution mode through the coding control bit, obtaining an MDCT vector based on MDCT codebook index retrieval, and restoring an original voice signal through de-emphasis filtering. Through the design of MDCT layered coding, the coder can dynamically adjust coding bit distribution according to the code rate condition, so that the method adapts to the actual communication environment and deals with the sudden network condition.

The embodiment of the invention provides a voice coding method and decoding method based on time sequence modeling and a related device. The voice coding method comprises the following steps: acquiring target voice; extracting voice features corresponding to the voice waveform of the target voice through an input cavity convolution extraction module; performing down-sampling processing on the voice features through a down-sampling module to obtain a down-sampling sequence; performing long time sequence modeling on the down-sampling sequence through a long and short time memory neural network added with exponential activation and matrix operation to obtain a coding sequence; and converting the feature dimension of the coding sequence into 128 dimensions through the first convolutional layer to obtain a target code stream. According to the embodiment of the invention, the feature extraction module, the down-sampling module, the long and short term memory neural network and the output cavity convolution layer code the target voice, so that the compression of the voice with the extremely low bit rate can be realized, and the reconstruction of the voice with the extremely low bit rate and high quality under 150bps is realized.

The invention relates to a speech recognition method, device and equipment based on an end-to-end cross-language large model. The method comprises the following steps: constructing a cross-language large model; obtaining a training data set formed by multiple groups of voice text label pairs, and inputting the training data set into the cross-language large model for end-to-end training until a trained cross-language large model is obtained; a to-be-recognized voice signal is input into the trained cross-language model, and filter feature and wav2vec2 feature extraction and splicing, text embedded feature representation, dimension alignment, splicing and decoding translation are carried out through a voice coding module, a text coding module, a feature alignment module and a decoding module in the model in sequence. And outputting and obtaining the speech recognition text of the target language. By adopting the method, the accuracy and robustness of speech recognition can be improved, and cross-language speech recognition is realized.

Techniques and solutions are described for encoding and decoding signals, such as audio data. Disclosed innovations can find particular use in speech coding applications, such as for real time communications. Using a neural network, contextual coding can be used to encode latent features for a current frame using a prediction from reconstructed latent features of past frames as a context. An extractor learns a residual-like feature based on such prediction and latent features of the current frame obtained using an encoder. The residual-like feature is then quantized. At a decoder portion of a coding framework, the quantized feature is dequantized and then combined with a prediction from prior reconstructed latent features to provide reconstructed features of a current frame, which can then be processed by a decoder to provide a reconstructed signal.

The invention provides a voice processing method and device, equipment, a medium and a product. The method comprises the following steps: acquiring a target voice and a translation mode of the target voice; based on a voice encoder module and a translation mode in the voice processing model, encoding the target voice to obtain voice content features and acoustic features of the target voice; translating the target voice based on a large language model, a translation mode and voice content features in the voice processing model to obtain a target translation text of the target voice; and based on a voice decoder module, the translation mode and the acoustic features in the voice processing model, performing voice synthesis on the target translation text to obtain a target translation voice of the target voice. According to the invention, adaptive processing is carried out on voice coding, translation and voice decoding through a voice processing model of an integrated framework integrating voice translation and voice simultaneous transmission in combination with a translation mode, so that the deployment cost is better reduced, and the real-time performance and quality of voice processing are optimized.

A method includes receiving training data that includes a set of transcribed speech utterances where each respective transcribed speech utterance is paired with a corresponding transcription. For each respective transcribed speech utterance, the method includes generating an encoded audio representation and an encoded textual representation, generating a higher order audio feature representation for a corresponding encoded audio representation, generating a higher order textual feature representation for a corresponding encoded textual representation, and determining a loss for the respective transcribed speech utterance based on the higher order audio feature representation and the higher order textual feature representation. The method also includes training a speech encoder and a text encoder of a correction model based on the loss determined for each transcribed speech utterance of the set of transcribed speech utterances.

The invention discloses a training method and device of a dialect recognition model and a dialect recognition method. The model training method comprises the following steps: training a first initial model comprising a feature extraction module, a voice encoder and a natural language large model by using a first dialect sample set with a text label to obtain a first dialect recognition model; training a second initial model containing a feature extraction module, a voice encoder and a voice decoder in the first dialect recognition model by using a text result of a first dialect sample set predicted by the first dialect recognition model to obtain a second dialect recognition model; and finally, training a second dialect recognition model by using the second dialect sample set without text annotation and a text result of the second dialect sample set predicted by the first dialect recognition model to obtain a target dialect recognition model. The technical problems that a large number of samples are needed in traditional dialect recognition model training, the data quality is difficult to guarantee, and the labeling cost is high are solved.

Disclosed are a speech encoder training method performed by a computer device. The method includes: masking a first sub-feature representation at a first feature position in a first text feature representation to obtain a first masked feature representation; performing feature prediction on a masked first feature position in the first masked feature representation based on a first speech feature representation to obtain a first predicted feature representation; and training a first speech encoder based on a difference between the first predicted feature representation and the first sub-feature representation to obtain a second speech encoder. The first speech encoder is trained by combining data in a speech modality with data in a text modality, and information included in the data in the text modality is adopted so that the first speech encoder can learn relatively high-level semantic representations of speech, thereby improving the prediction accuracy of representations.

A method includes receiving training data that includes a plurality of sets of training utterances each associated with a respective language. Each training utterance includes a corresponding reference speech representation paired with a corresponding input text sequence. For each training utterance, the method includes generating a corresponding encoded textual representation for the corresponding input text sequence, generating a corresponding speech encoding for the corresponding reference speech representation, generating a shared encoder output, and determining a text-to-speech (TTS) loss based on the corresponding encoded textual representation, the corresponding speech encoding, and the shared encoder output. The method also includes training a TTS model based on the TTS losses determined for the training utterances in each set of the training utterances to teach the TTS model to learn how to synthesize speech in each of the respective languages.

The invention discloses a call establishment method and device and network side equipment, and belongs to the technical field of communication, and the call establishment method comprises the steps that a first core network element receives a first call establishment request message sent by a first terminal; the first core network element sends a second call establishment request message to the second terminal; the second call establishment request message and the first call establishment request message meet at least one of the following conditions: a voice coding mode carried by the second call establishment request message is different from a voice coding mode carried by the first call establishment request message; the number of parameters included in the first message header carried by the second call establishment request message is different from the number of parameters included in the first message header carried by the first call establishment request message; under the condition that the first call establishment request message carries the second message header, the second call establishment request message does not carry the second message header.

The invention relates to the field of artificial intelligence, and can improve the fluency and naturalness of multilingual speech synthesis in the aspect of speech code conversion. Relates to a speech synthesis method and device, equipment and a medium. The method comprises the following steps: acquiring initial feature vectors respectively corresponding to at least two types of initial speech data; calling the trained speech synthesis model; determining a target language expert unit corresponding to each type of initial voice data, and inputting the initial feature vector corresponding to each type of initial voice data into the corresponding target language expert unit for feature extraction to obtain first feature information corresponding to each type of initial voice data; inputting the initial feature vectors corresponding to all the initial voice data into a language-independent expert unit for feature extraction to obtain second feature information; determining target feature information according to the second feature information and the first feature information corresponding to each type of initial voice data; and performing voice synthesis on the target feature information to obtain target voice data.

The invention discloses a barrier speech recognition and reconstruction method based on a speech data retrieval enhancement technology. The method comprises the following steps: segmenting a to-be-recognized obstacle speech into a plurality of segments; for each segment, obtaining a plurality of similar voice segments and corresponding text contents by retrieving a barrier voice database; respectively coding the similar voice segments and the to-be-recognized obstacle voice to obtain similar voice codes and to-be-recognized obstacle voice codes; respectively converting the similar speech codes and the to-be-recognized obstacle speech codes into first codes and second codes understood by a large language model; the first code and the second code are input into a large language model, and text information corresponding to the to-be-recognized obstacle voice is obtained by using set cue words, and the cue words comprise historical dialogue content of the to-be-recognized obstacle voice and text content corresponding to similar voice segments; and performing voice reconstruction on the recognized text information. According to the invention, the accuracy and generalization ability of obstacle speech recognition can be improved.

The invention provides a large-model real-time voice interaction method and device based on autoregressive voice synthesis, and the method comprises the steps: obtaining a voice instruction marked with a target text response and a target voice response, enabling a voice encoder to code the voice instruction into voice representation, and enabling a voice adapter to carry out the dimension reduction and feature conversion of the original voice representation; the large language model generates a hidden state according to the converted voice representation and samples the hidden state to obtain a text sequence; and processing the text sequence by adopting a text-voice language model based on an autoregression Transform structure, generating a voice marking sequence in a streaming manner, and converting the voice marking sequence into a voice signal through a vocoder. According to the method provided by the invention, the naturalness and fluency of speech synthesis are greatly improved while high real-time performance is ensured. The optimized voice decoding architecture effectively reduces the voice generation delay and improves the response speed of the voice interaction system.

The invention relates to the technical field of speech synthesis, and particularly discloses a speech synthesis method and system for controllable latent variable modeling based on semantic distillation, and the method comprises the steps: converting a Mel spectrum into continuous latent variable distribution through a speech coding module, generating continuous latent variables through re-parameterization sampling, introducing a self-supervised model for semantic distillation, and carrying out the semantic distillation. According to the method, alignment of latent variables and semantic features is constrained through marginal cosine similarity and distance matrix structure loss, a text encoder maps a phoneme sequence into latent variable distribution, time sequence alignment of a text and the latent variables is achieved in combination with monotonic alignment search, and a decoder reconstructs the latent variables into a Mel spectrum. According to the method, waveform synthesis through a vocoder and total loss function joint optimization reconstruction, KL divergence, distillation, text alignment and confrontation loss are carried out, discrete information loss is avoided through continuous latent variable modeling, semantic consistency and text alignment efficiency are enhanced, the naturalness, coherence and real-time performance of synthesized voice are improved, and the method is suitable for scenes such as voice assistants and virtual anchors.

The embodiment of the invention relates to a method and device for training a voice coding model, equipment and a readable medium. The method comprises: processing a speech feature representation of a speech sample using a first speech coding model to generate a set of discrete features; generating label information based on the set of discrete features, the label information comprising a set of labels, the set of labels indicating clustering centers corresponding to the corresponding discrete features; processing an intermediate feature representation generated based on the voice feature representation by using a second voice coding model to generate probability information corresponding to the tag information; the training loss is determined based on the label information, the probability information and the weight information, and the weight information is determined based on the distance from each discrete feature to the corresponding clustering center; and adjusting parameters of the second speech coding model based on the training loss. Therefore, the data volume required by training can be reduced, the stability of the unsupervised model training process can be improved, the training effect of the model is improved, and the model capability is further improved.

The invention discloses an efficient simultaneous interpretation method based on an expert routing threshold, and relates to the field of voice processing, the method is based on a classical Transform architecture model, an expert routing strategy model based on the expert routing threshold is constructed, and multi-language streaming translation is realized, the model comprises a streaming voice encoder, and the streaming voice encoder adopts a hybrid design and is connected with the expert routing threshold. The block-by-block autoregression block is composed of an autoregression block and a non-autoregression block; the text decoder is used for simultaneously processing the complete offline voice and the randomly truncated voice prefix to generate a hidden state; the routing threshold module is realized by a feedforward network and projects the final hidden state into a scalar value to determine an expert weight; and the hybrid expert post-processing module shares a language model head with the text decoder, and predicts a target translation sequence in combination with prefix information and global information. According to the method, a mixed expert threshold scheme is adopted to learn the strategy, the self-learning ability of the neural network is fully played, good effects are achieved in streaming translation and streaming TTS, and the method can be used for generating more streaming sequences.

The invention relates to a voice recognition method and device, electronic equipment, a storage medium and a program product. The method may comprise the following steps: acquiring a to-be-recognized voice, the to-be-recognized voice comprising a wake-up voice and a target voice; the to-be-recognized voice is input into a pre-trained voice recognition model to obtain a prediction text, the prediction text is a text used for representing voice content of a target user in the target voice, and the target user is a speaker waking up the voice; wherein the speech recognition model comprises a speech encoder, and the speech encoder is used for obtaining a first feature vector comprising semantic features and speaker features according to the speech to be recognized so as to determine a prediction text. Therefore, in a noisy scene such as multiple speakers, the voice content of the target user in the target voice can be accurately recognized. In addition, the voice encoder is subjected to self-supervised training by using single-person voice, so that the dependence on annotated data can be effectively reduced, and the generalization ability and adaptability of the model are enhanced.

The invention relates to the technical field of artificial intelligence, and discloses a speech synthesis method and device based on artificial intelligence, electronic equipment and a storage medium. The method comprises the steps of obtaining a to-be-synthesized text and voice prompt data; inputting the to-be-synthesized text to a preset phoneme encoder, and analyzing the to-be-synthesized text through the phoneme encoder to obtain first phoneme data; inputting the voice prompt data to a preset rhythm extractor, and analyzing the voice prompt data through the rhythm extractor to obtain first rhythm data; inputting the voice prompt data to a preset first voice encoder, and analyzing the voice prompt data through the first voice encoder to obtain first voice feature prompt data; embedding the first phoneme data into the first rhythm data to obtain second rhythm data; and processing the second rhythm data and the first voice feature prompt data through a preset second voice decoder to obtain a target synthetic voice. According to the method, the stability of speech synthesis based on the large language model is improved.

The invention provides a decoding server suitable for multi-coding voice data, and the decoding server is provided with a plurality of decoders with different voice coding formats, and each decoder is optimized through the following method: a buffer which comprises a plurality of buffer queues and is used for receiving a voice decoding request, distributing the received voice decoding request to a cache queue for caching according to a dynamic distribution function; the voice decoding request comprises a coding format, a data frame length and a coding data stream; and the parallel processor comprises a plurality of decoding channels, each decoding channel corresponds to one buffer queue, and the parallel processor is used for reading the voice decoding request from the buffer, calling a corresponding decoder for decoding according to the encoding format in the voice decoding request, obtaining voice decoding data and transmitting the voice decoding data.

The invention provides an ultra-low bit rate voice coding and decoding system based on text semantic information fidelity, and relates to the technical field of voice coding and decoding. The system comprises the following steps: performing voice feature extraction and text feature extraction on original voice through a multi-modal text-voice combined encoder to obtain voice features and text features, embedding the text features into the voice features to obtain text-voice features of the original voice, and performing voice feature extraction and text feature extraction on the original voice through a multi-modal text-voice combined encoder; sending the text-speech features of the original speech to a receiving end in a semantic communication system; and the receiving end inputs the received text-speech features into a text semantic fidelity decoder based on an attention mechanism to obtain reconstructed speech features and reconstructed text features, decodes the reconstructed speech features layer by layer, and performs attention calculation on the reconstructed speech features and the reconstructed text features to obtain decoded speech. By means of the voice coding and decoding technology, good voice perception quality can still be kept under the ultra-low code rate, and voice intelligibility is guaranteed.

The present disclosure describes techniques for improving speech recognition using a machine learning model. The machine learning model comprises a speech encoder configured to generate acoustic representations based on input speech, an adapter configured to generate adapted representations based on the acoustic representations, and a decoder configured to generate text corresponding to the input speech. A matching loss is applied during training the machine learning model. The matching loss is configured to explicitly force acoustic representations generated by the adapter to align with text embeddings. The machine learning model is fine-tuned by employing parameter-efficient low-rank adaptation. The machine learning model is trained to perform automatic speech recognition with performance improvement and parameter efficiency.