82 results about "Phonetic representation" patented technology

The invention discloses a feature attention and bilinear gating fused speech emotion recognition method and device, and the method comprises the following steps: 1, collecting an audio file, obtaining corresponding label information, generating audio waveform and time frequency representation data through preprocessing, and constructing an audio waveform mask and a time frequency mask to mark an effective information region; 2, constructing a dual-path feature extraction module which comprises a time-frequency feature extraction module and a pre-training acoustic feature coding module; wherein the time-frequency feature extraction module models emotion correlation through local convolution and a multi-dimensional attention mechanism, and performs global time sequence modeling based on a bidirectional gated loop network; the pre-training acoustic feature coding module extracts high-level speech representation with high expression ability for emotion distinguishing by using a pre-training model; and step 3, constructing a feature fusion module and an emotion classification module, and combining with a dual-path feature extraction module to form a speech emotion recognition model.

The invention discloses an audio and video depth forgery detection method based on quality perception and multi-scale alignment, and the method comprises the following steps: coding a synchronous audio and video sequence, and obtaining a frame-level visual feature, a facial action unit and a phoneme-level voice representation; a visual quality evaluation module is introduced to generate a spatial reliability mask, and quality weighting is carried out on the visual features; designing a global-local multi-scale cross-modal alignment mechanism, performing bidirectional cross-attention modeling on voice and face dynamic synchronization globally, and performing physiological coupling alignment on phonemes and face action units locally; and an uncertainty perception reasoning and calibration scheme is provided, adaptive temperature scaling is carried out according to quality and consistency, and uncertainty calibration is carried out by self-supervision loss. According to the method, the problems of insufficient robustness and excessive self-confidence misjudgment of an existing method in a low-quality video and high-synchronization counterfeit scene are solved, and the cross-dataset generalization capability and the actual deployment reliability are remarkably improved.

The invention relates to the technical field of speech recognition and natural language processing, and particularly discloses a law enforcement recorder audio semantic retrieval method and system based on retrieval enhancement, and the system comprises a data acquisition module which is used for acquiring audio data and text query; the voice adapter module is used for projecting the audio data to a text embedding space to obtain voice representation; the cross-modal retriever is used for carrying out cross-modal retrieval on the voice representation and the text query to obtain voice Tokens; the voice language model is used for obtaining character hypotheses; the query generation module is used for extracting query fragments possibly containing entity names; the entity retrieval module is used for retrieving an entity database according to the query fragment to obtain a related entity name; the context construction module is used for constructing context information; and the large language model is used for obtaining a semantic retrieval result. According to the method, the identification and retrieval accuracy of the entity name and key information in the law enforcement recorder audio is improved.

A method (500) includes obtaining an untagged training data corpus (358) including a plurality of spoken utterances (360), each corresponding spoken utterance of the plurality of spoken utterances including audio data (362) characterizing the corresponding spoken utterance. The method also includes receiving a target domain (324). The method further includes using a contrast data selection model (310) to select a subset of utterances corresponding to the target domain from the untagged corpus of training data. The method includes training an automatic speech recognition (ASR) model on a subset of utterances (200).

A method includes receiving training data that includes a plurality of sets of training utterances each associated with a respective language. Each training utterance includes a corresponding reference speech representation paired with a corresponding input text sequence. For each training utterance, the method includes generating a corresponding encoded textual representation for the corresponding input text sequence, generating a corresponding speech encoding for the corresponding reference speech representation, generating a shared encoder output, and determining a text-to-speech (TTS) loss based on the corresponding encoded textual representation, the corresponding speech encoding, and the shared encoder output. The method also includes training a TTS model based on the TTS losses determined for the training utterances in each set of the training utterances to teach the TTS model to learn how to synthesize speech in each of the respective languages.

A method includes receiving audio features and generating a latent speech representation based on the audio features. The method also includes generating a target quantized vector token and a target token index for a corresponding latent speech representation. The method also includes generating a contrastive context vector for a corresponding unmasked or masked latent speech representation and deriving a contrastive self-supervised loss based on the corresponding contrastive context vector and the corresponding target quantized vector token. The method also include generating a high-level context vector based on the contrastive context vector and, for each high-level context vector, learning to predict the target token index at the corresponding time step using a cross-entropy loss based on the target token index. The method also includes predicting speech recognition hypotheses for the utterance and training a multilingual automatic speech recognition (ASR) model using an unsupervised loss and a supervised loss.

The invention provides a large-model real-time voice interaction method and device based on autoregressive voice synthesis, and the method comprises the steps: obtaining a voice instruction marked with a target text response and a target voice response, enabling a voice encoder to code the voice instruction into voice representation, and enabling a voice adapter to carry out the dimension reduction and feature conversion of the original voice representation; the large language model generates a hidden state according to the converted voice representation and samples the hidden state to obtain a text sequence; and processing the text sequence by adopting a text-voice language model based on an autoregression Transform structure, generating a voice marking sequence in a streaming manner, and converting the voice marking sequence into a voice signal through a vocoder. According to the method provided by the invention, the naturalness and fluency of speech synthesis are greatly improved while high real-time performance is ensured. The optimized voice decoding architecture effectively reduces the voice generation delay and improves the response speed of the voice interaction system.

Disclosed herein are method and system aspects for translating between a sign language and a target language and presenting such translations. For example, a method receives input language data and translates the input language data into sign language grammar. The method retrieves phonetic representations that correspond to the sign language grammar from a sign language database and generates coordinates from the phonetic representations using a generative network. The phonetic representations are digital representations of individual signs created through manual input of body configuration information corresponding to the individual signs. Further, the method renders an avatar that moves between the coordinates. In another example, a bidirectional communication system allows for realtime communication between a signing entity and a non-signing entity.

The invention discloses a children voice expression error recognition and correction method based on comparative learning, and the method comprises the steps: carrying out the preprocessing of an inputted children voice signal, obtaining a logarithmic Mel spectrum feature sequence, and converting the logarithmic Mel spectrum feature sequence into voice semantic coding features through an improved Transform encoder; on the basis of an age-adaptive positive and negative sample selection mechanism, voice features are optimized by using a comparative learning method, and an enhanced voice representation vector is obtained; constructing a multi-modal fusion network, combining an enhanced voice vector and BERT language model features, realizing adaptive fusion through multi-head cross attention and a gating mechanism, adopting a bidirectional LSTM to design an error positioning module to recognize the position and the type of a pronunciation error, and using a joint loss function to execute end-to-end training; standard correction audio is generated according to the error type, and correction guidance is provided for children. According to the invention, high-precision recognition, positioning and correction of children's speech expression errors are realized, and technical support is provided for children's language development.

The invention discloses a voice clone model generation method and device, a storage medium and computer equipment, relates to the technical field of voice synthesis and medical treatment, and mainly aims at solving the problems that a voice clone model trained by an existing method is relatively weak in noisy voice recognition capability and insufficient in robustness. The method mainly comprises the steps of obtaining a voice training sample set; a training model of the voice cloning model is constructed, and the training model of the voice cloning model comprises an initial speaker encoder, a self-supervised voice representation learning model and an initial voice synthesizer; training an initial speaker encoder and an initial voice synthesizer by using a natural voice training sample and a recognition result of a self-supervised voice representation learning model on the natural voice training sample; and extracting a speaker encoder and a voice synthesizer in the training model of the voice clone model, and generating the voice clone model according to the speaker encoder, the voice synthesizer and the text encoder. The method is mainly used for training a speech clone model.

A method for synthesizing speech based on a textual input includes receiving a textual input including native words of a native language and foreign words of a foreign language and processing the textual input to determine a phonetic representation of the textual input. The processing includes determining a native phonetic symbol representation of the native word and determining a localized phonetic symbol representation of the foreign word. Determining the localized phonetic representation includes forming a foreign phonetic representation of the foreign word using the set of foreign phonemes, and mapping the foreign phonetic representation to the localized phonetic representation according to a model of pronunciation of the foreign word by the local speaker.

A method includes receiving a transcription of an utterance, processing, using a first model, the transcription to generate a first text segment that represents an initial portion of a response to the utterance, processing, using a TTS system, the first text segment to generate a first synthesized speech representation, and providing, for audible output, the first synthesized speech representation. The method also includes providing, to a second model different from the first model, the transcription and the first text segment, the second model comprising an LLM configured to process the transcription and the first text segment to generate a second text segment that represents a remaining portion of the response to the utterance. The method further includes obtaining a second synthesized speech representation generated from the second text segment, and providing, for audible output by the user device, the second synthesized speech representation.

According to one embodiment, a control device (3; 3-2) recognizes an object area from image information that corresponds to an object as a target in automated handling operations. The control device generates a handling strategy for the object and outputs a speech representation of the handling strategy. The control device constructs the object area based on a speech representation of an object shape encompassed in the handling strategy. The control device evaluates the handleability of the object based on geometric information about the object area. The control device evaluates the adaptability of a handling tool (14) to the object based on the speech representation of the object shape.Based on the manageability and adaptability of the handling tool (14), the control device determines whether handling of the object is possible and / or generates a handling position and a posture of the handling tool (14).

A direct speech-to-speech translation (S2ST) model (200 includes an encoder (210) configured to receive an input speech representation (102) that to an utterance (108) spoken by a source speaker (104) in a first language and encode the input speech representation into a hidden feature representation (215). The S2ST model also includes an attention module (220) configured to generate a context vector (225) that attends to the hidden representation encoded. The S2ST model also includes a decoder (230) configured to receive the context vector generated by the attention module and predict a phoneme representation (235) that corresponds to a translation of the utterance in a second different language. The S2ST model also includes a synthesizer (300) configured to receive the context vector and the phoneme representation and generate a translated synthesized speech representation (355) that corresponds to a translation of the utterance spoken in the different second language.

A method for disentangled variational speech representation learning for voice conversion, performed by at least one processor, is provided. The method includes receiving input speech segments, encoding the input speech segments via a shared encoder to generate a speaker embedding and a content embedding, encoding the posterior distributions of the speaker embedding via a speaker encoder and encoding the posterior distributions of the content embedding via a content encoder to obtain encoded results, and decoding the encoded results by concatenating the encoded results to obtain a reconstructed speech output.

The invention relates to the technical field of voice semantics, can be applied to business system platforms of financial science and technology, medical treatment and health and the like, and discloses a controllable zero sample voice conversion method, device, equipment and medium, the method comprises the following steps: carrying out self-supervised voice learning on unlabeled voice data to obtain self-supervised voice representation; the method comprises the following steps of: extracting a content feature vector and a rhythm style vector represented by self-supervised speech, converting the content feature vector and the rhythm style vector into a discrete content token and a discrete rhythm token, performing mask generation on the discrete rhythm token to obtain a target rhythm token, obtaining reference speech of a target user, extracting user style embedding in the reference speech, and obtaining a target user. And performing stream matching on the discrete content token, the target rhythm token and the user style embedding to generate a target Mel spectrogram, and performing voice waveform reconstruction and optimization on the target Mel spectrogram to obtain a zero sample voice conversion result. According to the invention, under the condition of no annotated voice data, personalized, high-fidelity and style-consistent zero-sample voice conversion is realized.

The embodiment of the present application provides a training method and system of a self-supervised training predictor for speech separation. The method comprises: extracting self-supervised training features of each single person voice source speech by using a pre-training model; extracting shallow features for speech representation and deep features for context information in the self-supervised training features, and determining the shallow features and the deep features of each single person voice source speech as training labels of the self-supervised training predictor; inputting training mixed speech generated by each single person voice source speech into the self-supervised training predictor to obtain estimated features of each single person voice source speech; and training the self-supervised training predictor based on a loss function determined based on the estimated features and the training labels corresponding to each single person voice source speech. The self-supervised training predictor is trained and applied in a speech separation model in the embodiment of the present application, so that the accuracy of the self-supervised training features is improved, the performance of the speech separation system is improved, and the model parameters and the calculation complexity are reduced.

A method of providing emotive text-to-speech includes obtaining input text characterizing a natural language response generated by an assistant LLM to a query input by a user during a conversation between the user and the assistant LLM, and processing, using the assistant LLM, the input text conditioned on an emotion detection task prompt to predict, as output from the assistant LLM, an emotional state of the natural language response. The method also includes determining, based on the emotional state of the natural language response predicted as output from the assistant LLM, an emotional embedding for the input text and instructing a TTS model to process the input text and the emotional embedding to generate a synthesized speech representation of the natural language response conveying the emotional state of the natural language response as specified by the emotional embedding.

The present application discloses an audio optimization method and related devices, electronic devices, and storage media. Among them, the audio optimization method includes: extracting a first audio representation of the collected audio and extracting a second audio representation of the reference audio; respectively extracting a first echo representation, a first speech representation, and a first noise representation based on the first audio representation and the second audio representation; performing interactive processing on the first speech representation with the first echo representation and the first noise representation respectively to obtain a second speech representation, a second echo representation, and a second noise representation; where the interactive processing includes: echo suppression, noise suppression, and speech enhancement; obtaining an optimized target audio based on at least one of the second speech representation, the second echo representation, and the second noise representation. The above solution can improve the audio optimization effect.

According to embodiments of this disclosure, a method, apparatus, device, and storage medium for text-to-speech conversion are provided. The method includes generating a predicted speech representation of the target text read by a first speaker, based on a target text to be converted and a first timbre of a first speaker. The predicted speech representation indicates speech features that vary over time. The method further includes generating a predicted time-frequency representation of the target text read by a second speaker, based on the predicted speech representation and a second timbre of a second speaker. The predicted time-frequency representation indicates the speech signal strength that varies over time at different frequencies. The method further includes converting the predicted time-frequency representation into audio of the target text read by the second speaker. This reduces the difficulty of prediction and improves the sound quality of the generated audio.

In various examples, first textual data may be applied to a first MLM to generate an intermediate speech representation (e.g., a frequency-domain representation), the intermediate audio representation and a second MLM may be used to generate output data indicating second textual data, and parameters of the second MLM may be updated using the output data and ground truth data associated with the first textual data. The first MLM may include a trained Text-To-Speech (TTS) model and the second MLM may include an Automatic Speech Recognition (ASR) model. A generator from a generative adversarial networks may be used to enhance an initial intermediate audio representation generated using the first MLM and the enhanced intermediate audio representation may be provided to the second MLM. The generator may include generator blocks that receive the initial intermediate audio representation to sequentially generate the enhanced intermediate audio representation.

The invention relates to the technical field of artificial intelligence, finance and medical health, and discloses a voice authentication method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring to-be-identified voice information; inputting the voice information to be subjected to authentic identification into the self-supervised large model for feature extraction to obtain an extraction result; inputting the potential speech representation features into a classification model for deep coding to obtain a coding result, converting the coding result into low-dimensional representation, and calculating the probability distribution of speech authenticity to obtain a classification result; and outputting a classification result. By implementing the method provided by the invention, the capability of identifying the forged voice in high-imitation audio and noise environments can be remarkably improved, and reliable guarantee is provided for voice safety in key fields such as finance, medical health and the like.

The present invention relates to the technical fields of speech recognition and natural language processing, and specifically discloses a method and system for semantic retrieval of law enforcement recorder audio based on retrieval enhancement. The system comprises: a data acquisition module for acquiring audio data and text queries; a speech adapter module for projecting the audio data into a text embedding space to obtain a speech representation; a cross-modal retriever for performing cross-modal retrieval of the speech representation and text query to obtain speech tokens; a speech language model for obtaining text hypotheses; a query generation module for extracting query fragments that may contain entity names; an entity retrieval module for searching an entity database based on the query fragments to obtain relevant entity names; a context construction module for constructing contextual information; and a large language model for obtaining semantic retrieval results. The present invention improves the recognition and retrieval accuracy of entity names and key information in law enforcement recorder audio.

To provide a system that trains a front-end feature extractor using a deep neural network (DNN) with a triplet network architecture.SOLUTION: A DNN includes three feedforward neural networks trained according to a batch processing that utilizes a cohort set of negative training samples. After each batch of training samples is processed, the DNN is trained according to a loss function that utilizes, for example, a cosine measure of similarity between respective samples, along with positive and negative margins, to provide a robust representation of voiceprints.SELECTED DRAWING: Figure 2A

A mechanism for filtering or attenuating a representation of voice in an audio signal of a personal care device. The audio signal is split or divided into a plurality of audio signal portions. Each audio signal portion is processed to predict whether or not the audio signal portion contains a representation of voice. One or more properties of the audio signal portions identified as containing a representation of voice are used to attenuate a representation of voice in the audio signal.

The present disclosure provides a training method for a speech translation model, a speech translation method, a device and a computer-readable medium. The training method includes: obtaining a source speech representation sequence corresponding to source speech data and a source text representation sequence corresponding to the source speech data; obtaining a mixed sequence according to the source speech representation sequence and the source text representation sequence; processing the source speech representation sequence using the speech translation model to output a first probability distribution of a first target text, and processing the mixed sequence using the speech translation model to output a second probability distribution of a second target text; calculating a total loss function according to the first probability distribution and the second probability distribution; and training the speech translation model according to the total loss function.

A method (600) includes receiving training data (301) including a plurality of sets of text-to-speech (TTS) spoken utterances (510), each set of TTS spoken utterances being associated with a respective language and including TTS utterances of synthetic speech including a corresponding reference speech representation (504) paired with a corresponding input text sequence (502). For each TTS utterance, the method includes generating a corresponding TTS encoded text representation for the corresponding input text sequence (512); generating a corresponding speech code for the TTS utterance of the corresponding synthetic speech (514); generating a shared encoder output (532, 534); generating a predicted speech representation for the TTS utterance of the corresponding synthetic speech (522); and determining a reconstruction loss (545). The method further includes training a TTS model based on the reconstruction loss determined for the TTS utterance in each set of TTS spoken training utterances (501).

A method, computer program product, and computing system for processing a speech signal. A sensitive portion of the speech signal is identified. A pseudo-speech representation of the sensitive portion is generated using a voice converter system. Speech processing is performed on the speech signal and the pseudo-speech representation of the sensitive portion using a speech processing system.

A method for speech conversion includes obtaining a speech conversion model configured to convert input utterances of human speech directly into corresponding output utterances of synthesized speech. The method further includes receiving a speech conversion request including input audio data corresponding to an utterance spoken by a target speaker associated with atypical speech and a speaker identifier uniquely identifying the target speaker. The method includes activating, using the speaker identifier, a particular sub-model for biasing the speech conversion model to recognize a type of the atypical speech associated with the target speaker identified by the speaker identifier. The method includes converting, using the speech conversion model biased by the activated particular sub-model, the input audio data corresponding to the utterance spoken by the target speaker associated with atypical speech into output audio data corresponding to a synthesized canonical fluent speech representation of the utterance spoken by the target speaker.

The application relates to the fields of artificial intelligence and medical health technology, and discloses a mood regulation instruction recommendation method, device, equipment and medium, wherein the method comprises the following steps: adopting a pre-trained dialogue semantic representation model to generate a dialogue semantic representation vector of psychological counseling data; adopting a pre-trained facial expression representation model to generate a micro-expression representation vector of the psychological counseling data; adopting a pre-trained voice representation model to generate a voice representation vector of the psychological counseling data; adopting a pre-trained abnormal mood comprehensive recognition model to generate a comprehensive representation vector of the dialogue semantic representation vector, the micro-expression representation vector and the voice representation vector; and adopting a pre-trained mood regulation instruction recommendation model to recommend a mood regulation instruction for the comprehensive representation vector and a personal information feature vector corresponding to a counseling object, so as to obtain a recommendation result. Therefore, the accuracy of the recommendation result is improved.