234 results about "Prosody" patented technology

A neural end-to-end system is provided for the face and voice preserving translation of videos. The system is a pipeline of multiple models that produces a video of the original speaker speaking in the target language with modified lip movement to match the target speech, while preserving emphases and prosody of the original speech, and voice characteristics of the original speaker. The pipeline starts with automatic speech recognition including emphasis detection, followed by the translation model. The translated text is then synthesized by a Text-to-Speech model that recreates the original emphases in the target sentence. The resulting synthetic speech is then converted back to the original speakers' voice using a voice conversion model. Finally, to synchronize the lips of the speaker with the translated audio, a generative model generates frames of adapted lip movements which are combined with the audio to produce the final output. The disclosure further describes several use-cases and configurations that apply these techniques to video conferencing, dubbing, low-bandwidth transmission, speech enhancement and assistive technology for the hearing impaired.

A neural network audio codec system and related methods are provided. In one example, a method is provided comprising: obtaining speech audio to be encoded; applying the speech audio to an audio encoder that is part of a neural network audio codec system that includes the audio encoder and an audio decoder. The audio encoder and the audio decoder have been trained in an end-to-end manner. The speech audio is encoded with the audio encoder to generate embedding vectors that represent a snapshot of speech audio attributes over successive timeframes of the raw speech audio, and from the embedding vectors, codeword indices are generated to entries in a codebook. The codeword indices are then transmitted or stored for later retrieval and processing by the audio decoder. Further aspects include converting the speech audio to text based on decoding the codeword indices, translating the speech audio from a first language into a second language by mapping codeword indices from a first language to a second language, converting the speech audio prosody by mapping the codeword indices, text generation and discrimination tasks.Yet another aspect includes converting text to speech audio with a different prosody.

The invention relates to the technical field of artificial intelligence, can be applied to the field of medical health, and discloses a timbre conversion method, device, equipment and medium based on cultural semanteme, the method comprises the following steps: constructing a cultural semantic timbre library comprising a semantic label and timbre characteristic parameter mapping relationship, the semantic label characterizing emotional semanteme of a target timbre, and the timbre characteristic parameter mapping relationship between the semantic label and the timbre characteristic parameter; the timbre characteristic parameters comprise a pitch range, rhythm rhythm and a harmonic structure; performing feature extraction based on text, image and audio multi-mode information to obtain semantic keywords, visual emotion features and audio acoustic features; performing attention weight fusion on the features through a multi-modal fusion deep learning model, and dynamically adjusting model parameters in combination with a semantic timbre library to generate a target timbre; and finally, intelligent conversion from the multi-mode information to the adaptive tone is realized. Through semantic-driven multi-modal feature collaborative optimization, the defect that timbre conversion machinery is stiff and lacks emotional expression is overcome, and the integrating degree of timbre expression and semantic scenes is improved.

This specification describes a computer-implemented method of generating context-dependent speech audio in a video game. The method comprises obtaining contextual information relating to a state of the video game. The contextual information is inputted into a prosody prediction module. The prosody prediction module comprises a trained machine learning model which is configured to generate predicted prosodic features based on the contextual information. Input data comprising the predicted prosodic features and speech content data associated with the state of the video game is inputted into a speech audio generation module. An encoded representation of the speech content data dependent on the predicted prosodic features is generated using one or more encoders of the speech audio generation module. Context-dependent speech audio is generated, based on the encoded representation, using a decoder of the speech audio generation module.

The invention discloses a traffic domain service voice adaptive generation method and system based on a thinking chain fine-tuning large model, and the method comprises the steps: converting an input voice signal into a high-dimensional voice feature signal through a voice encoder, and generating a text output signal according to the high-dimensional voice feature signal through a text decoder and a pinyin decoder; according to the invention, a variational de-quantization joint data enhancement mechanism, a dual-channel multi-mode discriminator architecture and a layered composite loss function are adopted to carry out high-fidelity emotional speech generation and robustness synthesis in a complex noise environment, and semantic-driven dynamic rhythm optimization and professional term precise pronunciation are synchronously supported; in an application scene in the traffic field, a multi-task speech recognition method can be adopted to realize efficient linkage of character-level recognition, audio-to-pinyin conversion and sentence-level accent classification modules, so that complex accent, more noise and traffic term polyphone challenges are effectively handled, and the method is suitable for being widely popularized and used.

The invention discloses a voice cloning system and method fusing rhythm characteristics, belongs to the technical field of voice synthesis and natural language, and is applied to the aspect of fine-grained rhythm control in zero-sample voice synthesis. The implementation method comprises the following steps of: 1, extracting rhythm features and audio features of an audio file, and further respectively acquiring pause features, speed features and tone features in the rhythm features by sequentially adopting transcriptional text inverse coding, syllable-level speed registration quantization and pitch sequence feature splicing modes; 2, fusing the features of the audio files in a manner of discarding feature screening without guidance of a classifier; 3, generating a target Mel spectrogram based on conditional flow matching; generating a target audio file from a to-be-cloned audio file through the trained voice cloning model controlled by the fusion rhythm; compared with the prior art, fine-grained rhythm control of tone and rhythm feature decoupling is realized in zero-sample speech synthesis, so that intonation accuracy based on a context scene is improved.

The invention discloses a spoken language practice scene-oriented education large model evaluation method, which comprises the following steps of: obtaining voice data and a corresponding standard text, interacting with a test model through the voice data, and obtaining a recognition text, audio data and an answer text provided by the test model; the standard text and the recognition text are compared and evaluated, and the voice recognition accuracy is obtained; carrying out pronunciation accuracy, fluency and rhythm evaluation on the audio data based on multi-modal feature fusion to obtain an audio score; performing grammar accuracy evaluation on the answer text based on a dynamic punishment mechanism and the constructed prompt large model, and performing question degree evaluation on the answer text based on the constructed prompt large model to obtain an answer text score; and comprehensively summarizing the voice recognition accuracy, the audio score and the answer text score to obtain a final evaluation result.

The invention provides a voice synthesis scheme and device, electronic equipment, a storage medium and a program product, and relates to the technical field of voice processing, and the method comprises the steps: inputting a text content into an LLM semantic understanding module, and obtaining a deep semantic feature and multi-modal context semantic information corresponding to the text content; transmitting the deep semantic features to a neural codec, and outputting compressed acoustic features; inputting the compressed acoustic features into an acoustic modeling module, and outputting high-precision acoustic features; inputting the high-precision acoustic features and the multi-mode context semantic information into an emotion and rhythm control module, and inputting voice parameters with emotion and rhythm marks; and inputting the voice parameters with the emotion and rhythm marks and the reference audio into a tone migration module to obtain a synthetic voice of a tone corresponding to the reference audio.

The invention relates to the technical field of voice processing, and provides a rhythm migration method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a decoupled rhythm feature based on a source rhythm voice, and a decoupled tone feature based on the voice of a target speaker, the decoupled rhythm feature represents the rhythm of the source rhythm voice, and the decoupled tone feature represents the tone of the target speaker; the decoupled timbre features represent the timbre of the voice of the target speaker; generating a target voice vector sequence based on the text features of the target text, the voice features of the voice of the target speaker, the decoupled rhythm features and the decoupled timbre features; and synthesizing a target audio based on the target voice vector sequence. According to the method and the device, the decoupled rhythm features and the decoupled timbre features are acquired, and the target voice is generated based on the features, so that the problem of feature mixing is effectively relieved, the timbre purity of the target speaker in cross-person rhythm migration is ensured, the expressive force of rhythm migration is improved, and the synthesized audio is more natural and vivid.

The present disclosure provides a speech synthesis model training method. The method can comprise: acquiring initial training data; selecting a plurality of first text segments, second text segments, and third text segments from coherent text, and acquiring first audio segments, second audio segments, and third audio segments from coherent audio; detecting whether the plurality of audio segments and text segments meet a stitching condition; on the basis of the text sequence, stitching, by alternatively arranging text and audio, the plurality of text segments and audio segments meeting the stitching condition to generate combined training data, so as to generate a combined training data set; on the basis of the combined training data set, training the initial speech generation model to obtain a trained speech synthesis model, wherein during the training, prosodic, tonal, and / or emotional features from the second text segments and / or the third text segments are extracted. Further disclosed is a speech synthesis method. The present disclosure allows for use of the context of the text and audio to realize text comprehension to capture prosody, tonality, and emotion in the context.

Systems and techniques described herein relate to a diffusion-based model for generating converted speech from a source speech based on target speech. For example, a device may extract first prosody data from input data and may generate a content embedding based on the input data. The device may extract second prosody data from target speech, generate a speaker embedding from the target speech, and generate a prosody embedding from the second prosody data. The device may generate, based on the first prosody data and the prosody embedding, converted prosody data. The device may then generate a converted spectrogram based on the converted prosody data, the speaker embedding, and the content embedding.

To effectively resolve the issue of inappropriate handovers during communication sessions, methods, systems, and machine-readable mediums which utilize speech features captured by microphones in the original wireless peripheral device and / or the wireless peripheral device to which the communication session is to be handed over to determine if a handover should proceed or should be reversed. Speech features refer to the composite attributes of spoken language that encompass both acoustic and linguistic features. Acoustic features characterize the sound properties of speech and include, but are not limited to, timbre, pitch, intonation, speaking rate, articulation, prosody, melody, spectral features, formant frequencies, and the like. Linguistic features pertain to the actual content conveyed, comprising words, phrases, syntax, and semantics. This includes the analysis of words, phrases, syntax, and semantics to understand the context and continuity of the conversation.

A speech-processing system may be configured to generate expressive synthesized speech. The system may include a prosody prediction model that generates a combination of durations and acoustic representations that may be based on the content of the text as well as additional context information. The model may be trained to predict a joint probability between linguistic representations (e.g., derived from text) and combined duration / acoustic representations (e.g., derived from audio). At inference, the model can process linguistic representations derived from text to predict combined duration / acoustic representations. In some implementations, the model may process additional information; for example, semantic embeddings output by a language model based on the text. In another example, the model may receive a speaker embedding representing voice characteristics of a particular speaker. A decoder may process the durations and acoustic representations output by the model to generate audio data representing the synthesized speech and representing expressive prosodic variation.

The invention provides a voice interaction-based intelligent medicine use reminding system and method for the elderly department. The voice interaction-based intelligent medicine use reminding method comprises the following steps of: extracting phoneme characteristics of a target elderly in each historical dialogue round from each interaction voice; determining the pronunciation loss of the target old person in each historical dialogue round according to the phoneme similarity between the interactive voices in each historical dialogue round and the rhythm feature of each interactive voice, and further determining the semantic constraint condition of the target old person in each historical dialogue round; performing multi-intention recognition on the interaction voice of the target old person in the current dialogue round to obtain different semantic intentions of the target old person in the current dialogue round; further determining the semantic integrity of the target old person in the current dialogue round; and correcting the voice content of the target old person in the current dialogue round through the semantic integrity and the intention confidence of each semantic intention. By adopting the scheme of the invention, the fuzzy and distorted voice content of the elderly can be corrected in the voice interaction process.

Disclosed are systems, methods, and devices, that overcome timing and self-expression limitations experienced by vocalists when using prerecorded vocal backing tracks to enhance live performances. The disclosed system, devices, and methods, dynamically synchronizes prerecorded vocal backing tracks with a live vocal stream by extracting vocal elements, such as phonemes, vector embeddings, or vocal audio spectra, from the live vocal performance in real-time. These extracted vocal elements are matched against corresponding timestamped vocal elements previously derived from the prerecorded vocal backing track, enabling precise real-time adjustment and alignment of the backing track timing to the live performance. Additionally, the system enhances expressive performance by identifying prosody factors, such as pitch, vibrato, accent, stress, dynamics, and level, in the live vocal performance, and dynamically adjusting corresponding prerecorded prosody factors within predefined ranges. This maintains naturalness and spontaneity in the vocalist's live performance, overcoming traditional limitations associated with prerecorded vocal backing tracks.

The invention discloses an intelligent communication content adaptation system and method based on user intention recognition. The system integrates and processes texts, voices, emoticons and unstructured behavior data through a multi-modal input analysis module; the deep learning intention recognition engine adopts a triple attention mechanism, current input and weighted historical interaction features are fused, and multi-level intention classification including basic operation, semantic targets and emotion driving is output; the real-time emotion state analysis module fuses acoustics, semantics and physiological indexes to generate a dynamic emotion matrix; the content adaptation decision engine is based on the intention confidence and the emotional state; and the multi-channel output optimization module performs cooperative adjustment on the speech synthesis rhythm, the text abstract and the visual interface according to the speech synthesis rhythm and the text abstract. According to the method, the perception precision of the deep intention and emotion of the user in a complex scene is remarkably improved, the individuation and multi-channel adaptive optimization of the response content are realized, and the communication efficiency and the user experience are effectively improved.

The invention discloses a cross-language voice interaction method and system based on multi-modal semantic understanding and a storage medium, and relates to the technical field of artificial intelligence and natural language processing. The method comprises the following steps: acquiring a source language speech stream, performing parallel double-flow feature extraction, extracting text semantic features through a mixed language code recognition model based on a unified phoneme space, and extracting acoustic features containing rhythm information at the same time; performing alignment fusion on the text and the acoustic features by using a cross-modal attention mechanism to generate multi-modal semantic representation; analyzing the explicit intention and the implicit emotion based on the representation, and generating a reply strategy and an emotion control label of the target language; and finally, synthesizing a target voice with a corresponding emotion color. According to the invention, the bottleneck that the traditional cascade architecture loses side language information is broken through, the precise understanding and strategic feedback of complex contexts such as Chinese-English mixed language codes (Code-mixing), anti-quiescence, hesitation and the like are realized, and the method is particularly suitable for transnational business negotiation and international customer service scenes.

Provided are systems, methods, and machine learning models for filling in gaps (e.g., of up to one second) in speech samples by leveraging an auxiliary textual input. Example machine learning models described herein can perform speech inpainting with the appropriate content, while maintaining speaker identity, prosody and recording environment conditions, and generalizing to unseen speakers. This approach significantly outperforms baselines constructed using adaptive TTS, as judged by human raters in side-by-side preference and MOS tests.

The invention discloses a speech emotion recognition method and system based on multi-modal feature fusion, and relates to the technical field of speech emotion recognition. The speech emotion recognition method and system based on multi-modal feature fusion comprises the following steps: S1, collecting a speech emotion data set, and carrying out label unified coding and normalization processing; s2, frame-level acoustics and construction of frequency spectrum, rhythm and sound quality emotion features are carried out; s3, a speech emotion representation generation method fusing multi-sub-mode depth coding and a gating cooperative attention mechanism; s4, performing random forest weight initialization and two-order variation grey wolf mapping evaluation; and S5, voice emotion recognition and operation feedback adaptive updating are carried out. According to the method, the feature selection efficiency and the emotion classification accuracy in voice emotion recognition are effectively improved, and the problems that existing voice emotion feature selection is single in stage and single in index, emotion retention and real-time performance are difficult to consider while dimension reduction is performed, and the overall performance is limited are solved.

A device, system and method for analyzing audio speech signals to detect fraudulent calls to a contact center comprising splitting an audio recording of a call in real-time into a foreground speech signal attributed to a main speaker and a background audio signal, extracting audio features from the foreground speech signal and background audio signal, inputting the extracted audio features into an ensemble model comprising multiple different machine learning models co-trained to cumulatively detect fraud, wherein the multiple different machine learning models include: a speaker audio model to detect audio speech anomalies, a speaker intent model to classify intent of the main speaker, a synthetic voice detection model to identify a non-human entity, and a prosody model to detect voice intonation of the main speaker. A prediction may be output, by the ensemble model, indicating whether the call is fraudulent.

A method for training a non-autoregressive TTS model includes receiving training data that includes a reference audio signal and a corresponding input text sequence. The method also includes encoding the reference audio signal into a variational embedding that disentangles the style / prosody information from the reference audio signal and encoding the input text sequence into an encoded text sequence. The method also includes predicting a phoneme duration for each phoneme in the input text sequence and determining a phoneme duration loss based on the predicted phoneme durations and a reference phoneme duration. The method also includes generating one or more predicted mel-frequency spectrogram sequences for the input text sequence and determining a final spectrogram loss based on the predicted mel-frequency spectrogram sequences and a reference mel-frequency spectrogram sequence. The method also includes training the TTS model based on the final spectrogram loss and the corresponding phoneme duration loss.

A speech-processing system receives input data representing text. A prosody prediction component processes the input data to determine prosody embedding data corresponding to prosody of the text. A decoder processes the prosody embedding data and phoneme encoded data derived from the input data to determine audio output data corresponding to the text and the prosody.

A response system for user voice interruption in a voice interaction process comprises an information importance mapping (ICM) generation module, an interruption response configuration file (BRP) generation module, an acoustic feature extraction module, a response control module, a semantic understanding module, a rhythm generation module, a transient response trigger and a response manager, according to the method, interruption intentions can be distinguished, the context importance can be evaluated, and differentiated responses with low delay, certainty and controllable sound characteristics can be realized.

The invention discloses a multi-user emotion recognition and digital human feedback method based on behavior data, and belongs to the field of artificial intelligence emotion calculation. Sensing user approaching, identifying and confirming the identity of a registered user, distributing an identifier, distributing an identifier for a visitor, and establishing an independent session channel in a multi-user scene; capturing time sequence dialogue behavior data containing voice rhythm features, dialogue interaction modes and linguistic features in real time; inputting the data into a sequence information processing model based on a state space theory and provided with a data dependence selection mechanism and an emotion analysis model spliced by static context features, and outputting multi-dimensional emotion and cognitive state tags; generating an emotion support strategy through a decision-making model in combination with the personal file of the user; and generating a multi-mode control instruction, and driving the digital human to synchronously present special effects such as expressions and the like. According to the method, privacy concerns are eliminated through non-intrusive collection, the method is adaptive to multi-user scenes, deep cognitive states can be recognized, and interaction effectiveness is improved for a long time.

The invention belongs to the technical field of speech analysis, and discloses a spoken English pronunciation quality evaluation method based on multi-modal speech feature analysis, which comprises the following steps: acquiring a spoken English speech signal of a target user, and performing multi-domain decomposition on the speech signal to obtain multi-modal speech features; performing time-frequency domain corresponding relation analysis and feature extraction on the multi-mode speech features to obtain a pronunciation detail feature sequence; performing multi-scale matching on the pronunciation detail feature sequence and a preset standard pronunciation template, constructing a multi-dimensional representation model based on a multi-scale matching result, and calculating a fine-grained quality score of a phoneme unit in each multi-dimensional representation model in combination with rhythm and rhythm parameters in the multi-modal speech features, the rhythm coherence score and the overall fluency score are fused to generate a comprehensive pronunciation quality evaluation result and a visual diagnosis report of pronunciation deviation; the oral English pronunciation evaluation method realizes comprehensive and refined evaluation of oral English pronunciation, and provides a scientific guidance basis for personalized language learning.

The invention relates to the technical field of speech synthesis, financial science and technology and medical health, and discloses a speech synthesis method, device and equipment of potential rhythm based on diffusion and a medium, and the method comprises the steps: generating a reference rhythm vector corresponding to each frequency band of an initial audio according to a Mel spectrogram and a preset real phoneme duration; according to the text hidden representation and the speaker hidden representation, constructing an analysis phoneme duration corresponding to the reference rhythm vector; constructing an analysis rhythm vector by using the text hidden representation, the speaker hidden representation and the analysis phoneme duration; generating a potential rhythm vector according to the reference rhythm vector and the analysis rhythm vector; and performing speech synthesis by using the text hidden representation, the speaker hidden representation and the potential rhythm vector to obtain a synthesized speech. The time step number is remarkably reduced through model analysis, the generation speed is increased, and meanwhile, the smoothness of the generated voice is remarkably improved through rhythm vector quantization.

The application discloses a kind of oral conversation data processing methods based on agent, belong to artificial intelligence big data processing technical field, including receiving the voice input file sent by user end, voice signal is converted into text data, while extracting emotional characteristics, generate emotional label and voice prosody characteristics, text and emotional information are combined to form dialogue anchor point information, construct path graph, calculate edge weight;According to the similarity of current dialogue anchor point and historical path node, construct propagation matrix, carry out intelligent path reasoning, calculate path score, ensure that emotional fluctuation influences task path selection, agent selects optimal path according to path score to carry out subsequent task processing, and real-time correction is carried out after user feedback;Through emotional analysis technology, real-time adjust the anchor point information of current conversation, and adjust the reasoning process according to feedback, form closed-loop adaptive adjustment mechanism, to optimize the response quality of system, enhance the sense of trust of user to system.

The invention discloses a multi-language intelligent dubbing generation system based on sound cloning and emotion migration. The multi-language intelligent dubbing generation system comprises a sound cloning module, a cross-language synthesis module, an emotion migration module and a lip shape synchronization module. By constructing a few-sample speaker encoder, a cross-language rhythm migration module, a fine-grained emotion control module and a video lip shape synchronization module, end-to-end automatic generation from original dubbing audio to multi-language target dubbing is realized, and tone consistency, emotion authenticity and picture synchronism are kept.

The invention relates to a speech synthesis method and device, and the method comprises the steps: obtaining a to-be-synthesized text; performing phoneme conversion on the to-be-synthesized text to obtain a first phoneme sequence, the first phoneme sequence comprising at least one phoneme and a tone of a syllable where the at least one phoneme is located; decoupling phonemes and tones in the first phoneme sequence, and extracting phoneme features of the to-be-synthesized text based on a decoupled second phoneme sequence; wherein the second phoneme sequence comprises the at least one phoneme; performing text coding on the to-be-synthesized text in a syllable dimension, and extracting semantic features of the to-be-synthesized text; and performing fusion processing on the semantic features and the phoneme features, and generating voice based on the fused features. According to the invention, the synthesized voice is accurate in pronunciation and natural in rhythm, and it is ensured that the synthesized voice achieves expected effects in the aspects of naturalness, rhythm and semantic consistency.

A device, system and method for analyzing audio speech signals to detect fraudulent calls to a contact center comprising splitting an audio recording of a call in real-time into a foreground speech signal attributed to a main speaker and a background audio signal, extracting audio features from the foreground speech signal and background audio signal, inputting the extracted audio features into an ensemble model comprising multiple different machine learning models co-trained to cumulatively detect fraud, wherein the multiple different machine learning models include: a speaker audio model to detect audio speech anomalies, a speaker intent model to classify intent of the main speaker, and a prosody model to detect voice intonation of the main speaker. A prediction may be output, by the ensemble model, indicating whether the call is fraudulent.