569 results about "Speech generation" patented technology

The invention provides an intelligent real-time interactive question-answering system based on a virtual digital human, and belongs to the technical field of voice signal processing and voice recognition, and the system comprises a data acquisition module which receives a voice or text interaction request input by a user, collects the expression dynamic parameter sequence and limb movement sequence data of the user in real time, and transmits the data to a user interaction module; obtaining a standardized voice feature vector and structured text data; the cross-modal fusion module is used for constructing an interactive feature matrix; the behavior decision module outputs a decision instruction set; the knowledge retrieval module is used for generating an answer text with emotional adaptability and voice features; and the voice generation module is used for generating a mouth shape animation key frame, a micro expression parameter sequence and a limb action track of the virtual digital human, generating a voice response in combination with the answer text and the voice characteristics, and pushing the voice response to the user terminal. According to the method, the interaction experience and adaptability of the virtual digital human are remarkably improved.

The invention relates to the technical field of artificial intelligence, can be applied to business scenes of medical health, financial science and technology and the like, and discloses a voice generation method which comprises the following steps: constructing a multi-language voice synthesis model, obtaining plain text data and paired voice text data, and constructing an expansion vocabulary; updating a language perception embedding layer and model parameters, and converting an input text into a mark sequence; and the encoder extracts context semantic features, extracts pronunciation rule features, and the decoder fuses the features to generate an acoustic feature sequence, and converts the acoustic feature sequence into target voice data. According to the invention, the multi-language speech synthesis model is combined with the language perception embedding layer, so that the speech generation capability of a low-resource language is improved; the text conversion accuracy is improved by expanding the vocabulary, the target language learning ability is enhanced by unsupervised training, the low data environment adaptability is optimized by supervised training, and the speech naturalness and fluency are improved by feature fusion.

The invention relates to the technical field of artificial intelligence, can be applied to business scenes of medical health, financial science and technology, cultural transmission and the like, and discloses a voice generation method based on multi-modal fusion, which comprises the following steps: collecting audio data to extract timbre features, and training a field feature timbre generation model; analyzing text semantic recognition emotion information, adjusting speech synthesis parameters, combining personalized information to construct a parameter mapping table, fusing to generate a synthesis control parameter sequence, aligning the synthesis control parameter sequence with character labels, visual elements and background music data, driving a domain feature timbre generation model, and generating synthesis data of synchronous speech, text, vision and music. Domain timbres are generated through timbre feature training, speech expression is optimized by combining semantic analysis and emotion recognition, user requirements are matched based on personalized information, and time alignment is performed by fusing text, vision and music data, so that the synthesized speech has domain features, emotion adaptability and personalization, and the speech quality is improved. And the voice immersion and the information transmission capability are improved.

Methods, systems, and computer programs are presented for audio-visual speech generation with multi-modal latent diffusion models. One method includes encoding raw audio signals and video frames into respective latent spaces using audio and visual autoencoders. A text transcript is processed into phoneme sequences using a text transcript processor. The audio and visual latent spaces are conditioned using the text transcript and a conditioning variable. Joint distributions of the visual and audio latent spaces, text transcript, and conditioning variable are learned using a multi-modal latent diffusion model. The model adds noise to the latent audio-visual representations and predicts the noise through denoising neural networks. An inverted diffusion process is utilized to generate diverse speech content and speaker characteristics, resulting in realistic audio-visual speech. The technology presented provides a novel approach to conditional speech generation with potential applications in speech synthesis, voice conversion, and speech recognition.

The invention relates to the technical field of speech synthesis, can be applied to business scenes such as financial science and technology, medical health and the like, and discloses a speech generation method, device, equipment and medium. And inputting a text speech language model to generate an intermediate code in combination with a speech code extracted based on a codebook generation mode, extracting a speaker feature vector in the prompt speech, decoding the intermediate code and the speaker feature vector by a generative adversarial decoder, and outputting a target speech. According to the method, fine-grained text representation is established by fusing character semantics and pinyin pronunciation information, and voice cloning is completed through unified voice language modeling and an adversarial generation mechanism in combination with codebook-driven acoustic coding and speaker personality characteristics, so that the naturalness, similarity and pronunciation accuracy of generated voices are improved.

The invention relates to the technical field of voice processing, can be applied to business scenes in the fields of medical health, financial science and technology, barrier-free service and the like, and discloses a text-to-voice method based on knowledge distillation, which comprises the following steps: carrying out standardization processing on an input text to generate a standard text sequence; the lightweight text encoder encodes the standard text sequence to generate a text implicit vector; the non-autoregressive acoustic feature prediction module maps the text implicit vector into a student acoustic feature sequence, and calculates alignment loss through knowledge distillation; and performing structured pruning and parameter quantization based on alignment loss, generating an acoustic feature sequence by the optimized model, and converting the acoustic feature sequence into a voice waveform by a vocoder. According to the method, through knowledge distillation, pruning optimization and parameter quantification, the reasoning speed and the cross-equipment adaptability are improved while the model size and the calculation requirement are reduced, so that the TTS system can realize efficient, low-delay and low-power-consumption voice generation in a resource-constrained environment.

The present application provides training and speech generation methods and apparatuses for a speech generation model, an electronic device, a computer-readable storage medium, and a computer program product. The method comprises: obtaining a first speech generation model; obtaining sample data of a plurality of modalities; on the basis of a prompt image sequence and speech text, respectively calling a plurality of encoders to perform encoding, so as to obtain a multi-modal encoding vector sequence; on the basis of the multi-modal encoding vector sequence, calling a decoder to perform decoding, so as to obtain decoded text; determining a probability distribution for the decoded text and the sample data of the plurality of modalities, and determining a target loss on the basis of the probability distribution; and on the basis of the target loss, updating parameters of the decoder and at least one of the encoders, wherein the updated decoder and the plurality of updated encoders are configured to form a second speech generation model, and the second speech generation model is used to generate target speech text corresponding to a prompt image sequence of a target object.

The invention relates to the technical field of voice processing, can be applied to business scenes such as financial science and technology and medical health, and discloses an environmental acoustics simulation voice generation method, device and equipment and a medium, and the method comprises the steps: obtaining and separating mixed voice data, and generating original voice content and original environmental acoustics information; converting the original voice content into first text information; determining a target environment acoustic tag in combination with the original environment acoustic tag, the first text information and the target geographical location information; acquiring target environmental acoustic information from a preset sound data set based on the tag, and adjusting the amplitude characteristic of the target environmental acoustic information to match the original environmental acoustic information; and synthesizing the adjusted target environment acoustic information and the original voice content into simulated voice data. According to the method, the target geographic position information is introduced to participate in acoustic feature determination and amplitude adjustment, so that the generated simulated voice data is more consistent in geographic semantics and acoustic performance, and the authenticity and concealment of voice environment disguise are effectively improved.

The invention relates to the technical field of speech synthesis, can be applied to business scenes of medical health, financial science and technology and the like, and discloses a hierarchical emotional speech generation method, which comprises the following steps: acquiring an input text and an emotional speech sample, extracting a text embedding feature from the input text, extracting a Mel spectrum feature from the emotional speech sample, and obtaining a text embedding feature; the method comprises the following steps: extracting phoneme-level, word-level and statement-level sentiment distribution characteristics through a hierarchical sentiment distribution extraction module, carrying out time dimension alignment, generating a multi-level sentiment guidance matrix, and inputting the multi-level sentiment guidance matrix and text embedding characteristics into a sentiment synthesis module to generate a target Mel spectrum; and finally, converting the target Mel spectrum into target voice through a vocoder. The emotion control is expanded from the statement level to the fine-grained level of phonemes, words and the like, and the multi-level emotion guidance matrix is combined for generation, so that the generated speech is finer and more abundant in emotion expression and conforms to the context, and the naturalness of speech generation and the accuracy of emotion transmission are improved.

The invention discloses a traffic domain service voice adaptive generation method and system based on a thinking chain fine-tuning large model, and the method comprises the steps: converting an input voice signal into a high-dimensional voice feature signal through a voice encoder, and generating a text output signal according to the high-dimensional voice feature signal through a text decoder and a pinyin decoder; according to the invention, a variational de-quantization joint data enhancement mechanism, a dual-channel multi-mode discriminator architecture and a layered composite loss function are adopted to carry out high-fidelity emotional speech generation and robustness synthesis in a complex noise environment, and semantic-driven dynamic rhythm optimization and professional term precise pronunciation are synchronously supported; in an application scene in the traffic field, a multi-task speech recognition method can be adopted to realize efficient linkage of character-level recognition, audio-to-pinyin conversion and sentence-level accent classification modules, so that complex accent, more noise and traffic term polyphone challenges are effectively handled, and the method is suitable for being widely popularized and used.

The invention belongs to the technical field of three-dimensional reconstruction, and discloses a three-dimensional digital human generation method and system capable of voice interaction. According to the invention, brand new speaking audios in different languages are automatically generated according to different languages of the input target text and the sampled human voice audios; the sequential stability and detail reduction capability of three-dimensional human motion are guaranteed by using multi-model joint estimation and a sequential loss function, and facial expression details and hand postures in the image can be accurately estimated. After the high-precision three-dimensional human body model is obtained through estimation, human body action and expression generation is carried out based on voice driving, accurate synchronization of actions and expressions generated through voice is achieved, and facial expression movement and body posture movement, namely a whole-body three-dimensional human body model, conforming to brand-new speaking audio are accurately generated; and finally, rendering the whole-body three-dimensional human body model into a real digital human capable of voice interaction by using a three-dimensional neural rendering model. According to the invention, the realization of single person picture input, high-precision three-dimensional digital person generation and voice interaction is facilitated.

The invention relates to the technical field of artificial intelligence, can be applied to business system platforms of financial science and technology, medical health and the like, and discloses an emotional voice generation method, device, equipment and medium, and the method comprises the steps: obtaining a target text of a to-be-generated voice and a target emotional prompt text based on natural language description; inputting the target emotion prompt text into a pre-trained emotion encoder to obtain an emotion embedding vector; inputting the target text into a pre-trained text encoder to obtain a semantic embedding vector; inputting the emotion embedding vector and the semantic embedding vector into a pre-trained joint speech generation model, performing fused speech feature prediction on the emotion embedding vector and the semantic embedding vector, and generating a speech feature with an expected emotion; and decoding the voice features to generate target emotional voice corresponding to the target text. The speech emotion is controlled through the emotion prompt text described by the natural language, and the flexibility and synthesis effect of emotion speech generation interaction are improved.

The invention relates to the technical field of speech synthesis, can be applied to service system platforms of medical health, financial science and technology and the like, and discloses a speech generation method, device, equipment and medium based on rhythm characteristics, and the method comprises the steps: obtaining an input text and a corresponding reference speech, extracting a Mel spectrum, and generating speech representation; extracting overall rhythm features through a multi-head attention mechanism, combining with a text embedding feature input condition flow matching module, generating intermediate rhythm features, and executing threshold adjustment processing to obtain a fine rhythm feature set; and fusing the fine rhythm feature set with the text embedded features to generate an acoustic parameter sequence, and converting the acoustic parameter sequence into a voice waveform through a pre-trained neural vocoder. According to the method, rhythm feature generation is optimized through condition flow matching, the flexibility of rhythm expression is enhanced in combination with threshold adjustment, and the matching degree of the text and the rhythm is improved through cross-modal attention fusion, so that the generated voice is more natural, diversified and controllable.

The invention discloses a privacy information desensitization method and system for a voice generation type large model, and relates to the technical field of artificial intelligence. Input voice data is discretized, Gaussian noise disturbance sensitive features are injected, and a cross-modal voice generation model is constructed in combination with three-stage training; and meanwhile, a cross-modal privacy enhancement mechanism is applied to detect and fuzzify sensitive information in real time in an output stage. According to the invention, the adaptability of a large-scale voice generation model in a privacy protection scene is improved, and comprehensive protection of user privacy is realized. And moreover, the capability of extracting user privacy information by an adversarial attacker is effectively limited, and the risk of sensitive information leakage in the data transmission, storage and generation process of the voice generation model is reduced. On the premise that privacy is ensured, the voice generation model can still keep high-quality generation performance, generated voice output has high naturalness and accuracy, and actual application requirements are met.

Techniques for determining when speech is directed at another individual of a dialog, and storing a representation of such user-directed speech for use as context when processing subsequently-received system-directed speech are described. A system receives audio data and / or video data and determines therefrom that speech in the audio data is user-directed. Based on this, the system determine whether the speech is able to be used to perform an action by the system. If the speech is able to be used to perform an action, the system stores a natural language representation of the speech. Thereafter, when the system receives system-directed speech, the system generates a rewrite of a natural language representation of the system-directed speech based on the previously-received user-directed speech. The system then determines output data responsive to the system-directed speech using the rewritten natural language representation.

The present disclosure provides a speech synthesis model training method. The method can comprise: acquiring initial training data; selecting a plurality of first text segments, second text segments, and third text segments from coherent text, and acquiring first audio segments, second audio segments, and third audio segments from coherent audio; detecting whether the plurality of audio segments and text segments meet a stitching condition; on the basis of the text sequence, stitching, by alternatively arranging text and audio, the plurality of text segments and audio segments meeting the stitching condition to generate combined training data, so as to generate a combined training data set; on the basis of the combined training data set, training the initial speech generation model to obtain a trained speech synthesis model, wherein during the training, prosodic, tonal, and / or emotional features from the second text segments and / or the third text segments are extracted. Further disclosed is a speech synthesis method. The present disclosure allows for use of the context of the text and audio to realize text comprehension to capture prosody, tonality, and emotion in the context.

The invention relates to the technical field of robot control, and discloses a spiking neural network-based robot control method, which comprises the following steps of: preprocessing collected multi-modal data to obtain an emotion pulse signal; inputting the emotion pulse signal into a pre-constructed emotion pulse neural network, and outputting a comprehensive emotion pulse; inputting the comprehensive emotion pulse into a central pattern generator, and outputting a behavior rhythm; acquiring an environment feedback signal generated by executing the behavior rhythm, and adjusting a connection weight according to the environment feedback signal; optimizing the comprehensive emotion pulse according to the connection weight, and converting the optimized comprehensive emotion pulse into emotion interaction voice information; and adjusting the emotion intensity according to the optimized comprehensive emotion pulse and the multi-modal data. According to the method, data are collected through the multi-mode sensor to generate emotion pulses, the emotion pulses are input into the central mode generator to generate behavior rhythms after SNN processing, weight optimization, emotional speech generation and emotional steady-state control setting are combined with the STDP algorithm, and the efficiency of a robot service scene is improved.

The invention belongs to the technical field of artificial intelligence, and particularly relates to a voice generation method, a voice generation device, a computer readable medium, electronic equipment and a computer program product. The method comprises the following steps: acquiring a natural language instruction, wherein the natural language instruction is used for describing a presentation effect of voice in a natural language; a sample pair having a similar voice presentation effect with the natural language instruction is obtained, the sample pair comprises an audio sample and a text sample, and the text sample is used for describing the voice presentation effect of the audio sample; combining the natural language instruction with the sample pair to obtain a multi-modal cue word; and generating voice according to the multi-mode prompt word. According to the invention, the control precision and flexibility of voice generation can be improved.

The invention relates to the field of artificial intelligence, and discloses a speech generation method and device based on a conditional flow matching model, computer equipment and a storage medium, and the method comprises the steps: obtaining a training text of a generated speech, and taking the speech as a training speech; sampling noise data, and inputting the training text, the training voice and the noise data into a conditional flow matching model for training; reasoning the conditional flow matching model by using a variable step size to obtain a vector field about noise and voice; and constructing a voice generation model based on the conditional flow matching model and the vector field, and performing reasoning prediction on a target text of a to-be-generated voice by using the voice generation model. According to the embodiment of the invention, the text of the generated voice is utilized to train the conditional flow matching model so as to construct the voice generation model, and in the training process of the conditional flow matching model, the variable step size is adopted for reasoning and solving, so that the voice can be quickly generated, and the voice generation quality can be improved.

The invention discloses a digital human generation method and device based on audio driving and a computer storage medium, and relates to the technical field of digital human speech generation, and the method comprises the steps: extracting the audio features of an input audio, and carrying out the partitioning according to the amplitude, and obtaining audio blocks; inputting the audio blocks into the DIT model, and performing sliding block processing on the audio blocks in combination with random noise and time vectors to obtain audio drive to generate feature key points; identifying face key points of an input face image, performing feature extraction on the face image, and determining original feature key points of a to-be-driven face; according to the time sequence information of the input audio, performing feature fusion on the audio drive generation feature key points and the to-be-driven face original feature key points to obtain fusion key point features; and based on the fused key point features, driving the facial expressions and actions of the digital human, and generating the facial animation matched with the input audio, so that high synchronization of the facial animation and the audio is realized, and the naturalness of the digital human speaking is improved.

The invention relates to the technical field of speech synthesis, can be applied to business system platforms of communication, medical health, financial science and technology and the like, and discloses a speech generation method, device, equipment and medium based on distribution prediction. And inputting the context features and the historical waveform feature sequence into a next distribution prediction module, predicting probability distribution for generating a voice waveform at a next moment, generating voice waveform points based on the probability distribution, updating the historical waveform feature sequence, performing loop execution until a termination condition is met, and finally performing combination to generate a target voice waveform. According to the method, context coding and historical waveform features are combined, the long-distance dependence modeling capability is improved, the voice coherence and naturalness are ensured, voice generation is optimized through distributed prediction, the reasoning efficiency is improved, and the real-time synthesis requirement is met.

The invention relates to an emotion recognition method and device in film and television play dubbing, and the method comprises the steps: obtaining a film and television play audio of a source language and a to-be-dubbed text of a target language; the movie and television play audio is processed by using an emotion recognition model corresponding to the trained source language, and target emotion features are extracted; wherein the emotion recognition model is based on an audio pre-training sub-model and a feature extraction sub-model, and is obtained by training marked training data; the feature extraction sub-model is used for extracting audio features, and the audio pre-training sub-model is used for carrying out feature coding on the extracted audio features to generate emotional features; and performing voice synthesis on the to-be-dubbed text based on the target emotional characteristics by using a voice generation model corresponding to the trained target language, and generating dubbing of the target language corresponding to the movie and television play audio. Through the dubbing method and device, the original emotion is reserved in dubbing, cross-language voice seamless conversion is achieved, and the dubbing achieves the original taste and flavor effect.

Embodiments of the disclosure provide a solution for speech generation. A method includes: determining, based on a target text, a plurality of phoneme feature representations corresponding to a sequence of phonemes in the target text and respective phoneme durations for the plurality of phoneme feature representations; extending the plurality of phoneme feature representations based on the respective phoneme durations, to obtain an extended sequence of phoneme feature representations; masking at least one phoneme feature representation in the extended sequence of phoneme feature representations, to obtain a sequence of masked phoneme feature representations; and generating a target speech corresponding to the target text at least based on the sequence of masked phoneme feature representations.

The invention belongs to the technical field of voice generation, and relates to a voice generation method and device, equipment and a storage medium thereof. Obtaining a visual feature vector sequence of the target video frame sequence; inputting the text feature vector sequence and the visual feature vector sequence into a cross-modal fusion layer to obtain a feature fusion representation output by the cross-modal fusion layer; obtaining acoustic features output by the variant acoustic model according to the feature fusion representation; and inputting the acoustic features into a preset vocoder, and generating a target voice waveform through multi-scale convolution and up-sampling processing. According to the method, not only text features but also video features are introduced during voice generation, and voice generation is realized through cross-modal fusion features. The method is applied to financial or medical service intelligent customer service answering or service product marketing introduction scenes, and more natural and real voice is generated in combination with visual context information.

Disclosed systems and methods convert user-supplied textual content into animated videos. Textual input is received and processed to identify narrative elements such as characters, settings, and events. These elements are then transformed into visual scene components. Still images generated based on these components are subsequently animated in line with the narrative context. The system can automatically implement storytelling techniques adapted to incorporate neuroscience principles. The system also synthesizes speech for dialogues or narrations using voice synthesis technology that considers emotional markers, tone, and pace. Generated media and metadata are stored in a data storage system that maintains data integrity and enables efficient retrieval. Users can interact with an export interface to choose video resolution, format, and sharing options. A feedback system employing machine learning algorithms collects and analyzes user feedback for real-time adjustments to the generated animated video.

The invention relates to the technical field of speech synthesis, can be applied to business scenes of medical health, financial science and technology, debate and the like, and discloses a speech generation method and device based on speech style adaptation, equipment and a medium, and the method comprises the steps: obtaining a target text, a target speaker speech and a reference style speech, extracting a phoneme feature sequence, acoustic features and rhythm coding information, and performing feature fusion to generate a fusion coding vector; and processing the fusion coding vector through a style adaptation module, generating an acoustic code containing a target style feature, generating a Mel-frequency spectrum feature based on the acoustic code, and inputting the Mel-frequency spectrum feature into a pre-training vocoder to generate a voice waveform. The voice style matching and rhythm control capability is improved through multi-feature fusion, the generalization capability of unseen speakers is enhanced by optimizing the style adaptation module, high-naturalness voice is generated in combination with the Mel spectrum and the vocoder, and the voice synthesis definition and adaptability are improved.

A device for creating digital persona, which includes a data collection module responsible for collecting personality data of a target object, a personality training module utilizing a large language model and the personality data to train and generate a virtual personality model with personality characteristics of the target object, thereby generating a virtual personality consistent with the personality characteristics of the target object, an appearance video generation module and voice generation module respectively utilizing face replacement and voice cloning technologies to extract the pictures and sounds from the personality data to generate a virtual personality model with the appearance and voice characteristics of the target object, a lip synchronization module, utilizing lip synchronization technology to ensure that the mouth shape and voice of the digital persona are synchronized, and an interactive module, providing an interactive interface that allows users to interact with the virtual personality and receive responses from it.

The invention relates to the technical field of voice semantics, can be applied to business scenes of financial science and technology, medical health and the like, and discloses a voice generation method, device and equipment based on pseudo-autoregression modeling and a medium, and the method comprises the steps: obtaining a training sample containing a text sequence, a prompt voice segment and a target semantic token sequence; performing continuous fragment mask training on the text-to-semantic model to obtain a pseudo-autoregression trained text-to-semantic model; generating candidate speech output by using the text-to-semantic model and the initial semantic-to-acoustic model which are subjected to pseudo-autoregression training, and constructing a preference data pair; updating the semantics-to-acoustics model based on the preference data pair to obtain a preference optimized semantics-to-acoustics model; and generating target voice output based on the target text and the target prompt voice. According to the method, the time sequence modeling capability of the model is enhanced through pseudo-autoregression training, and the voice generation quality is directly optimized through the preference data pair, so that the voice alignment precision and the subjective listening feeling performance are improved.

Embodiments of the present application provide a method for speech generation and a related device, the method includes: obtaining a first source data input to a speech generation model including multiple encoders and a decoder, where types of input data of the multiple encoders are different; generating a first acoustic feature by a first encoder among the multiple encoders based on the first source data; and converting a second acoustic feature determined from the first acoustic feature into a third acoustic feature by the decoder. The above technical solution can rely on a single model to perform different speech generation tasks.

The invention belongs to the technical field of speech recognition and acoustic modeling, and particularly relates to a high-fidelity end-to-end acoustic modeling method combining speech intelligibility reconstruction and autoregressive feedback optimization, which comprises the following steps of: firstly, establishing an intelligibility loss function based on a human speech intelligibility model; subjective definition features of the voice are reconstructed through a frequency spectrum reconstruction network, intelligibility related features are extracted and embedded in combination with a perceptual weighted filter bank, and the intelligibility related features and Mel frequency spectrum features are input into an acoustic encoder in parallel; based on this, by introducing a voice intelligibility reconstruction and autoregressive feedback mechanism, while the conciseness of an end-to-end voice modeling framework is maintained, systematic improvement of voice sharpness, naturalness and stability is realized, and the method can be widely applied to scenes such as intelligent voice assistants, voice transcription, virtual anchors, voice restoration, cross-language voice generation and the like. And the method has extremely high practical application value and popularization prospect.